{"id":3540,"date":"2017-04-16T11:12:51","date_gmt":"2017-04-16T07:12:51","guid":{"rendered":"https:\/\/nayarweb.com\/blog\/?p=3540"},"modified":"2017-04-20T21:56:54","modified_gmt":"2017-04-20T17:56:54","slug":"analysing-car-market-trends-using-big-data-tools-in-mauritius","status":"publish","type":"post","link":"https:\/\/nayarweb.com\/blog\/2017\/analysing-car-market-trends-using-big-data-tools-in-mauritius\/","title":{"rendered":"Analysing Car Market Trends using Big Data Tools in Mauritius"},"content":{"rendered":"

Who hasn’t heard of Big Data<\/em> in the 21th century? Big Data in itself isn’t much of a great deal as knowing how to extract useful information from the data. It is the analytics part which is the killer feature of Big Data. It comprises of both science and art!<\/p>\n

\"\"
source: www.fudzilla.com<\/figcaption><\/figure>\n

The first step of Big Data solutions is to gather data. There are lots of ways in which you can achieve this. Manual Data Entry<\/em> can still be done but you’ll require quite an army of minions for that. But I was\u00a0not “evil” enough to convince the minions to work for me. So I had to find other ways: Facebook Graph API allows you to get feeds from your\u00a0wall, car groups you’re in.<\/p>\n

{
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“message”: “Renault Scenic
\nRs138,000 – Vacoas, Plaines Wilhems, Mauritius<\/span><\/span><\/p>\n

For sale Renault Scenic Year 2005,STEPTRONIC gearbox,fully executive,1500 cc Petrol,never accidented,comes with digital dashboard,armrest,electric mirrors,alloy wheels and panoramic sunroof”,
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“updated_time”: “2017-04-16T05:57:29+0000”<\/span>,
\n<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0“id”: “14851777<\/a>“<\/span><\/span><\/p>\n

},<\/p><\/blockquote>\n

Good news is that the data is in JSON format. Bad news is that the message field contains unstructured data. We have to extract the informations we want from it. I use Collectiva Beta<\/a><\/span> service (from nayarweb.com) for data processing.<\/p>\n

\"\"<\/a>
Collectiva Beta<\/figcaption><\/figure>\n

Let’s extract the Make, Model, Price, Location and Year from the data. I use grok patterns on the message field.<\/p>\n

Make and Model:\u00a0^%{NOTSPACE:make} %{NOTSPACE:model}<\/p>\n

Year:\u00a0(y(ea)*r|an(n*e*))\\s*\\:*\\s*(?<year;int>[0-9]+) \/\/ works with an 96, year 2003, yr : 14, anne 2000<\/p>\n

Price & Location:\u00a0(((Rs|\\$|\u00a3|\u20b9)(?<price_string>([0-9,])*))|FREE)( – %{GREEDYDATA:location_string})?<\/p><\/blockquote>\n

Now we\u00a0can just throw data at it. Some will get properly parsed, some not. But with enough data, you can still get some pretty graphs. In the future, hopefully I get an AI do the extracting for me.<\/p>\n

Let’s get to the pretty parts now: graphs.<\/p>\n

Which car makes\u00a0are the most sold in Mauritius?<\/h2>\n

Easy: A Bar Chart<\/p>\n

\"\"<\/p>\n

How many models are within each make?<\/h3>\n

No need to have another graph. We just split the\u00a0split bars for\u00a0models. Voila!<\/p>\n

\"\"<\/p>\n

How much do the car models depreciate in Mauritius?<\/h3>\n

\"\"<\/p>\n

Notice that it is graphing the 50th percentile which is also known as median. Means<\/em>\u00a0are kinda useless because a single outlier can cause the graphs to be very skewed by a lot. We can show the 10th, 50th, 90th percentile on the same graph so that you can compare whether the car you’re buying falls between the accepted market price. Let’s say you want a Volkswagen Polo<\/p>\n

\"\"<\/p>\n

You can see the same lines above are not split into 3 lines. I don’t have much variation here because I’ve been collecting data for only 4 days. The lines are supposed to smoothen over time and the accuracy will increase.<\/p>\n

Wanna\u00a0see the Big Picture? Welcome the Pi Chart<\/h3>\n

In this graph, I aggregated the data into Make, Model, Year and Price. All in 1 graph. It’s like magic nah?<\/p>\n