Hadoop代写:CSE3BDC Big Data Tools Task2


接上次的大数据作业,Task2是对时间序列数据集进行大数据分析,要求分别使用Spark, Hive以及Hadoop来处理。

Task 2: Analysing Twitter Time Series Data

In this task we will be doing some analytics on real twitter data. The data is a section of the data from obtained from the infochimps.org web site.
The data set used for this task can be found inside the twitter directory of the assignment_datafiles.zip on LMS. Note the data file is tab (\t) delimited.
The data has the following attributes

Attribute number Attribute name Description
1 Token type In our data set all rows have Token type of hashtag. So this attribute is useless for this assignment.
2 Month The year and month specified like the following: YYYYMM. So 4 digits for year followed by 2 digits for month. So like the following 200905, meaning the year 2009 and month of May.
3 count An integer representing the number tweets of this hash tag for the given year and month.
4 Hash Tag Name The #tag name, e.g. babylove, mydate, etc.

Here is a small example of the Twitter data that we will use to illustrate the subtasks below:

Token type Month count Hash Tag Name
hashtag 200910 2 babylove
hashtag 200911 2 babylove
hashtag 200912 90 babylove
hashtag 200812 100 mycoolwife
hashtag 200901 201 mycoolwife
hashtag 200910 1 Mycoolwife
hashtag 200912 500 mycoolwife
hashtag 200905 23 abc
hashtag 200907 1000 abc

Using the twitter data set downloaded from LMS please perform the following.
a) [Spark] Find the single row that has the highest count and for that row report the month, count and hashtag name. So for the above small example data set the result would be:

Month: 200907, count: 1000, hash tag name: abc

b) [Do twice, once using Hive and once using Spark] Find the hash tag name that has tweeted the most in the entire data set. Report the total number of tweets for that hash tag name. So for the above small example data set the output would be:

abc    1023

c) [MapReduce] Given two months x and y, where y > x. Find the hashtag name that has increased the number of tweets the most from month x to month y. Ignore the tweets in the months between x and y, so just compare the number of tweets at month x and at month y. Report the hashtag name, the number of tweets in months x and y. Ignore any hashtag names that had no tweets in either month x or y. You can assume that the combination of hashtag and month is unique. Therefore, the same hashtag and month combination cannot occur more than once. Take x and y as command line arguments as was done in Task D of Lab 3. For the above small example data set:

Input: x = 200910, y = 200912
Output (hashtag, month x count, month y count):
mycoolwife, 1, 500

Appendix A: Working with Linux

As 3rd/4th year students of a computer science subject, we assume that you know how to use Linux. If you have not used Linux before, this is a good opportunity to gain hands-on experience. After startup, the VM will automatically log you in and load into a graphical environment. However, you may find it useful to work with the command shell (=bash, =terminal) sometimes.
It is possible to complete the entire assignment using just the tools currently installed in the VM without installing any other tools. Anyway, your user “cloudera” (password “cloudera”) possesses all necessary rights for downloading and installing additional application packages.

Appendix B: Transferring files between your host system and the Cloudera VM

The following information is useful particularly if your drag n drop is not working correctly.
VirtualBox: If you create a folder on your host system and mark is as a shared in the settings of your VirtualBox application.
You can mount it to the folder ~/Public of your Linux guest operating system using the following command:

sudo mount -t vboxsf Temp ~/Public