Introduction
接上次的大数据作业，Task2是对时间序列数据集进行大数据分析，要求分别使用Spark, Hive以及Hadoop来处理。
Task 2: Analysing Twitter Time Series Data
In this task we will be doing some analytics on real twitter data. The data is a section of the data from obtained from the infochimps.org web site.
The data set used for this task can be found inside the twitter directory of the assignment_datafiles.zip on LMS. Note the data file is tab (\t) delimited.
The data has the following attributes
| Attribute number | Attribute name | Description |
|---|---|---|
| 1 | Token type | In our data set all rows have Token type of hashtag. So this attribute is useless for this assignment. |
| 2 | Month | The year and month specified like the following: YYYYMM. So 4 digits for year followed by 2 digits for month. So like the following 200905, meaning the year 2009 and month of May. |
| 3 | count | An integer representing the number tweets of this hash tag for the given year and month. |
| 4 | Hash Tag Name | The #tag name, e.g. babylove, mydate, etc. |
Here is a small example of the Twitter data that we will use to illustrate the subtasks below:
| Token type | Month | count | Hash Tag Name |
|---|---|---|---|
| hashtag | 200910 | 2 | babylove |
| hashtag | 200911 | 2 | babylove |
| hashtag | 200912 | 90 | babylove |
| hashtag | 200812 | 100 | mycoolwife |
| hashtag | 200901 | 201 | mycoolwife |
| hashtag | 200910 | 1 | Mycoolwife |
| hashtag | 200912 | 500 | mycoolwife |
| hashtag | 200905 | 23 | abc |
| hashtag | 200907 | 1000 | abc |
Using the twitter data set downloaded from LMS please perform the following.
a) [Spark] Find the single row that has the highest count and for that row report the month, count and hashtag name. So for the above small example data set the result would be:
Month: 200907, count: 1000, hash tag name: abc
b) [Do twice, once using Hive and once using Spark] Find the hash tag name that has tweeted the most in the entire data set. Report the total number of tweets for that hash tag name. So for the above small example data set the output would be:
abc 1023
c) [MapReduce] Given two months x and y, where y > x. Find the hashtag name that has increased the number of tweets the most from month x to month y. Ignore the tweets in the months between x and y, so just compare the number of tweets at month x and at month y. Report the hashtag name, the number of tweets in months x and y. Ignore any hashtag names that had no tweets in either month x or y. You can assume that the combination of hashtag and month is unique. Therefore, the same hashtag and month combination cannot occur more than once. Take x and y as command line arguments as was done in Task D of Lab 3. For the above small example data set:
Input: x = 200910, y = 200912
Output (hashtag, month x count, month y count):
mycoolwife, 1, 500
Appendix A: Working with Linux
As 3rd/4th year students of a computer science subject, we assume that you know how to use Linux. If you have not used Linux before, this is a good opportunity to gain hands-on experience. After startup, the VM will automatically log you in and load into a graphical environment. However, you may find it useful to work with the command shell (=bash, =terminal) sometimes.
It is possible to complete the entire assignment using just the tools currently installed in the VM without installing any other tools. Anyway, your user “cloudera” (password “cloudera”) possesses all necessary rights for downloading and installing additional application packages.
Appendix B: Transferring files between your host system and the Cloudera VM
The following information is useful particularly if your drag n drop is not working correctly.
VirtualBox: If you create a folder on your host system and mark is as a shared in the settings of your VirtualBox application.
You can mount it to the folder ~/Public of your Linux guest operating system using the following command:
sudo mount -t vboxsf Temp ~/Public