Hadoop代写:CSE6232 Hadoop, Spark, Pig and Azure

代写四个关于大数据的作业,涉及到Hadoop, Spark和Pig的使用,最后需要在AWS和Azure上搭建环境进行计算。

Task 1: Analyzing a Graph with Hadoop/Java

Writing your first simple Hadoop program

Imagine that your boss gives you a large dataset which contains an entire email communication network from a popular social network site. The network is organized as a directed graph where each node represents an email address and the edge between two nodes (e.g., Address A and Address B) has a weight stating how many times A wrote to B. The boss is very interested in finding out the people most frequently contacted by others. Your task is to write a MapReduce program in Java to report the largest weight among all the weighted inbound edges for each node in the graph.

First, go over the Hadoop word count tutorial to get familiar with Hadoop and some Java basics. You will be able to complete this task with only some knowledge about Java. You should have already loaded two graph files into HDFS and loaded into your HDFS file system in your vm. Each file stores a list of edges as tab-separated-values. Each line represents a single edge consisting of three columns: (source node ID, target node ID, edge weight), each of which is separated by a tab (\t). Node IDs are positive integers, and weights are also positive integers. Edges are ordered randomly.

src  tgt  weight
117  51   1
194  51   1
299  51   3
230  151  51
194  151  79
51   130  10

Your code should accept two arguments upon running. The first argument (​args[0]) will be a path for the input graph file on HDFS (e.g., /user/cse6242/graph1.tsv), and the second argument (​args[1]) will be a path for output directory on HDFS (e.g., /user/cse6242/task1output1). The default output mechanism of Hadoop will create multiple files on the output directory such as part-00000, part-00001, which will be merged and downloaded to a local directory by the supplied run script. Please use the run scripts for your convenience.

The format of the output should be such that each line represents a node ID and the largest weight among all its inbound edges. The ID and the largest weight must be separated by a tab (\t). Lines do not need be sorted. The following example result is computed based on the toy graph above. Please exclude nodes that do not have incoming edges (e.g., those email addresses that never get contacted by anybody).

For the toy graph above, the output is as follows.

51   3
151  79
130  10

Test your program on graph1.tsv and graph2.tsv. To demonstrate how your MapReduce procedure works, ​use the inline example above, trace the input and output of your map and reduce functions. That is, given the above graph as the input, describe the input and output of your map and reduce function(s) and how the functions transform/process the data (provide examples whenever appropriate). Write down your answers in ​description.pdf​. You are welcome to explain your answers using a combination of text and images.

Designing a MapReduce algorithm (and thinking in MapReduce)

Design a MapReduce algorithm that accomplishes the following task: for each node i in a directed graph G, find that node’s in neighbors’ in neighbors​. Node u is considered to be an in neighbor of node v if there is a directed edge pointing from node u to node v. In other words, your task is find every “2-hop” neighbor of every node i in the graph G, where such a neighbor is connected by at least one directed path of length 2 that reaches node i.

NOTE: You only need to submit pseudo code, a brief explanation of your algorithm, and trace of input and output of your map and reduce functions for the graph given below. No coding is required​.

Task 2: Analyzing a Large Graph with Spark/Scala

Please go over this ​Spark word count tutorial​ to get more background about Spark/Scala.

Goal

Your task is to calculate the gross accumulated node weights for each node in graph1.tsv and graph2.tsv from edge weights using Spark and Scala. Assume the graph to be a representation of a network flow where each edge represents the number of items flowing from source to target. The gross accumulated node weight for a node is now defined as the number of items produced/consumed by the node.

When loading the edges, parse the edge weights using the ​toInt method and filter out (ignore) all edges whose edge weights equal 1 i.e., only consider edges whose edge weights do not equal 1.

Your Scala program should handle the same two arguments as in Task 1 for input and output from the console, and should generate the same formatted output file on the supplied output directory (tab-separated-file). Please note that the default Spark saveastextfile method uses a saving format that is different from Hadoop’s, so you need to format the result before saving to file (Tip: use map and mkString). The result doesn’t need to be sorted.

Task 3: Analyzing Large Amount of Data with Pig on AWS

You will try out PIG (​http://pig.apache.org​) for processing n-gram data on Amazon Web Services (AWS). This is a fairly simple task, and in practice you may be able to tackle this using commodity computers (e.g., consumer-grade laptops or desktops). However, we would like you to use this exercise to learn and solve it using distributed computing on Amazon EC2, and gain experience (very helpful for your future career in research or industry), so you are prepared to tackle more complex problems.

The services you will primarily be using are Amazon S3 storage, Amazon Elastic Cloud Computing (EC2) virtual servers in the cloud, and Amazon Elastic MapReduce (EMR) managed Hadoop framework.

This task will ideally​ use up​ only a very small fraction of your $100 credit​. AWS allows you to use up to 20 instances in total (that means 1 master instance and up to 19 core instances) without filling out a “limit request form”. For this assignment, you should not exceed this quota of 20 instances​. You can learn about these instance types, their specs, and pricing at Instance Types​.

Please read the AWS Setup Guidelines provided to set up your AWS account. In this task, you will use subsets of the Google books n-grams dataset (full dataset for reference), on which you will perform some analysis. An ‘n -gram’ is a phrase with n words; the full n-gram dataset lists n-grams present in the books on books.google.com along with some statistics.

You will perform your analysis on two custom datasets, extracted from the Google books bigrams (2-grams), that we have prepared for you: a small one and a large one. To help you evaluate the correctness of your output, we have uploaded the output for the small dataset on T-Square (the link is here​ ).

VERY IMPORTANT​ : Both these datasets are in the US-Standard (US-East) region. Using machines in other regions for computation would incur data transfer charges. Hence, set your region to US East (N. Virginia) in the beginning (not Oregon which is the default). This is extremely important otherwise your code may not work and you may be charged extra.

Goal

For each unique bigram, compute its average number of appearances per book, with at least 50 occurrences for each recorded year. For the above example, the results will be:

I am       (342 + 211) / (90 + 10) = 5.53
very cool  (500 + 3210 + 9994) / (10 + 1000 + 3020) = 3.40049628

Output the 10 bigrams having the highest average number of appearances per book along with their corresponding averages, in tab-separated format​, sorted in descending order, with at least 50 occurrences for each recorded year. If multiple bigrams have the same average, order them alphabetically. For the example above, the output will be:

I am       5.53
very cool  3.40049628

You will solve this problem by writing a PIG script on Amazon EC2 and save the output.

You can use the interactive PIG shell provided by EMR to perform this task from the command line (grunt). In this case, you can copy the commands you used for this task into a single file to have the PIG script and the output from the command line into a separate file. Please see this for how to use PIG shell. Also, you can upload the script and create a task on your cluster.

Task 4: Analyzing a Large Graph using Hadoop service onMicrosoft Azure

Goal

Your task is to write a MapReduce program to calculate the degree distribution of a graph. Note that this task shares some similarities with Task 1 (e.g., both are analyzing graphs). Task 1 can be completed using your own computer. This task is to be completed using Azure. We recommend that you first complete Task 1.

You will use data files small.tsv(~75MB) and large.tsv(~3GB), for this question. Each file stores a list of edges as tab-separated-values. Each line represents a single edge consisting of two columns: (Node A, Node B), each of which is separated by a tab. Node IDs are positive integers and the rows are already sorted by Node A.

src  tgt
51   130
51   194
51   299
130  200
151  230
151  194

Your code should accept two arguments upon running. The first argument (args[0]) will be a path for the input graph file, and the second argument (args[1]) will be a path for output directory. The default output mechanism of Hadoop will create multiple files on the output directory such as part-00000, part-00001, which will have to be merged and downloaded to a local directory.

The format of the output should be as follows. Each line represents the degree and its frequency. The degree and the frequency of the degree must be separated by a tab(\t), and lines don’t have to be sorted. The following example result is computed based on the toy graph above.

Hint​: One way of doing it is using mapreduce procedure twice. First for finding the degree of each node and second for calculating the frequency of each degree. You will have to make appropriate changes in the skeleton code for this.