Write the MapReduce of Analyzing a Graph with Hadoop/Java using Go Language.
Budget: $8 – $15 USD
Imagine that your boss gives you a large dataset which contains an entire email communication network from a popular social network site. The network is organized as a directed graph where each node represents a person’s email address and an edge between two nodes (e.g., address A and address B) has a weight stating how many times A has written to B. You have been tasked with finding the person that each person has written to the most, along with that count (see the example below for more clarification).YourtaskistowriteaMapReduceprograminJavatoreport,foreachnodeX(the“source”, or “src” for short) in the graph, the person Y (the “target” or “tgt” for short) that X has written to the most, and the number of times X has written to Y (the outbound “weight”, f rom X to Y). If a person has written tomultipletargetsthathaveexactlythesame(largest)numberoftimes, returnthetargetwith the smallest node id.
First, go over the Hadoop word count tutorial to familiarize yourself with Hadoop and some Java basics. You will be able to complete this question with only some knowledge about Java. You should have already loaded two graph files into HDFS and loaded into your HDFS file system in your VM. Each file stores a list of edges as tab-separated-values. Each line represents a single edge consisting of three columns: (source node ID, target node ID, edge weight), each of which is separated by a tab (\t). Node IDs and weights are positive integers. Below is a small toy graph, for illustration purposes (on your screen, the text may appear out of alignment).
src tgt weight
10 110 3
10 200 1
200 150 30
100 110 10
110 130 15
110 200 67
10 70 3
Your program should not assume the edges to be sorted or ordered in any ways (i.e., your program should work even when the edge ordering is random).
Your code should accept two arguments. The first argument (args[0]) will be a path for the input graph fileonHDFS(e.g.,data/graph1.tsv),andthesecondargument(args[1])willbeapath foroutputdirectory on HDFS (e.g., data/q1output1). The default output mechanism of Hadoop will create multiple files on the output directory such as part-00000, part-00001, which will be merged and downloaded to a local directory by the supplied run script. Please use the run1.sh and run2.sh scripts for your convenience.
The format of the output: each line contains a node ID, followed by a tab (\t), and the expected “target node ID,weight” tuple (without the quotes; and note there is no space character before and after the comma). Lines do not need to be sorted. The following example result is computed based on the toy graph above. Please exclude nodes that do not have outgoing edges (e.g., those email addresses which have not sent out any communication).
For the toy graph above, the output is as follows.
10 70,3
200 150,30
100 110,10
110 200,67
First, go over the Hadoop word count tutorial to familiarize yourself with Hadoop and some Java basics. You will be able to complete this question with only some knowledge about Java. You should have already loaded two graph files into HDFS and loaded into your HDFS file system in your VM. Each file stores a list of edges as tab-separated-values. Each line represents a single edge consisting of three columns: (source node ID, target node ID, edge weight), each of which is separated by a tab (\t). Node IDs and weights are positive integers. Below is a small toy graph, for illustration purposes (on your screen, the text may appear out of alignment).
src tgt weight
10 110 3
10 200 1
200 150 30
100 110 10
110 130 15
110 200 67
10 70 3
Your program should not assume the edges to be sorted or ordered in any ways (i.e., your program should work even when the edge ordering is random).
Your code should accept two arguments. The first argument (args[0]) will be a path for the input graph fileonHDFS(e.g.,data/graph1.tsv),andthesecondargument(args[1])willbeapath foroutputdirectory on HDFS (e.g., data/q1output1). The default output mechanism of Hadoop will create multiple files on the output directory such as part-00000, part-00001, which will be merged and downloaded to a local directory by the supplied run script. Please use the run1.sh and run2.sh scripts for your convenience.
The format of the output: each line contains a node ID, followed by a tab (\t), and the expected “target node ID,weight” tuple (without the quotes; and note there is no space character before and after the comma). Lines do not need to be sorted. The following example result is computed based on the toy graph above. Please exclude nodes that do not have outgoing edges (e.g., those email addresses which have not sent out any communication).
For the toy graph above, the output is as follows.
10 70,3
200 150,30
100 110,10
110 200,67
Related categories:
Golang