Distributed System Map Reduce Using Software provided by Department

Given an URL and an integer K as an input, recursively crawl through the URLs and retrieve the
URLs referred in each webpage, which further are scraped. Using this information you will create a
directed graph, where the nodes are the unique URLs visited in your scraping, and an edge exists
from node U to node V, if on scraping U, you find a link to V. (i.e., U refers V).
Please note you don’t have to exhaustively find all the links (ie., some links are embedded etc., even
a one pass “href” collector is okay to use.) But you will have to get a considerable amount of links
from the page. Because it will be important for Q3 too. (You won’t be graded for getting ALL the
links, but have to get a fair amount of them)
The depth K will be ~25 (not strict, if it is too time consuming, we will consider K=15 too. The
code working for a large K isn’t a requirement for grading, depending on whether it works
appropriately grades will be given) and the input URL will be to a relatively simple website. (Not
like Pizza Hut or Amazon with a lot of links)
Example:
(not real, just for understanding)
Input:
https://courses.iiit.ac.in/my/ 2
Expected Run:
On scraping https://courses.iiit.ac.in/my/ (lets call it O) (iteration 1), among other urls, you will find
a reference to a course C. You will also find a reference to an assignment A of the course C. On
recursively scraping for another iteration (iteration 2),
● When you scrape A, you will find a reference to C (as it is the course of that assignment),
● And on scraping C, you will find a reference to A (as it is an assignment in the course).
Therefore a part of the graph here will be (listing out the edges from Node X to Node Y):
OC
OA
AC
CA
There will be more edges other than this, but this is for understanding of what output is expected.
The output will be a text file with the first line being two integers, Number of Nodes (URLS) and
Number of Edges (unique references, a webpage may be referred more than once, but it should only
be counted once), with a whitespace between them:
10 34
O C
O A
A C
C A
…. 30 more lines denoting references