Web crawler / forum crawler / python / simple statistics summary
Budget: $10 – $30 USD
Overview:
Python script needed that would be able to count occurences of tags in new threads and number of new comments per given tag for a given day. The source is a web forum. Each new thread has a name tag related to it.
Characteristics:
1. The script should be ideally written in Python though I can discuss alternative ways if there are more affordable and easier ways of achieving the goal.
2. Script should allow for parameterization so as of default it should be run daily but there should be an option to specify dates range as well
How it should:
Part 1:
1. Script should crawl through a forum (I'll provide link to interested parties)
2. It creates a list of tags that were created on a given day (script is run in the evening and counts from the beginning of the day). Tag is created for each of the new thread in the forum and there is only one tag per thread.
3. It counts how many tags (new thread per tag) were created on a given day.
4. It returns a list of a tag and count of occurrences: TAG | Number_of_new_threads | Date
Part 2:
1. This could be possibly another script.
2. It checks tags(threads) for a new comments/posts
3. If there is a new comment for a tag then it counts how many comments were posted for a given tag
4. Returns a list/table of TAGS | No_of_new_posts_per_tag | Date
Additional:
At the end I would like to have some sort of documentation - it can be very short but some manual should be at minimum. In the future I would like to have the script being triggered automatically everyday so this should be accounted for - i.e. the parameterization of dates and output in such form that it could be further used as e.g. table posted on a website.
Python script needed that would be able to count occurences of tags in new threads and number of new comments per given tag for a given day. The source is a web forum. Each new thread has a name tag related to it.
Characteristics:
1. The script should be ideally written in Python though I can discuss alternative ways if there are more affordable and easier ways of achieving the goal.
2. Script should allow for parameterization so as of default it should be run daily but there should be an option to specify dates range as well
How it should:
Part 1:
1. Script should crawl through a forum (I'll provide link to interested parties)
2. It creates a list of tags that were created on a given day (script is run in the evening and counts from the beginning of the day). Tag is created for each of the new thread in the forum and there is only one tag per thread.
3. It counts how many tags (new thread per tag) were created on a given day.
4. It returns a list of a tag and count of occurrences: TAG | Number_of_new_threads | Date
Part 2:
1. This could be possibly another script.
2. It checks tags(threads) for a new comments/posts
3. If there is a new comment for a tag then it counts how many comments were posted for a given tag
4. Returns a list/table of TAGS | No_of_new_posts_per_tag | Date
Additional:
At the end I would like to have some sort of documentation - it can be very short but some manual should be at minimum. In the future I would like to have the script being triggered automatically everyday so this should be accounted for - i.e. the parameterization of dates and output in such form that it could be further used as e.g. table posted on a website.