I am looking for python developer
Budget: $250 – $750 USD
My tasks will involve REAL-TIME scraping (not historical) with the fastest possible methods (keeping cost efficiency in mind) to achieve the lowest latency scraping. As this can be across many different sites, I will give an example below.
https://www.businessinsider.com/ajax/content-api/vertical?templateId=river&id=homepage&page[limit]=20
Get all data in real-time (posts/articles) and post output in telegram bot maybe (or anything else).
https://www.marketwatch.com/latest-news?messageNumber=0&channelId=e0893c81-6641-413b-a450-25c14139ffd0&position=1.2&partial=true
Do BOTH sites. I want to see output in real time. If can complete successfully you are hired.
Remember for some sites there will be blocks and protection. Your goal is to successfully bypass. Use proxies (I’ll provide some free sources below but pls research best ones to use - once I hire you I will have my own IPs you may use) use multiple accounts, different fingerprints etc. the whole goal here is to get data as FAST as possible (even around 100ms).
Make sure we can compare with time published and time received (and with time received it has little to no gap in between the time published in API after you scrape).
Note: Don't want past data I want to built a bot to get NEW data as fast as possible immediately on any NEW set of data coming in
Will need to create a bot to display output in real-time.
Let's discuss more!
Thanks!
https://www.businessinsider.com/ajax/content-api/vertical?templateId=river&id=homepage&page[limit]=20
Get all data in real-time (posts/articles) and post output in telegram bot maybe (or anything else).
https://www.marketwatch.com/latest-news?messageNumber=0&channelId=e0893c81-6641-413b-a450-25c14139ffd0&position=1.2&partial=true
Do BOTH sites. I want to see output in real time. If can complete successfully you are hired.
Remember for some sites there will be blocks and protection. Your goal is to successfully bypass. Use proxies (I’ll provide some free sources below but pls research best ones to use - once I hire you I will have my own IPs you may use) use multiple accounts, different fingerprints etc. the whole goal here is to get data as FAST as possible (even around 100ms).
Make sure we can compare with time published and time received (and with time received it has little to no gap in between the time published in API after you scrape).
Note: Don't want past data I want to built a bot to get NEW data as fast as possible immediately on any NEW set of data coming in
Will need to create a bot to display output in real-time.
Let's discuss more!
Thanks!