Fix problems with simple HTTP scraping project, created using .NET Framework Console. -- 2
Budget: £20 – £250 GBP
We have a simple .NET Framework Console program to scrape several websites. The project is functional. However, it has numerous intermittent issues, including:
• Connection timeout issues
• HTTP forwarding issues
• SSL connection issues
• Incomplete downloads
In addition to solving these reliability problems, we will require:
• A belief explanation of how they were solved
• Any follow-up questions to be answered
Project
----------------------------------------------------------------------------------
Implemented steps are:
1. Downloads sitemap (links to the sitemaps are embedded in project)
2. Loops sitemap URLs
3. For each URL, either
a. Performs HTTP GET and download HTML page
or
b. Performs HTTP POST (with embedded details)
4. Saves HTML or JSON responses to a file
Notes
----------------------------------------------------------------------------------
• All requests are to go via a proxy (connection details provided on award)
• We require an ongoing relationship with someone with scrapping experience
Platform
----------------------------------------------------------------------------------
• Type: .NET Framework Console Application
• Language: C# or VB
• Connection timeout issues
• HTTP forwarding issues
• SSL connection issues
• Incomplete downloads
In addition to solving these reliability problems, we will require:
• A belief explanation of how they were solved
• Any follow-up questions to be answered
Project
----------------------------------------------------------------------------------
Implemented steps are:
1. Downloads sitemap (links to the sitemaps are embedded in project)
2. Loops sitemap URLs
3. For each URL, either
a. Performs HTTP GET and download HTML page
or
b. Performs HTTP POST (with embedded details)
4. Saves HTML or JSON responses to a file
Notes
----------------------------------------------------------------------------------
• All requests are to go via a proxy (connection details provided on award)
• We require an ongoing relationship with someone with scrapping experience
Platform
----------------------------------------------------------------------------------
• Type: .NET Framework Console Application
• Language: C# or VB