Fix problems with simple HTTP scraping project, created using .NET Framework Console. -- 2

Job ID: 32043882

Budget: £20 – £250 GBP

We have a simple .NET Framework Console program to scrape several websites. The project is functional. However, it has numerous intermittent issues, including:
• Connection timeout issues
• HTTP forwarding issues
• SSL connection issues
• Incomplete downloads

In addition to solving these reliability problems, we will require:
• A belief explanation of how they were solved
• Any follow-up questions to be answered

Project
----------------------------------------------------------------------------------
Implemented steps are:
1. Downloads sitemap (links to the sitemaps are embedded in project)
2. Loops sitemap URLs
3. For each URL, either
a. Performs HTTP GET and download HTML page
or
b. Performs HTTP POST (with embedded details)
4. Saves HTML or JSON responses to a file

Notes
----------------------------------------------------------------------------------
• All requests are to go via a proxy (connection details provided on award)
• We require an ongoing relationship with someone with scrapping experience

Platform
----------------------------------------------------------------------------------
• Type: .NET Framework Console Application
• Language: C# or VB
Related categories: .NET Web Scraping HTTP