zvg scraper
Budget: €250 – €750 EUR
Here are 3 screenshots of the website. it is a website that lists real estate foreclosures in germany. the first one (1) is the search mask https://www.zvg-portal.de/index.php?button=Termine%20suchen where you have to select a federal state. I am interested in all foreclosures of all german federal states. after selecting one state, you receive a list of real estates (2) if you click on details you receive the page with the actual pdfs (3). I would like to have all pdfs of a foreclosure downloaded in a specific folder.
1. Technical Analysis of the Website Technologies and Frameworks The ZVG portal is operated by the state judicial administrations and hosted by IT.NRW. The HTTP headers indicate that the server uses Apache along with PHP-based scripts (for example, paths like template.internet.showZvg.php appear in the robots.txt). The pages are delivered in ISO-8859-1 encoding, which points to a more classic web application design. No modern front-end frameworks (such as Angular or React) are evident; instead, the site relies on traditional HTML forms, table layouts, and complete server-side rendering. JavaScript is used only sparingly (for instance, for handling forms), and there is no single-page application behavior. Provision of Search Results (Server-Side vs. AJAX) The list of auction results is generated server-side and delivered as a complete HTML page. Users fill out a search form (e.g. selecting a federal state, district court, etc.) and submit it. The form data is sent via an HTTP POST to the server, which then returns the matching results as an HTML page. For example, to retrieve all dates in a specific state like Schleswig-Holstein, a POST request using parameters such as ger_id=0 and ger_name="-- Alle Amtsgerichte --" is sent, and the server responds with a page containing a table of matching entries. Since the results are provided as fully rendered HTML (with tables that sometimes span multiple rows per entry), there is no reliance on AJAX to load additional data dynamically. This means that every search results in a full page reload, simplifying HTML parsing for automated tools. Provision of PDF Documents On the detail page of an auction, PDF documents (such as appraisal reports) are often listed as attachments. These PDFs are not accessed via static file URLs; instead, they are delivered through a server-side script. In practice, calls are made to URLs like index.php?button=showAnhang&file_id=X&zvg_id=Y&land_abk=... which stream the PDF content with the correct Content-Type: application/pdf. Consequently, when a user clicks on a PDF link, the browser either displays the document using its built-in PDF viewer or offers it for download. There is no special in-page PDF viewer (e.g. PDF.js) integrated—modern browsers handle the display natively. HTTP Requests and Required Headers for PDF Retrieval PDF retrieval is handled via an HTTP GET to the showAnhang URL. A key requirement is the Referer header. The server expects the Referer to be the corresponding detail page (a “showZvg” URL). If this header is missing or incorrect, the request will be rejected or yield an empty/erroneous response (users have reported that copying and pasting the URL directly results in an error). Apart from the Referer, no unusual headers are needed. However, it is advisable to send a standard User-Agent header since the server might scrutinize requests from non-browser clients. Although session cookies (like a PHPSESSID) might be issued, they do not appear to be essential for simply reading the documents—what is crucial is the Referer validation. Access Restrictions and Security Mechanisms Even though the ZVG portal is publicly accessible, there are technical measures to restrict automated or direct access. Without following the proper sequence (search → detail → attachment), no data is returned because the server checks the Referer to ensure that, for instance, detail pages are only accessed from the search page and PDFs only from the detail page. This Referer check acts as a simple security mechanism (akin to a reverse CSRF measure). There are no other strong protection mechanisms such as CAPTCHAs or login requirements. However, the robots.txt disallows crawling of certain paths (like the detail and attachment pages), and standard security headers (e.g. X-Frame-Options: SAMEORIGIN, X-XSS-Protection) are in place. In summary, although there is no explicit authentication, the combination of Referer dependency (and possibly session cookies) ensures that the site is used in the intended flow. 2. Automation Possibilities for Downloading PDFs Extracting Search Results To retrieve the search results automatically, you can mimic the HTTP request made by the search form. This involves: • Replicating the Form Submission: Identify the required form parameters (such as the federal state via land_abk, district court via ger_id and ger_name, sorting options, and any additional criteria like partial file numbers or address fields) and send them in a POST request to index.php?button=Suchen. • Parsing the Returned HTML: The server returns a complete HTML page containing the search results in a table. Libraries like BeautifulSoup (in Python) can be used to parse this HTML and extract the rows, which include links to the detail pages (typically formatted as <a> tags in the table cells). Since the response is fully rendered in HTML (in ISO-8859-1 encoding), automated extraction can be performed without needing to simulate a full browser—simple HTTP clients (such as Python’s requests) suffice. Automating the PDF Download Process For each auction entry, the automation workflow can follow these steps: 1. Obtain Detail Page URL: From the parsed search results, extract the URL of the auction’s detail page (usually a “showZvg” URL). 2. Request the Detail Page: Issue a GET request to the detail page. Important: Set the Referer header to the URL of the search results page to satisfy the server’s check. 3. Extract PDF Links: On the detail page, locate the PDF attachment links. These links follow a pattern like index.php?button=showAnhang&file_id=X&zvg_id=Y&land_abk=... and are embedded in the HTML as clickable elements. 4. Download the PDF: For each PDF link, send an HTTP GET request while setting the Referer header to the detail page URL. If the site uses session cookies, use a persistent HTTP session (e.g. with Python’s requests.Session) so that cookies are automatically managed. The server should then return the PDF content, which can be saved locally as a binary file. Potential Challenges • Referer Dependency: The server enforces that requests for detail pages and PDFs include the appropriate Referer. Simply using tools like wget without setting this header will result in failures. • HTML Parsing Complexity: The search result tables may span multiple rows per entry with missing fields represented by empty cells. This requires robust parsing logic. • Rate Limiting: Although there are no explicit IP bans or CAPTCHAs, making too many requests too quickly might trigger server-side rate-limiting. It is advisable to add delays between requests. • Encoding Considerations: Since the HTML is in ISO-8859-1, ensure that your script correctly handles the character encoding to properly parse non-ASCII characters. Recommended Tools and Methods • HTTP Client & HTML Parser: Use Python’s requests library in combination with BeautifulSoup. This method offers full control over request headers (including Referer) and is efficient because it does not require loading a full browser. • Browser Automation Tools (e.g. Selenium): If you prefer to simulate the actual user workflow (filling out forms, clicking links), you can use Selenium with a headless browser (such as Chrome or Firefox). Selenium automatically manages cookies and Referer headers, but this approach is more resource-intensive and generally slower. • Scraping Frameworks: Frameworks like Scrapy (in Python) can simplify the process by managing cookies, headers, and link-following automatically. They also offer built-in throttling to avoid overwhelming the server. • Existing Libraries: Given that the ZVG portal is a known target for data enthusiasts, you might find existing GitHub projects or scripts that already implement the required steps. These can be used as a starting point to save development time. In Summary The ZVG portal uses a classic, server-side approach to render search results and detail pages. PDFs are served through a scripted endpoint that requires a proper Referer header, meaning that any automated tool must replicate the complete flow (from search to detail to PDF download) and correctly set HTTP headers (and possibly manage cookies). With careful handling of these requirements—using tools like Python’s requests and BeautifulSoup, or Selenium for a more complete browser simulation—it is entirely feasible to automate the retrieval of the provided PDFs
1. Technical Analysis of the Website Technologies and Frameworks The ZVG portal is operated by the state judicial administrations and hosted by IT.NRW. The HTTP headers indicate that the server uses Apache along with PHP-based scripts (for example, paths like template.internet.showZvg.php appear in the robots.txt). The pages are delivered in ISO-8859-1 encoding, which points to a more classic web application design. No modern front-end frameworks (such as Angular or React) are evident; instead, the site relies on traditional HTML forms, table layouts, and complete server-side rendering. JavaScript is used only sparingly (for instance, for handling forms), and there is no single-page application behavior. Provision of Search Results (Server-Side vs. AJAX) The list of auction results is generated server-side and delivered as a complete HTML page. Users fill out a search form (e.g. selecting a federal state, district court, etc.) and submit it. The form data is sent via an HTTP POST to the server, which then returns the matching results as an HTML page. For example, to retrieve all dates in a specific state like Schleswig-Holstein, a POST request using parameters such as ger_id=0 and ger_name="-- Alle Amtsgerichte --" is sent, and the server responds with a page containing a table of matching entries. Since the results are provided as fully rendered HTML (with tables that sometimes span multiple rows per entry), there is no reliance on AJAX to load additional data dynamically. This means that every search results in a full page reload, simplifying HTML parsing for automated tools. Provision of PDF Documents On the detail page of an auction, PDF documents (such as appraisal reports) are often listed as attachments. These PDFs are not accessed via static file URLs; instead, they are delivered through a server-side script. In practice, calls are made to URLs like index.php?button=showAnhang&file_id=X&zvg_id=Y&land_abk=... which stream the PDF content with the correct Content-Type: application/pdf. Consequently, when a user clicks on a PDF link, the browser either displays the document using its built-in PDF viewer or offers it for download. There is no special in-page PDF viewer (e.g. PDF.js) integrated—modern browsers handle the display natively. HTTP Requests and Required Headers for PDF Retrieval PDF retrieval is handled via an HTTP GET to the showAnhang URL. A key requirement is the Referer header. The server expects the Referer to be the corresponding detail page (a “showZvg” URL). If this header is missing or incorrect, the request will be rejected or yield an empty/erroneous response (users have reported that copying and pasting the URL directly results in an error). Apart from the Referer, no unusual headers are needed. However, it is advisable to send a standard User-Agent header since the server might scrutinize requests from non-browser clients. Although session cookies (like a PHPSESSID) might be issued, they do not appear to be essential for simply reading the documents—what is crucial is the Referer validation. Access Restrictions and Security Mechanisms Even though the ZVG portal is publicly accessible, there are technical measures to restrict automated or direct access. Without following the proper sequence (search → detail → attachment), no data is returned because the server checks the Referer to ensure that, for instance, detail pages are only accessed from the search page and PDFs only from the detail page. This Referer check acts as a simple security mechanism (akin to a reverse CSRF measure). There are no other strong protection mechanisms such as CAPTCHAs or login requirements. However, the robots.txt disallows crawling of certain paths (like the detail and attachment pages), and standard security headers (e.g. X-Frame-Options: SAMEORIGIN, X-XSS-Protection) are in place. In summary, although there is no explicit authentication, the combination of Referer dependency (and possibly session cookies) ensures that the site is used in the intended flow. 2. Automation Possibilities for Downloading PDFs Extracting Search Results To retrieve the search results automatically, you can mimic the HTTP request made by the search form. This involves: • Replicating the Form Submission: Identify the required form parameters (such as the federal state via land_abk, district court via ger_id and ger_name, sorting options, and any additional criteria like partial file numbers or address fields) and send them in a POST request to index.php?button=Suchen. • Parsing the Returned HTML: The server returns a complete HTML page containing the search results in a table. Libraries like BeautifulSoup (in Python) can be used to parse this HTML and extract the rows, which include links to the detail pages (typically formatted as <a> tags in the table cells). Since the response is fully rendered in HTML (in ISO-8859-1 encoding), automated extraction can be performed without needing to simulate a full browser—simple HTTP clients (such as Python’s requests) suffice. Automating the PDF Download Process For each auction entry, the automation workflow can follow these steps: 1. Obtain Detail Page URL: From the parsed search results, extract the URL of the auction’s detail page (usually a “showZvg” URL). 2. Request the Detail Page: Issue a GET request to the detail page. Important: Set the Referer header to the URL of the search results page to satisfy the server’s check. 3. Extract PDF Links: On the detail page, locate the PDF attachment links. These links follow a pattern like index.php?button=showAnhang&file_id=X&zvg_id=Y&land_abk=... and are embedded in the HTML as clickable elements. 4. Download the PDF: For each PDF link, send an HTTP GET request while setting the Referer header to the detail page URL. If the site uses session cookies, use a persistent HTTP session (e.g. with Python’s requests.Session) so that cookies are automatically managed. The server should then return the PDF content, which can be saved locally as a binary file. Potential Challenges • Referer Dependency: The server enforces that requests for detail pages and PDFs include the appropriate Referer. Simply using tools like wget without setting this header will result in failures. • HTML Parsing Complexity: The search result tables may span multiple rows per entry with missing fields represented by empty cells. This requires robust parsing logic. • Rate Limiting: Although there are no explicit IP bans or CAPTCHAs, making too many requests too quickly might trigger server-side rate-limiting. It is advisable to add delays between requests. • Encoding Considerations: Since the HTML is in ISO-8859-1, ensure that your script correctly handles the character encoding to properly parse non-ASCII characters. Recommended Tools and Methods • HTTP Client & HTML Parser: Use Python’s requests library in combination with BeautifulSoup. This method offers full control over request headers (including Referer) and is efficient because it does not require loading a full browser. • Browser Automation Tools (e.g. Selenium): If you prefer to simulate the actual user workflow (filling out forms, clicking links), you can use Selenium with a headless browser (such as Chrome or Firefox). Selenium automatically manages cookies and Referer headers, but this approach is more resource-intensive and generally slower. • Scraping Frameworks: Frameworks like Scrapy (in Python) can simplify the process by managing cookies, headers, and link-following automatically. They also offer built-in throttling to avoid overwhelming the server. • Existing Libraries: Given that the ZVG portal is a known target for data enthusiasts, you might find existing GitHub projects or scripts that already implement the required steps. These can be used as a starting point to save development time. In Summary The ZVG portal uses a classic, server-side approach to render search results and detail pages. PDFs are served through a scripted endpoint that requires a proper Referer header, meaning that any automated tool must replicate the complete flow (from search to detail to PDF download) and correctly set HTTP headers (and possibly manage cookies). With careful handling of these requirements—using tools like Python’s requests and BeautifulSoup, or Selenium for a more complete browser simulation—it is entirely feasible to automate the retrieval of the provided PDFs