Cross-Platform Web & desktop PDF Scraper

I’m building a desktop application that lets me harvest information from two sources with the same, seamless workflow:

• Logged-in web pages already open in Google Chrome, BRAVE , Mozilla Firefox
• Local or network PDFs that I choose from an “Open File” dialog or that are currently opened on the desktop

For websites, I need the tool to capture text and table content, and in some instances download linked PDFs linked to each record. From PDFs themselves I only care about the text layer. All captured data has to flow into a neatly structured Excel workbook (one row per record, tidy column mapping) while the corresponding PDF files are stored in a folder bearing the same name as the spreadsheet for easy reference.

Key points on functionality
– Runs natively on both Windows and Linux without asking the user to install extra dependencies beyond a standard runtime (for example, a Python one-file executable or an Electron package—open to your preference).
– Detects that I’m already authenticated in Chrome and simply works with the live session; no separate login logic necessary.
– Simple, intuitive GUI: select “Scrape Web”, “Scrape PDF”, choose output directory, press Start, see a progress bar and a final summary of how many rows and PDFs were saved.
– Robust parsing of website tables (including pagination) and text blocks; graceful handling of captchas or timeouts would be a plus.
– Excel output in .xlsx with auto-generated column headers and basic formatting.
– Source code provided so I can tweak selectors later, plus a brief read-me explaining how to add new sites or PDF patterns.

Acceptance criteria
1. I can run the packaged application on Windows 10/11 and Ubuntu 22.04, click through the GUI, and produce an .xlsx file plus matching PDFs.
2. Website extraction works on at least two sample sites we will define during testing, while logged in through Chrome.
3. PDF extraction correctly writes the text from a sample set (scanned or image-only PDFs can be skipped).
4. No data duplication and no crashes after a several hour continuous scrape.

________________________________________________________________________________
Program Specification: Automated Data Scraper with PDF Handling: an interactive data extraction tool with a GUI and automation workflow

Develop a cross-platform program (Windows & Linux desktop compatible) that automates data collection from websites that are logged in and open in the browser and alternatively also can scrape and collect from any selected/ desktop opened PDFs, exporting the results into structured Excel spreadsheets, while also saving associated PDFs for each data record.
Core Features
1. User-Controlled Start/Stop
- Program has a GUI with an ON/OFF “Start/Stop” button. Also an “automatic mode that will run for a specified period of time on a specific site or document timer can be set for 0.1 hours up to 12.0 hours maximum (and if it runs out of data or time, it automatically saves and closes.
- When ON:
User can direct the program to a specific website. (Log in if needed prior to “start”)
Program begins crawling/parsing data sequentially. (Can automatically click links only to search & collect data, not execute programs)

- When OFF:
Current process stops.
Current Excel file is saved and named automatically.
2. Data Collection Workflow
-Program reads through line items (e.g., rows, listings, or links) on the given web page.
-For each line item:

1. Collect textual data from the web page (structured fields).
2. If a link opens a PDF, automatically click/follow it.
3. Download and save the PDF locally.
4. Parse PDF contents (text, tables, or metadata).
5. Insert all data into a structured Excel row.
6. Associate the saved PDF with that Excel row (e.g., file path reference).

3. Sequential Processing
-After handling one line item and its PDF, move on to the next line item.
-Repeat until all items on the page are processed.
4. Excel Output
- Each run creates a new Excel file.
- File is automatically named (e.g., Dataset_<Date>_<Time>.xlsx).
- Columns should include:
Line Item ID / Name
Extracted fields from webpage
Extracted fields from PDF
Local PDF file path

5. PDF Handling
- Save a local copy of each PDF with a systematic filename (e.g., LineItem123.pdf).
- Store PDFs in a dedicated folder per run, linked to the Excel output.

6. Multiple Sessions
If the program is directed to a different website or a standalone PDF:
It should start a new Excel file for that session.
Store related PDFs in a new folder tied to that session.
7. Error Handling
If a link is broken or PDF fails to load:
Log error in the Excel file under the line item.
Continue with next item.
If the program is manually stopped, partial data must still be saved.
________________________________________
Technical Notes for Developer
• Platform Compatibility
Must run on Windows desktop (primary).
Android compatibility is optional (possible via porting to Kotlin or running Python in Termux/Pydroid).
• Suggested Languages & Frameworks
Python (recommended for prototyping):
Crawling → requests, BeautifulSoup, Scrapy, or Selenium (if dynamic JavaScript).
PDF parsing → pdfplumber, PyPDF2.
Excel export → pandas, openpyxl.
GUI → Tkinter, PyQt, or Kivy (Kivy would also allow Android portability).
o Java/Kotlin (for future Android-native app):
Crawling → Jsoup.
PDF parsing → Apache PDFBox.
Excel → Apache POI.
GUI → JavaFX (desktop), Android native UI (mobile).
• Architecture
o Frontend: GUI with start/stop buttons, site/PDF input field.
o Backend: Crawling/parsing engine that processes URLs, extracts data, and writes to Excel.
o Storage:
Excel files auto-saved per session.
PDFs stored in session-specific folders.
________________________________________
Example Workflow
1. User opens program.
2. Enters target website URL (or single PDF opened on desktop).
3. Clicks Start.
4. Program begins:
Reads each line item on site. Scrolls as needed.
Opens/downloads linked PDFs.
Extracts and parses text.
Saves associated PDF in a folder linked to any excel line items accordingly.
Records results in Excel.
5. User clicks Stop.
6. Program saves and names Excel file (Dataset_2025-08-30_1450.xlsx).
7. Session complete.

_________________________________________________________________________________________________

If you’ve built similar Selenium, BeautifulSoup, Puppeteer, PDFMiner, or PyMuPDF solutions before, I’d love to see them. Looking forward to your approach and an estimated timeline for a first working prototype.