PDF Preprocessing & Chunking for LLMs/ML (Micro Project)
Budget: $10 – $30 USD
I need a PDF preprocessing expert to help me with providing a general purpose function for intelligent chunking of the PDF documents suitable for input to LLM.
Note: Low budget, beginners with pdf preprocessing & chunking expertise preffered.
Key Requirements:
- OCR the pdf, if not ocred.
- Preprocess PDFs: Clean and refine PDF data for readability and efficient handling by language models (LLMs) and machine learning (ML) models.
- Chunking Strategy: Implement an intelligent chunking method based on chapters, headings, and subheadings. The goal is to create smaller, more manageable sections that can be effectively processed by NLP models & is not split in between.
- Token size of variable size. The function should accept an integer along with pdf file. The integer value is the token size. Experiment with various sizes of 128 to 32000 token size.
- Preferably, provide Original Formatting: maintain the original formatting of the text throughout the process. This ensures that the resulting chunks align with the structure of the original document. So the headings and subheadings are seggregated.
- The function should run on all the pdfs generally available, especially technical books.
Deliverables:
- function to generate Organized Chunks: The function should return a df consisting of of intelligently chunked sections based on the provided PDFs. 'Page#, Book name, Author, Edition, Chapter#, Chapter Name, Contents, Heading, SubHeading, Header, Footer' for each chunk.
- Suitable Format: Preferably the chunks in the df['Contents'] have html tags for bold / bullets / Italics / underline etc
Ideal Candidate:
- Experience with PDF preprocessing and data cleaning
- Familiarity with chunking strategies for text data for NLP and ML model requirements
- Previous experience working with LLMs would be a significant advantage.
Note: Low budget, beginners with pdf preprocessing & chunking expertise preffered.
Key Requirements:
- OCR the pdf, if not ocred.
- Preprocess PDFs: Clean and refine PDF data for readability and efficient handling by language models (LLMs) and machine learning (ML) models.
- Chunking Strategy: Implement an intelligent chunking method based on chapters, headings, and subheadings. The goal is to create smaller, more manageable sections that can be effectively processed by NLP models & is not split in between.
- Token size of variable size. The function should accept an integer along with pdf file. The integer value is the token size. Experiment with various sizes of 128 to 32000 token size.
- Preferably, provide Original Formatting: maintain the original formatting of the text throughout the process. This ensures that the resulting chunks align with the structure of the original document. So the headings and subheadings are seggregated.
- The function should run on all the pdfs generally available, especially technical books.
Deliverables:
- function to generate Organized Chunks: The function should return a df consisting of of intelligently chunked sections based on the provided PDFs. 'Page#, Book name, Author, Edition, Chapter#, Chapter Name, Contents, Heading, SubHeading, Header, Footer' for each chunk.
- Suitable Format: Preferably the chunks in the df['Contents'] have html tags for bold / bullets / Italics / underline etc
Ideal Candidate:
- Experience with PDF preprocessing and data cleaning
- Familiarity with chunking strategies for text data for NLP and ML model requirements
- Previous experience working with LLMs would be a significant advantage.