n8n RAG Implementation & Document Chunking
Budget: $30 – $250 USD
I need help with a n8n RAG implementation. Larger documents are failing as I am running out of memory on my n8n cloud instance. I want to do something like this.
Document Chunking While Preserving Identity
The key is to maintain document metadata and relationships between chunks. Here's how to implement this:
Add a Document ID Field: Before splitting, assign a unique document ID to the entire document that will be preserved across all chunks.
Create Position Metadata: For each chunk, include metadata like:
Document ID (to group all chunks)
Chunk sequence number (e.g., 1 of 8, 2 of 8)
Original filename
Total chunks in document
Overlap Content Between Chunks: Include a small overlap between consecutive chunks (50-100 tokens) to maintain context across chunk boundaries.
Tag in Vector Database: When storing in your vector database, ensure all chunks are tagged with the same document identifier.
Retrieval Handling: When retrieving information, you can:
Pull all chunks with the same document ID when the entire document is needed
For queries, return the most relevant chunks but maintain context of which document they belong to
Implementation in n8n
In your n8n workflow, you could:
Add a "Set" node after your file download to generate a document ID and total file size
Add a "Split Binary Data" node to segment large files into manageable chunks
Use a "Loop" node to process each chunk while maintaining document metadata
When inserting into your vector database, ensure the document ID is preserved
This way, your system will treat the chunks as parts of a single document during processing but will avoid the memory issues of handling the entire file at once.
The RAG system will still function properly as long as the embedding and retrieval processes understand that these chunks belong together. During retrieval, relevant chunks can be returned based on the query, but they'll be identified as parts of the same source document.
Document Chunking While Preserving Identity
The key is to maintain document metadata and relationships between chunks. Here's how to implement this:
Add a Document ID Field: Before splitting, assign a unique document ID to the entire document that will be preserved across all chunks.
Create Position Metadata: For each chunk, include metadata like:
Document ID (to group all chunks)
Chunk sequence number (e.g., 1 of 8, 2 of 8)
Original filename
Total chunks in document
Overlap Content Between Chunks: Include a small overlap between consecutive chunks (50-100 tokens) to maintain context across chunk boundaries.
Tag in Vector Database: When storing in your vector database, ensure all chunks are tagged with the same document identifier.
Retrieval Handling: When retrieving information, you can:
Pull all chunks with the same document ID when the entire document is needed
For queries, return the most relevant chunks but maintain context of which document they belong to
Implementation in n8n
In your n8n workflow, you could:
Add a "Set" node after your file download to generate a document ID and total file size
Add a "Split Binary Data" node to segment large files into manageable chunks
Use a "Loop" node to process each chunk while maintaining document metadata
When inserting into your vector database, ensure the document ID is preserved
This way, your system will treat the chunks as parts of a single document during processing but will avoid the memory issues of handling the entire file at once.
The RAG system will still function properly as long as the embedding and retrieval processes understand that these chunks belong together. During retrieval, relevant chunks can be returned based on the query, but they'll be identified as parts of the same source document.