Constraint-Aware RAG Architecture for SLMs
Budget: ₹37,500 – ₹75,000 INR
Hypothesis:
A constraint-aware, evidence-linked RAG architecture specifically optimized for Small Language Models (SLMs) can achieve legal reasoning accuracy and hallucination reduction comparable to or better than larger LLM-based RAG systems, while operating at significantly lower computational cost.
So, basically you can create a new RAG architecture than performs much better than a currently available Plain SLM.
Steps:
1. Check these SLMs first to benchmark on Retrieval/QA tasks: Inlegal‑Sbert (Retrieval) / InLegalBERT embeddings (Retrieval) / InLegalLlama (QA)
2. Dataset in consideration:
Dataset:
a) IPC Sections with conditions, exceptions, constraints, relations
b) Supreme court judgement copies
c) IPC_and_case_related_QA_dataset
3. Evaluate Inlegal‑Sbert and InLegalBERT embeddings on retrieval task and evaluate InLegalLlama on QA task from the above dataset
4. Evaluation
Metrics for Retrieval:
nDCG@10
Recall@5 / Recall@10
Mean Reciprocal Rank (MRR)
Precision@k
Metrics for QA Evaluation:
ROUGE-L
BERTScore
Faithfulness (are all claims backed by retrieved text?)
Citation Accuracy
Hallucination Rate
5. Now build constraint-aware, evidence-linked RAG architecture and add it with SLM to see if it can beat the previous metrics evaluated from the SLM alone.
6. For building the constraint-aware, evidence-linked RAG architecture, refer following,
Core components:
===============
1. Corpus & Index — statute chunks with hierarchy metadata (section/subsection/illustration/exception).
2. Embedding retriever — InLegal-SBERT / InLegalBERT sentence embeddings + FAISS index.
3. Reranker — cross-encoder reranker (InLegalBERT fine-tuned) to re-rank top-k.
4. Constraint module — set of lightweight rule engines/filters enforcing legal constraints before generation.
5. Evidence formatter — organizes retrieved snippets + provenance into generator context.
6. Generator (SLM) — small seq2seq (FLAN-T5 small/TinyLlama / Phi-4) fine-tuned with LoRA/QLoRA to output multi-field evidence-linked answers.
7. Hallucination detector / verifier — checks claims against retrieved evidence; optionally runs a secondary verification model.
A constraint-aware, evidence-linked RAG architecture specifically optimized for Small Language Models (SLMs) can achieve legal reasoning accuracy and hallucination reduction comparable to or better than larger LLM-based RAG systems, while operating at significantly lower computational cost.
So, basically you can create a new RAG architecture than performs much better than a currently available Plain SLM.
Steps:
1. Check these SLMs first to benchmark on Retrieval/QA tasks: Inlegal‑Sbert (Retrieval) / InLegalBERT embeddings (Retrieval) / InLegalLlama (QA)
2. Dataset in consideration:
Dataset:
a) IPC Sections with conditions, exceptions, constraints, relations
b) Supreme court judgement copies
c) IPC_and_case_related_QA_dataset
3. Evaluate Inlegal‑Sbert and InLegalBERT embeddings on retrieval task and evaluate InLegalLlama on QA task from the above dataset
4. Evaluation
Metrics for Retrieval:
nDCG@10
Recall@5 / Recall@10
Mean Reciprocal Rank (MRR)
Precision@k
Metrics for QA Evaluation:
ROUGE-L
BERTScore
Faithfulness (are all claims backed by retrieved text?)
Citation Accuracy
Hallucination Rate
5. Now build constraint-aware, evidence-linked RAG architecture and add it with SLM to see if it can beat the previous metrics evaluated from the SLM alone.
6. For building the constraint-aware, evidence-linked RAG architecture, refer following,
Core components:
===============
1. Corpus & Index — statute chunks with hierarchy metadata (section/subsection/illustration/exception).
2. Embedding retriever — InLegal-SBERT / InLegalBERT sentence embeddings + FAISS index.
3. Reranker — cross-encoder reranker (InLegalBERT fine-tuned) to re-rank top-k.
4. Constraint module — set of lightweight rule engines/filters enforcing legal constraints before generation.
5. Evidence formatter — organizes retrieved snippets + provenance into generator context.
6. Generator (SLM) — small seq2seq (FLAN-T5 small/TinyLlama / Phi-4) fine-tuned with LoRA/QLoRA to output multi-field evidence-linked answers.
7. Hallucination detector / verifier — checks claims against retrieved evidence; optionally runs a secondary verification model.