Multi-Agent Collaborative RAG Design & Testing
Budget: $10 – $30 USD
Design a Multiple Agent Collobarative RAG and test on baselines as listed:
Baselines
[ ] Chain-of-Thought (CoT)
Zero-shot prompting with direct step-by-step reasoning.
Prompt: “Please think step by step and then solve the task.”
[ ] Self-Consistency (SC)
Generate diverse CoT traces with temperature = 0.8.
Apply rule-based majority vote for most consistent answer.
Reported as SC@9 for fair comparison.
[ ] Self-Refine
Predictor receives feedback from self-reflector.
Stop when self-reflector outputs “correct.”
Max reflections = 5 → worst case = 11 calls (1 + 2×5).
[ ] Multi-Agent Debate
3 agents debate for 3 rounds.
Aggregator judges final prediction.
Total = 10 agents (3×3 + 1).
[ ] ADAS (Automated Design of Agentic Systems)
Uses Gemini 1.5 as optimizer + evaluator.
Conditioned on prior baseline evaluations.
30 rounds of search, each evaluated 3× on validation set.
[ ] AFlow
Workflow design via Monte-Carlo Tree Search.
LLM optimizer = Claude 3.5 Sonnet.
Executor = Gemini 1.5 Pro.
Setup: 20 rounds, 5 validations per round, k=3.
Note: Out-of-time errors should be minimized due to infinite loops.
B.3. New Collobarative Rag Details & Construction Rules
[ ] Topology Search Space
Defined per task ; task vs results
[ ] Stage (1) Block-Level Prompt Optimization
Building block specs in need to be designed well.
[ ] Construction Rule
Fixed order: [summarize → reflect → debate → aggregate].
Aggregate controls number of parallel chains.
Chain length defined by pre-set order.
Baselines
[ ] Chain-of-Thought (CoT)
Zero-shot prompting with direct step-by-step reasoning.
Prompt: “Please think step by step and then solve the task.”
[ ] Self-Consistency (SC)
Generate diverse CoT traces with temperature = 0.8.
Apply rule-based majority vote for most consistent answer.
Reported as SC@9 for fair comparison.
[ ] Self-Refine
Predictor receives feedback from self-reflector.
Stop when self-reflector outputs “correct.”
Max reflections = 5 → worst case = 11 calls (1 + 2×5).
[ ] Multi-Agent Debate
3 agents debate for 3 rounds.
Aggregator judges final prediction.
Total = 10 agents (3×3 + 1).
[ ] ADAS (Automated Design of Agentic Systems)
Uses Gemini 1.5 as optimizer + evaluator.
Conditioned on prior baseline evaluations.
30 rounds of search, each evaluated 3× on validation set.
[ ] AFlow
Workflow design via Monte-Carlo Tree Search.
LLM optimizer = Claude 3.5 Sonnet.
Executor = Gemini 1.5 Pro.
Setup: 20 rounds, 5 validations per round, k=3.
Note: Out-of-time errors should be minimized due to infinite loops.
B.3. New Collobarative Rag Details & Construction Rules
[ ] Topology Search Space
Defined per task ; task vs results
[ ] Stage (1) Block-Level Prompt Optimization
Building block specs in need to be designed well.
[ ] Construction Rule
Fixed order: [summarize → reflect → debate → aggregate].
Aggregate controls number of parallel chains.
Chain length defined by pre-set order.