Adding some Hardcoded examples in Generations through BigCode Harness Code
Budget: ₹600 – ₹2,000 INR
The code generation LM evaluation harness is a framework designed to evaluate code generation models. It is inspired by the EleutherAI/lm-evaluation-harness, which focuses on evaluating language models in general. The evaluation harness supports various code generation tasks and provides features such as multi-GPU text generation, Docker support for security and reproducibility, and compatibility with autoregressive models available on the Hugging Face model hub.
Code generation Python tasks: The harness includes four code generation Python tasks, namely HumanEval, APPS, MBPP, and DS-1000. These tasks can be evaluated using both completion (left-to-right) and insertion (FIM) modes.
The task is to modify the utils.py file in the lm_eval package to append hard-coded prompts and solutions from the HumanEval dataset (below link) to the TokenizedDataset.
https://huggingface.co/datasets/openai_humaneval/viewer/openai_humaneval/test
Code generation Python tasks: The harness includes four code generation Python tasks, namely HumanEval, APPS, MBPP, and DS-1000. These tasks can be evaluated using both completion (left-to-right) and insertion (FIM) modes.
The task is to modify the utils.py file in the lm_eval package to append hard-coded prompts and solutions from the HumanEval dataset (below link) to the TokenizedDataset.
https://huggingface.co/datasets/openai_humaneval/viewer/openai_humaneval/test