Artificial intelligence LLM
Budget: $10 – $200 USD
You must use the template provided on Moodle For PartAyou will need to compare
different approaches to adapting an LLM so that it can solve a summarisation task,
as demonstrated by the following dataset:
https://huggingface.co/datasets/EdinburghNLP/xsum
The dataset includes articles and corresponding summaries as shown in the image
above. You should pick a Large Language Model of your choice (either one shown
during tutorials or another suitable model from Hugging Face, considering computational constraints). The dataset follows a standard Hugging Face dataset template.
Before applying the learning approaches detailed below in steps 1-3 you will need to:
decide how to split the data; define the data processing function; configure the trainer
etc.). To help with this, you can refer to Unit 4 tutorial where you fine-tuned models
for a similar task.
For PartA, you will need to hand in the Jupyter notebook file evidencing step 1-5 below, with all of the code outputs included (you must follow the template provided
on Moodle). Make sure it can be run via Google Colab. Remember to import and list
all required libraries. The contents of step 5 should be included within the same
notebook file using dedicated markdown cells. Good Luck!
1. Use zero-shot learning to evaluate the performance of the LLM on this task. Make
sure you only test the model on the test set. (5 marks)
2. Use few-shot learning to evaluate the performance of the LLM on this task.Again,
test it only on the test set. (5 marks)
Page 6 of 6
3. Finally, employ prompt engineering to solve this task. Again, evaluate only on test
set. Focus on carefully articulating instructions, constraints (e.g., length, style), or
any special conditions for the summary. Make sure to assess both zero- (3 marks)
and few-shot settings (3 marks).
4. For the evaluation, you need to pick an appropriate evaluation metric. Create a table that shows the four different approaches and the results they obtained during
the evaluation. (5 marks)
5. Assess and interpret the model performance. Provide a brief discussion concerning Results Interpretation (max 200 words) covering:
• Findings: Did any method perform significantly better? • Surprising errors:Any unexpected outputs or failure cases?
• Benchmark vs actual performance: Did the results align with expectations?
(9 marks)
What to hand in on Moodle: AJupyter notebook file following the template provided containing everything. A cover sheet provided on Moodle declaring that this is
your own work.
different approaches to adapting an LLM so that it can solve a summarisation task,
as demonstrated by the following dataset:
https://huggingface.co/datasets/EdinburghNLP/xsum
The dataset includes articles and corresponding summaries as shown in the image
above. You should pick a Large Language Model of your choice (either one shown
during tutorials or another suitable model from Hugging Face, considering computational constraints). The dataset follows a standard Hugging Face dataset template.
Before applying the learning approaches detailed below in steps 1-3 you will need to:
decide how to split the data; define the data processing function; configure the trainer
etc.). To help with this, you can refer to Unit 4 tutorial where you fine-tuned models
for a similar task.
For PartA, you will need to hand in the Jupyter notebook file evidencing step 1-5 below, with all of the code outputs included (you must follow the template provided
on Moodle). Make sure it can be run via Google Colab. Remember to import and list
all required libraries. The contents of step 5 should be included within the same
notebook file using dedicated markdown cells. Good Luck!
1. Use zero-shot learning to evaluate the performance of the LLM on this task. Make
sure you only test the model on the test set. (5 marks)
2. Use few-shot learning to evaluate the performance of the LLM on this task.Again,
test it only on the test set. (5 marks)
Page 6 of 6
3. Finally, employ prompt engineering to solve this task. Again, evaluate only on test
set. Focus on carefully articulating instructions, constraints (e.g., length, style), or
any special conditions for the summary. Make sure to assess both zero- (3 marks)
and few-shot settings (3 marks).
4. For the evaluation, you need to pick an appropriate evaluation metric. Create a table that shows the four different approaches and the results they obtained during
the evaluation. (5 marks)
5. Assess and interpret the model performance. Provide a brief discussion concerning Results Interpretation (max 200 words) covering:
• Findings: Did any method perform significantly better? • Surprising errors:Any unexpected outputs or failure cases?
• Benchmark vs actual performance: Did the results align with expectations?
(9 marks)
What to hand in on Moodle: AJupyter notebook file following the template provided containing everything. A cover sheet provided on Moodle declaring that this is
your own work.