Text summarization using pre-trained language models by fine-tuning a gpt2 model

We have to fine-tune a pre-trained GPT2 laguage model for the task of text summarization, the dataset to train on is the amazon reviews dataset, which contains reviews of amazon products, we use the reviews_body and reviews title columns, with the body being the text and the titles column to use for generating the summary. The dataset is multi-lingual but we only use the English language part. we also only use reviews with title length greater than two words, as the dataset is quite biased along these sizes and to generate meaningful summaries.