Fine-tuning BERT transformer for multitask

I need to fine-tune the Bert model to first retrieve the similar top pieces of evidence (let's say based on some threshold score) from the knowledge base for the given claim, then classify the similarity between the claim and returned pieces of evidence into four classes based on the level of the similarity.