Multi task learning in NLP to classify Arabic offensive langauge

Due to the increase in Arabic content on the Internet, especially inappropriate content, which led to the spread of the language of offensive and hated among the releases, the presence of a model to classify many tasks and datasets and train it to recognize the offensive speech has become an urgent necessity, and in conjunction with the multi task learning revolution, which in turn performs a mission of Classification for more than one task, However MTL dealing with many challenges such as overfitting to low resource tasks, catastrophic forgetting, and negative task transfer. In this research we propose a Transformer based Adapter consisting of a new conditional attention mechanism as well as a set of task-conditioned modules that facilitate weight sharing, According to the studies of the current baseline of the Transformer model, when adding layers to it, it enhancing the process of sharing parameters and avoiding the tasks imbalance issue. Through this construction, we achieve more efficient parameter sharing and mitigate forgetting by keeping half of the weights of a pre-trained model fixed. We also use a new multi-task data sampling strategy to mitigate the negative effects of data imbalance across tasks.


Objectives:
1. To Propose better handling of the catastrophic forgetting issue by prioritising tasks with uncertainty based multi task data sampling to help balancing the sampling of tasks.
2. Improve pertained knowledge retention of Arabert model and multi task inductive knowledge transfer by conditioning adaptive learning using pre-trained weights.