Ensembling 2 Models from Research Papers on Google Colab
Budget: $100 – $150 USD
Request details
Essentially the problem I'm trying to work on is how to improve entity recognition, specifically financial entity recognition.
I'm drawing from 2 papers that have made progress in the area:
Loukas et al.,: this paper was the first to define the problem of financial entity recognition (FinNER). They define their problem specifically as creating XBRL tags (essentially just a specific data format tag that is mandated by the SEC, details don't matter too much about XBRL).
Key draws from this paper is:
a) the dataset: the paper contains the dataset of unstructured text sentences which I hope to train my new model on too.
b) use of shape tokens: they showed that replacing numbers with 'shape' tokens, i.e. just replacing all numbers with digit representations instead, drastically improved the ability to identify financial entities for XBRL tags (i.e. changing [52.2] → [XX.X]). Essentially relying more on context around the number to generate the tag. (I may be wrong, but to my understanding this is kind of like sequence tagging?)
This paper proposes a SEC-BERT-Shape Model
Zhang and Zhang: this paper takes a different approach to FinNER, thinking of the problem as machine reading comprehension (MRC). To my understanding, MRC just means changing a task into a query: For example, the task of assigning the LOC (LOCATION) label to "So far U.S. soldiers have discovered nearly 600 million dollars hidden around [Baghdad]" is formalized as answering the query "Which location is mentioned in the text?". The paper looks at 3 ways of constructing queries:
Keyword: The keyword representing the entity tag serves as the query.
Rule-based query template: Templates are used to generate the queries.
Wikipedia: The Wikipedia definition of the entity is used to build the query.
Key draw from this paper is the use of the MRC framework. Forming queries instead allows the incorporation of 'previous knowledge' of certain tags.
This paper proposes a Fin-BERT-MRC model.
Paper 1 Link: https://arxiv.org/pdf/2203.06482v1.pdf
Paper 1 Repo: https://github.com/nlpaueb/finer
Paper 2 Link: https://arxiv.org/pdf/2205.15485.pdf
Paper 2 Repo: https://github.com/zyz0000/FinBERT-MRC
Deliverables
I'm hoping to propose a solution where I literally just use both models independently on the dataset from Loukas et al. To generate predictions and then vary the weights of the predictions coming out of each model. I figure trying to merge the two into one model is probably going to be impossible for me given the time constraint.
Output must be in google colab, current notebook:
https://colab.research.google.com/drive/1KwNwxMXgzL_IaJzJ3Uk5qr9mmDL2OZui?usp=sharing
Essentially the problem I'm trying to work on is how to improve entity recognition, specifically financial entity recognition.
I'm drawing from 2 papers that have made progress in the area:
Loukas et al.,: this paper was the first to define the problem of financial entity recognition (FinNER). They define their problem specifically as creating XBRL tags (essentially just a specific data format tag that is mandated by the SEC, details don't matter too much about XBRL).
Key draws from this paper is:
a) the dataset: the paper contains the dataset of unstructured text sentences which I hope to train my new model on too.
b) use of shape tokens: they showed that replacing numbers with 'shape' tokens, i.e. just replacing all numbers with digit representations instead, drastically improved the ability to identify financial entities for XBRL tags (i.e. changing [52.2] → [XX.X]). Essentially relying more on context around the number to generate the tag. (I may be wrong, but to my understanding this is kind of like sequence tagging?)
This paper proposes a SEC-BERT-Shape Model
Zhang and Zhang: this paper takes a different approach to FinNER, thinking of the problem as machine reading comprehension (MRC). To my understanding, MRC just means changing a task into a query: For example, the task of assigning the LOC (LOCATION) label to "So far U.S. soldiers have discovered nearly 600 million dollars hidden around [Baghdad]" is formalized as answering the query "Which location is mentioned in the text?". The paper looks at 3 ways of constructing queries:
Keyword: The keyword representing the entity tag serves as the query.
Rule-based query template: Templates are used to generate the queries.
Wikipedia: The Wikipedia definition of the entity is used to build the query.
Key draw from this paper is the use of the MRC framework. Forming queries instead allows the incorporation of 'previous knowledge' of certain tags.
This paper proposes a Fin-BERT-MRC model.
Paper 1 Link: https://arxiv.org/pdf/2203.06482v1.pdf
Paper 1 Repo: https://github.com/nlpaueb/finer
Paper 2 Link: https://arxiv.org/pdf/2205.15485.pdf
Paper 2 Repo: https://github.com/zyz0000/FinBERT-MRC
Deliverables
I'm hoping to propose a solution where I literally just use both models independently on the dataset from Loukas et al. To generate predictions and then vary the weights of the predictions coming out of each model. I figure trying to merge the two into one model is probably going to be impossible for me given the time constraint.
Output must be in google colab, current notebook:
https://colab.research.google.com/drive/1KwNwxMXgzL_IaJzJ3Uk5qr9mmDL2OZui?usp=sharing