NLP：Assignment Two Sentiment classification for social media

Topic: Building a sentiment classifier for Twitter
Semeval competitions involve addressing different challenges pertaining to the extraction of meaning
from text (semantics). The organisers of those competitions provide a dataset and a task, so that
different participants can develop their system. In this exercise, we will focus on the Task 4 of Semeval
2017 (http://alt.qcri.org/semeval2017/task4/). We will focus particularly on Subtask A, i.e. classifying
the overall sentiment of a tweet as positive, negative or neutral.
As part of the classification task, you will need to preprocess the tweets. You may
want to tweak your preprocessing code to deal with particularities of tweets, e.g. #hashtags or @user
mentions.

Exercise guidelines
• Data: The training, development and test sets can be downloaded from the module website
(semeval-tweets.tar.bz2). This compressed archive includes 5 files, one that is used for training
(twitter-training-data.txt) another one for development (twitter-dev-data.txt) and another 3
that are used as different subsets for testing (twitter-test[1-3].txt). You may use the
development set as the test set while you are developing your classifier, so that you tweak your
classifiers and features; the development set can also be useful to compute hyperparameters,
where needed. The files are formatted as TSV (tab-separated-values), with one tweet per row
that includes the following values:
tweet-id<tab>sentiment<tab>tweet-text
where sentiment is one of {positive, negative, neutral}. The tweet IDs will be used as unique
identifiers to build a Python dictionary with the predictions of your classifiers, e.g.:
predictions = {‘163361196206957578’: ‘positive’,
‘768006053969268950’: ‘negative’,
…}
• Classifier: You are requested to develop classifiers that learn from the training data and test
on each of the 3 test sets separately (i.e. evaluating on 3 different sets). You are given the
skeleton of the code (sentiment-classifier.tar.bz2), with evaluation script included, which will
help you develop your system in a way that we will then be able to run on our computers.
Evaluation on different tests allows you to generalise your results. You may achieve an
2
improvement over a particular test set just by chance (e.g. overfitting), but improvement over
multiple test sets makes it more likely to be a significant improvement.
You should develop at least 3 different classifiers, which you will then present and compare in
your report. Please develop at least 2 classifiers based on (1) traditional machine learning
methods such as MaxEnt, SVM or Naïve Bayes trained on different sets of features (you could
use Scikit-learn library). Then, train an another classifier based on the LSTM using PyTorch
(and optionally the torchtext library) by following the steps below:
a) Download the GloVe word embeddings and map each word in the dataset into its pretrained GloVe word embedding.
First go to https://nlp.stanford.edu/projects/glove/ and download the pre-trained
embeddings from 2014 English Wikipedia into the "data" directory. It's a 822MB zip file
named glove.6B.zip, containing 100-dimensional embedding vectors for 400,000
words (or non-word tokens). Un-zip it. Parse the un-zipped file (it's a txt file) to build an
index mapping words (as strings) to their vector representation (as number vectors).
Build an embedding matrix that will be loaded into an Embedding layer later. It must be a
matrix of shape (max_words, embedding_dim), where each entry i contains the
embedding_dim-dimensional vector for the word of index i in our reference word index
(built during tokenization). Note that the index 0 is not supposed to stand for any word or
token -- it's a placeholder.