Textual Data Analysis in Python

Task 1.1: Define a document parsing function

Create a function parse_rcv1v2(stop_words, inputpath) to parse XML files in a folder.
Implement steps to read XML files, extract docID and index terms, and create Rcv1Doc objects.
Define a class Rcv1Doc with attributes like docID, terms (a dictionary of terms and their frequencies), and doc_len.
Tokenize text, remove stop words, and apply stemming using the Porter2 algorithm.
Define methods like getDocId() and get_term_list().

Task 1.2: Define a query parsing function

Create a function parse_query(query0, stop_words) to parse a query string.
Tokenize the query, remove stop words, and apply stemming.
Return a dictionary of terms with their frequencies in the query.

Task 1.3: Define a main function

Implement a main function to test parse_rcv1v2() and parse_query().
Print document details (docID, term count, doc_len), sorted terms, and save output to a text file.

Task 2.1: Calculate document-frequency (df)

Create a function my_df(coll) to calculate document-frequency for a given Rcv1Doc collection.
Return a dictionary of terms with their document frequencies.

Task 2.2: Calculate TF*IDF values

Define a function my_tfidf(doc, df, ndocs) to calculate TF*IDF values for terms in a document.
Use the TF*IDF formula with term frequency, document frequency, and total number of documents.

Task 2.3: Implement a TF*IDF based IR model

Create a main function to call my_tfidf() for each document and print the top 20 terms with TF*IDF weights.
Use original queries as titles to calculate ranking scores using TF*IDF.

Task 3.1: Calculate average document length

Implement a function avg_length(coll) to calculate the average document length in a collection.
Modify code to store and calculate document lengths in Rcv1Doc objects.

Task 3.2: Calculate BM25 scores

Define a function my_bm25(coll, q, df) to calculate BM25 scores for documents given a query.
Parse the query using parse_query() and apply the BM25 formula to calculate scores.

Task 3.3: Implement a BM25-based IR model

Create a main function to rank documents using BM25 scores for specified queries.
Print the top relevant documents based on BM25 scores.
Ensure your code is well-commented, follows the specified requirements, and provides accurate outputs.