Build an Arabic Keyphrase Extraction System using Machine Learning (Applied Research Implementation)
Budget: $2 – $8 USD
1. Project Overview
The goal of this project is to practically implement a research paper that presents a system for automatically extracting keyphrases from Arabic text. The core idea is to combine linguistic knowledge (like grammatical rules and lemmatization) with machine learning instead of relying solely on statistics.
You will be building a complete Python-based system that can take a raw Arabic text document and output a list of the most relevant keyphrases that summarize its content.
2. Scope of Work & Deliverables
The developer is expected to deliver the following:
Complete Source Code: Well-commented and organized Python code.
Trained Model: A serialized machine learning model ready for prediction.
Training Dataset: A small, curated dataset of Arabic documents with their correct keyphrases (or scripts to generate it).
Requirements File: A requirements.txt file listing all dependencies and their versions.
Documentation: A clear README.md file with instructions on how to install, run, and use the system.
Examples: Demonstration of the system working on sample Arabic texts.
3. Detailed Technical Requirements
The system must be built in Python and include the following modules:
Module 1: Linguistic Preprocessing & Analysis
Use modern Arabic NLP libraries (e.g., CAMeL Tools or Stanza) to perform:
Text Normalization and Tokenization (splitting text into words/sentences).
Part-of-Speech (POS) Tagging for each word (e.g., noun, verb, adjective).
Lemmatization (extracting the abstract root form of each word, e.g., converting "المشاريع" and "مشروع" to the same lemma).
Module 2: Candidate Phrase Extraction
Generate all possible 1, 2, and 3-word sequences (n-grams) from the text.
Implement syntactic filtering rules (as defined in the research paper) using the POS tags to remove invalid phrases. Example rules:
A phrase can only start with certain types of nouns.
A phrase can only end with a noun or an adjective.
Filter out phrases with prepositions in invalid positions.
Module 3: Feature Engineering
For each valid candidate phrase, calculate a set of numerical features that represent its importance. Key features from the research include:
NPW: Normalized Phrase Length (number of words).
PRF: Phrase Relative Frequency (how often the lemma of the phrase appears).
WRF: Word Relative Frequency (frequency of the most important word in the phrase).
NSL: Normalized Sentence Location (weights phrases found at the start/end of the document higher).
SCV: Sentence Contains a Verb (binary feature to help identify title-like phrases).
Module 4: Machine Learning Classification
Implement a classifier using scikit-learn.
The preferred algorithm is Linear Discriminant Analysis (LDA) as per the paper, but other suitable classifiers (e.g., Random Forest) can be discussed.
The model must be trained on a prepared dataset where each candidate phrase is labeled as a keyphrase (1) or not (0).
The final output should be a ranked list of the top N (e.g., 5-7) most probable keyphrases.
Module 5: Application Interface
A simple Command-Line Interface (CLI) is sufficient. The program should accept an input text file (.txt) and print or save the resulting keyphrases.
4. Technology Stack
Language: Python 3.x
Key Libraries:
camel-tools or stanza (for Arabic NLP)
scikit-learn (for machine learning)
pandas, numpy (for data manipulation)
nltk (for basic n-gram generation)
Version Control: Code must be delivered via Git (e.g., a GitHub repository).
5. Data Requirement
If a pre-labeled Arabic keyphrase dataset is not readily available, the developer is expected to create a small training set. This involves:
Collecting 20-30 Arabic articles (preferably from the technology domain).
Manually assigning 5-7 relevant keyphrases to each document to serve as the ground truth for training.
6. Acceptance Criteria
The project will be considered successful when:
The system can process a new Arabic text and output a list of keyphrases.
The extracted keyphrases are logically relevant to the input text's content (tested on unseen documents).
All source code, the trained model, and documentation are delivered.
Clear instructions for running the system are provided.
The goal of this project is to practically implement a research paper that presents a system for automatically extracting keyphrases from Arabic text. The core idea is to combine linguistic knowledge (like grammatical rules and lemmatization) with machine learning instead of relying solely on statistics.
You will be building a complete Python-based system that can take a raw Arabic text document and output a list of the most relevant keyphrases that summarize its content.
2. Scope of Work & Deliverables
The developer is expected to deliver the following:
Complete Source Code: Well-commented and organized Python code.
Trained Model: A serialized machine learning model ready for prediction.
Training Dataset: A small, curated dataset of Arabic documents with their correct keyphrases (or scripts to generate it).
Requirements File: A requirements.txt file listing all dependencies and their versions.
Documentation: A clear README.md file with instructions on how to install, run, and use the system.
Examples: Demonstration of the system working on sample Arabic texts.
3. Detailed Technical Requirements
The system must be built in Python and include the following modules:
Module 1: Linguistic Preprocessing & Analysis
Use modern Arabic NLP libraries (e.g., CAMeL Tools or Stanza) to perform:
Text Normalization and Tokenization (splitting text into words/sentences).
Part-of-Speech (POS) Tagging for each word (e.g., noun, verb, adjective).
Lemmatization (extracting the abstract root form of each word, e.g., converting "المشاريع" and "مشروع" to the same lemma).
Module 2: Candidate Phrase Extraction
Generate all possible 1, 2, and 3-word sequences (n-grams) from the text.
Implement syntactic filtering rules (as defined in the research paper) using the POS tags to remove invalid phrases. Example rules:
A phrase can only start with certain types of nouns.
A phrase can only end with a noun or an adjective.
Filter out phrases with prepositions in invalid positions.
Module 3: Feature Engineering
For each valid candidate phrase, calculate a set of numerical features that represent its importance. Key features from the research include:
NPW: Normalized Phrase Length (number of words).
PRF: Phrase Relative Frequency (how often the lemma of the phrase appears).
WRF: Word Relative Frequency (frequency of the most important word in the phrase).
NSL: Normalized Sentence Location (weights phrases found at the start/end of the document higher).
SCV: Sentence Contains a Verb (binary feature to help identify title-like phrases).
Module 4: Machine Learning Classification
Implement a classifier using scikit-learn.
The preferred algorithm is Linear Discriminant Analysis (LDA) as per the paper, but other suitable classifiers (e.g., Random Forest) can be discussed.
The model must be trained on a prepared dataset where each candidate phrase is labeled as a keyphrase (1) or not (0).
The final output should be a ranked list of the top N (e.g., 5-7) most probable keyphrases.
Module 5: Application Interface
A simple Command-Line Interface (CLI) is sufficient. The program should accept an input text file (.txt) and print or save the resulting keyphrases.
4. Technology Stack
Language: Python 3.x
Key Libraries:
camel-tools or stanza (for Arabic NLP)
scikit-learn (for machine learning)
pandas, numpy (for data manipulation)
nltk (for basic n-gram generation)
Version Control: Code must be delivered via Git (e.g., a GitHub repository).
5. Data Requirement
If a pre-labeled Arabic keyphrase dataset is not readily available, the developer is expected to create a small training set. This involves:
Collecting 20-30 Arabic articles (preferably from the technology domain).
Manually assigning 5-7 relevant keyphrases to each document to serve as the ground truth for training.
6. Acceptance Criteria
The project will be considered successful when:
The system can process a new Arabic text and output a list of keyphrases.
The extracted keyphrases are logically relevant to the input text's content (tested on unseen documents).
All source code, the trained model, and documentation are delivered.
Clear instructions for running the system are provided.