PySpark Dataframe Summarization and Classification
Budget: $30 – $250 USD
I'm looking for a skilled developer with experience in Azure AI Services, specifically Text Analytics, and Azure OpenAI. The project involves processing a large PySpark dataframe which contains text in English.
Key Requirements:
- Use Azure Text Analytics for Abstract Summarization: The candidate should be able to leverage Azure Text Analytics to generate concise summaries from the large amount of text data.
- Utilize Azure OpenAI for Text Classification: The project requires the use of Azure OpenAI to classify the texts.
- Overcome Token Limitation: As the data is extensive, it's crucial to chunk the text into smaller parts to bypass the token limit issue effectively.
Ideal Skills and Experience:
- Proficient in Azure AI Services: A deep understanding of Azure's Text Analytics service and experience with integrating it into Python code.
- Experienced in PySpark: A strong background in PySpark is necessary to efficiently manipulate the large dataframe in Databricks.
The dataset can be found here:
https://github.com/microsoft/Data-Discovery-Toolkit/blob/main/data/sport_articles.csv
Classification using Azure OpenAI: use the column `Text` to classify whether it's Football, Athletics, Cricket, Rugby, or Tennis.
Abstractive summarization: use Azure Cognitive Services to summarise the column `Text`.
Success Criteria: Should work with PySpark and return all the classified categories and summary with properly handling the token limitation, and preferably asynchronously. It should perform fast in PySpark.
Note: My code works but it hits the rate limit.
Key Requirements:
- Use Azure Text Analytics for Abstract Summarization: The candidate should be able to leverage Azure Text Analytics to generate concise summaries from the large amount of text data.
- Utilize Azure OpenAI for Text Classification: The project requires the use of Azure OpenAI to classify the texts.
- Overcome Token Limitation: As the data is extensive, it's crucial to chunk the text into smaller parts to bypass the token limit issue effectively.
Ideal Skills and Experience:
- Proficient in Azure AI Services: A deep understanding of Azure's Text Analytics service and experience with integrating it into Python code.
- Experienced in PySpark: A strong background in PySpark is necessary to efficiently manipulate the large dataframe in Databricks.
The dataset can be found here:
https://github.com/microsoft/Data-Discovery-Toolkit/blob/main/data/sport_articles.csv
Classification using Azure OpenAI: use the column `Text` to classify whether it's Football, Athletics, Cricket, Rugby, or Tennis.
Abstractive summarization: use Azure Cognitive Services to summarise the column `Text`.
Success Criteria: Should work with PySpark and return all the classified categories and summary with properly handling the token limitation, and preferably asynchronously. It should perform fast in PySpark.
Note: My code works but it hits the rate limit.