Real-time stream processing using Kafka and Spark
Budget: $30 – $250 USD
Hi all,
I have a Real-time stream processing project using Kafka and Spark
I need someone who is good in Python, PySpark, Kafka and Jupyter Notebook
I have done some of the work on this project (like 45%) and I need someone to finish it for me.
NOTE: IT IS EXTREMLY IMPORTANT THAT YOU USE JUPYTER NOTEBOOK AND THE TEMPLATES PROVIDED
● Task 1 (Already Done): Kafka Producer - Producing the streaming data, where you can use csv modules to read and publish the data to the Kafka stream.
● Task 2 (Partially Done): Kafka Consumer - Consuming the streaming data using Kafka consumer, use the csv module in any python libraries (e.g. Pandas) to process the ingested data from Kafka.
● Task 3 (Not attempted): Streaming application - Using Spark Structured Streaming together with Spark ML/SQL to process data streams. In task 3, pandas can only be used for plotting steps.
The excessive usage of Pandas for data processing is not recommended.
In this task, we will implement Spark Structured Streaming to consume the data from task 1 and perform streaming classification. You may use Spark Structured Streaming together with Spark SQL and ML. You are also provided with a set of pre-trained pipeline models to classify the binary classification flight-delays.
NOTE: PLEASE LABEL THE STEPS IN TASK 3 AS THEY CORRESPOD TO THE INSTRCTIONS FILE
You can find the rest of the instructions in the file (Instructions.pdf)
There are three ipynb files:
1. Task1_flight_producer.ipynb
2. Task2_flight_consumer.ipynb
3. Task3_streaming_application.ipynb
The dataset for this project consists of 20 csv flight files (flight*.csv) and the metadata.pdf and airports.csv. You only need to use flight*.csv files. The file is included in the attachments (flight-delays.zip)
The model to be used in Task 3 is also included in the attachments (model.zip)
The Timeline for this project is 2 days. Please let me know if you can do it in this time or not
I have a Real-time stream processing project using Kafka and Spark
I need someone who is good in Python, PySpark, Kafka and Jupyter Notebook
I have done some of the work on this project (like 45%) and I need someone to finish it for me.
NOTE: IT IS EXTREMLY IMPORTANT THAT YOU USE JUPYTER NOTEBOOK AND THE TEMPLATES PROVIDED
● Task 1 (Already Done): Kafka Producer - Producing the streaming data, where you can use csv modules to read and publish the data to the Kafka stream.
● Task 2 (Partially Done): Kafka Consumer - Consuming the streaming data using Kafka consumer, use the csv module in any python libraries (e.g. Pandas) to process the ingested data from Kafka.
● Task 3 (Not attempted): Streaming application - Using Spark Structured Streaming together with Spark ML/SQL to process data streams. In task 3, pandas can only be used for plotting steps.
The excessive usage of Pandas for data processing is not recommended.
In this task, we will implement Spark Structured Streaming to consume the data from task 1 and perform streaming classification. You may use Spark Structured Streaming together with Spark SQL and ML. You are also provided with a set of pre-trained pipeline models to classify the binary classification flight-delays.
NOTE: PLEASE LABEL THE STEPS IN TASK 3 AS THEY CORRESPOD TO THE INSTRCTIONS FILE
You can find the rest of the instructions in the file (Instructions.pdf)
There are three ipynb files:
1. Task1_flight_producer.ipynb
2. Task2_flight_consumer.ipynb
3. Task3_streaming_application.ipynb
The dataset for this project consists of 20 csv flight files (flight*.csv) and the metadata.pdf and airports.csv. You only need to use flight*.csv files. The file is included in the attachments (flight-delays.zip)
The model to be used in Task 3 is also included in the attachments (model.zip)
The Timeline for this project is 2 days. Please let me know if you can do it in this time or not