Help with data stream processing using Kafka broker and Spark Structured Streaming
Budget: $15 – $65 USD
Looking for help with data stream processing using Apache Kafka and Spark Structured Streaming.
Data is about Crime statistics in Chicago.
The solution should read data from the Kafka server, get a real-time query based on this data, and react to the occurring anomalies by registering their occurrences and storing them in a place like Docker.
The environment and Kafka producer are ready.
It needs a Spark Structured Streaming programme to process data, create ETL processes, and detect anomalies. (with a .jar and running script), and a consumer.
More information about the data and goals is attached. The data will be sent later, as it's too big to upload here.
Data is about Crime statistics in Chicago.
The solution should read data from the Kafka server, get a real-time query based on this data, and react to the occurring anomalies by registering their occurrences and storing them in a place like Docker.
The environment and Kafka producer are ready.
It needs a Spark Structured Streaming programme to process data, create ETL processes, and detect anomalies. (with a .jar and running script), and a consumer.
More information about the data and goals is attached. The data will be sent later, as it's too big to upload here.