Data Transfer Validation Flask Web System
Budget: $30 – $250 USD
I am seeking an experienced developer proficient in Docker, Python, Flask, SQL Server, Logstash, Elasticsearch, and Kibana to build a containerized data transfer validation system. The system will consist of multiple containers and facilitate the automated movement of data from SQL Server to Elasticsearch through Logstash, with a web interface for validation.
Project Scope:
Containerized Environment Setup:
Utilize the provided docker-compose.yml file to build and manage the following containers:
SQL Server container (populated with provided SQL scripts)
Logstash container configured with hwam.conf
Elasticsearch and Kibana containers
Python container
Data Population & Transfer:
Populate the SQL Server database automatically using the provided create_v_es_cdm_hwam.sql and insert_sample_data.sql scripts.
Transfer data from SQL Server to Elasticsearch using the provided Logstash configuration hwam.conf.
Flask Web Application Development:
Build a Flask web application to serve as a validation interface.
The web interface should:
Automatically parse a provided Logstash pipeline configuration (hwam.conf).
Connect to the SQL Server instance and list all data fields that are processed according to the configuration.
Connect to Elasticsearch using parameters specified in the configuration.
Validate data transfer by comparing SQL Server data points with their corresponding entries in Elasticsearch.
Deliverables:
Fully functioning Docker environment with all containers correctly communicating and operational.
Flask web interface with the capabilities listed above, ready to accept user input for different Logstash pipeline configurations.
Documentation on the setup, configuration, and use of the system, including how to test data validation through the web interface.
Requirements:
Proficiency in Docker and container orchestration
Strong knowledge of Python and Flask
Experience with Logstash pipelines, Elasticsearch, and Kibana
Familiarity with SQL Server and data querying
Ability to create an intuitive and user-friendly web interface for data validation
Files Provided:
docker-compose.yml
elasticsearch.yml
kibana.yml
hwam.conf (Logstash configuration)
logstash.yml
mssql-jdbc-12.8.1.jar
python_app folder
SQL scripts (create_v_es_cdm_hwam.sql and insert_sample_data.sql)
Project Scope:
Containerized Environment Setup:
Utilize the provided docker-compose.yml file to build and manage the following containers:
SQL Server container (populated with provided SQL scripts)
Logstash container configured with hwam.conf
Elasticsearch and Kibana containers
Python container
Data Population & Transfer:
Populate the SQL Server database automatically using the provided create_v_es_cdm_hwam.sql and insert_sample_data.sql scripts.
Transfer data from SQL Server to Elasticsearch using the provided Logstash configuration hwam.conf.
Flask Web Application Development:
Build a Flask web application to serve as a validation interface.
The web interface should:
Automatically parse a provided Logstash pipeline configuration (hwam.conf).
Connect to the SQL Server instance and list all data fields that are processed according to the configuration.
Connect to Elasticsearch using parameters specified in the configuration.
Validate data transfer by comparing SQL Server data points with their corresponding entries in Elasticsearch.
Deliverables:
Fully functioning Docker environment with all containers correctly communicating and operational.
Flask web interface with the capabilities listed above, ready to accept user input for different Logstash pipeline configurations.
Documentation on the setup, configuration, and use of the system, including how to test data validation through the web interface.
Requirements:
Proficiency in Docker and container orchestration
Strong knowledge of Python and Flask
Experience with Logstash pipelines, Elasticsearch, and Kibana
Familiarity with SQL Server and data querying
Ability to create an intuitive and user-friendly web interface for data validation
Files Provided:
docker-compose.yml
elasticsearch.yml
kibana.yml
hwam.conf (Logstash configuration)
logstash.yml
mssql-jdbc-12.8.1.jar
python_app folder
SQL scripts (create_v_es_cdm_hwam.sql and insert_sample_data.sql)