Big Data Project
Budget: $1,500 – $3,000 USD
The goal is to have a single file with the inventory with 100M plus entries, that is updated from 200+ sources multiple times per day. All sources will have data formatted in the same format. Each job will be sent to RabbitMQ. Input will be in a tab delimited file.
Database using open source, ideally Postgres. One database with the inventory, another for managing RabbitMQ jobs and logging.
RabbitMQ to process this, use standard web ui to view history and success of jobs. Need to reference the source. Logging will be tracked in the RabbitMQ database. Otherwise, there will be no UI to view this inventory data.
Code in C#
There should also be an option to automatically files from an ftp or folder on the disk. The first column in the tab delimited file will indicate the source.
Please see attached more detailed spec and sample file
Database using open source, ideally Postgres. One database with the inventory, another for managing RabbitMQ jobs and logging.
RabbitMQ to process this, use standard web ui to view history and success of jobs. Need to reference the source. Logging will be tracked in the RabbitMQ database. Otherwise, there will be no UI to view this inventory data.
Code in C#
There should also be an option to automatically files from an ftp or folder on the disk. The first column in the tab delimited file will indicate the source.
Please see attached more detailed spec and sample file