Developing a ETL Pipeline REST API using Node.js.
Budget: $30 – $250 USD
Looking for a programmer that has experience developing a ETL Pipeline REST API using Node.js.
1. Ability to run 1 to 2 ETL Pipeline simultaneously and store the data in a centralized database.
Must include the running options stated below:
Run Option 1: Run one ETL pipeline from local Database to another local centralized database (PostgreSQL/MSSQL). The second ETL pipeline is from the new centralized database(PostgreSQL/MSSQL) to a centralized GCP/AWS cloud database.
Run Option 2: Run one ETL pipeline from local Database to another local centralized database (PostgreSQL/MSSQL)
Run Option 3: Run one ETL pipeline from local Database to a centralized GCP/AWS Cloud database.
2. The ETL Pipeline must have the ability to connect to multiple local databases (MSSQL and PostgreSQL) and copy selected schema/tables from the databases into a new centralized database (support for PostgreSQL, MSSQL, GCP and AWS).
3. The ETL Pipeline should be programmed with a separate file for configuration and selecting the desired run options/parameters.
4. The ETL Pipeline must have the ability to set extraction interval in the configuration file.
5. If communication to the ETL Pipeline is loss or interrupted, the pipeline must be able to backfill the database to the time that communication was loss.
6. After the data is extracted, each dataset must append with the following information below when loading the data into the new centralized database. The information append will be store as individual column in the new centralized database to allow for filtering, lookup and associating the data to the proper account/customer. The information will be entered and configured in the configuration file.
1. Customer ID:
2. Customer Name:
3. Building Name:
4. System Name:
5. Database Name:
*See attached for diagram
1. Ability to run 1 to 2 ETL Pipeline simultaneously and store the data in a centralized database.
Must include the running options stated below:
Run Option 1: Run one ETL pipeline from local Database to another local centralized database (PostgreSQL/MSSQL). The second ETL pipeline is from the new centralized database(PostgreSQL/MSSQL) to a centralized GCP/AWS cloud database.
Run Option 2: Run one ETL pipeline from local Database to another local centralized database (PostgreSQL/MSSQL)
Run Option 3: Run one ETL pipeline from local Database to a centralized GCP/AWS Cloud database.
2. The ETL Pipeline must have the ability to connect to multiple local databases (MSSQL and PostgreSQL) and copy selected schema/tables from the databases into a new centralized database (support for PostgreSQL, MSSQL, GCP and AWS).
3. The ETL Pipeline should be programmed with a separate file for configuration and selecting the desired run options/parameters.
4. The ETL Pipeline must have the ability to set extraction interval in the configuration file.
5. If communication to the ETL Pipeline is loss or interrupted, the pipeline must be able to backfill the database to the time that communication was loss.
6. After the data is extracted, each dataset must append with the following information below when loading the data into the new centralized database. The information append will be store as individual column in the new centralized database to allow for filtering, lookup and associating the data to the proper account/customer. The information will be entered and configured in the configuration file.
1. Customer ID:
2. Customer Name:
3. Building Name:
4. System Name:
5. Database Name:
*See attached for diagram