Import data from a TXT to a DB in MySQL with PANDAS, PANGRES, ETC - BIGDATA

Job ID: 31130585

Budget: $250 – $750 USD

I have a Python script within a shared server of 1GB ram, which imports data from a TXT file (1GB) to a MySQL DB. The objectives of this project are the following:

1.1. Check the server where the script will run and install the necessary pip modules for the script to function.
1.2. Optimize the existing Python script to provide so that the whole data will be imported without errors. With ssh tunnel is working but when I run it in the server for lack of memory crashes.
1.3. Add to the code the validation of whether the record exists in the database, if it exists, to verify if it has undergone a change, if so, to update it, if it does not exist to insert it. - Maybe can be use upsert.
1.4. Create a "cron job" or some procedure to run the web application in Python on a daily or on-demand basis.
1.5. Add something to the script that can alert about errors of the import of the data.

***********************
import pandas as pd
#import timeit
import os
import mysql.connector as mariadb
import pymysql
from sqlalchemy import create_engine
names =
************************

Thank you very much for your time.
Greetings.
Linux, MySQL, PHP, Python, Web Hosting
Related categories: Python Linux Software Architecture MySQL Elasticsearch