Developing a Script for Downloading, Unpacking, Creating a Database, Populating Data from an Archive into a Database, Using JSON Files, in Python, with PostgreSQL
Budget: $30 – $250 USD
1. Introduction:
A script needs to be developed in the Python programming language to automate the following actions: downloading an archive containing JSON data, unpacking the archive, creating a PostgreSQL database, loading data from the archive into this database, and updating the database when the script is run again.
2. Task Description:
The script should perform the following steps:
2.1. Downloading the Archive:
The script should have the capability to download an archive containing JSON data from a specified link. It should handle possible errors and interruptions during the download to ensure successful retrieval.
2.2. Unpacking the Archive:
After downloading the archive, the script should unpack its contents into a specified directory. The archive may contain multiple JSON files.
2.3. Creating a Database:
The script should create a new PostgreSQL database or connect to an existing one. The database structure should be created based on the data from the JSON files. Each JSON file should be represented as a separate table, and the fields of the JSON objects should define the table structure.
2.4. Populating Data:
The script should analyze the unpacked JSON files and load data from them into the corresponding tables in the PostgreSQL database. It should handle duplicates and ensure data integrity.
2.5. Data Update:
When the script is run again, it should check for the existence of the database. If the database exists, the script should update the data from the archive, adding only new records and updating existing ones if necessary.
3. Script Requirements:
The script should be written in the Python programming language.
The script should use the psycopg2 library for working with the PostgreSQL database.
It should handle exceptions and errors during the download, unpacking, and data update processes.
The script should have parameters for configuring paths, links, database connection parameters, and other settings.
4. Expected Results:
It is expected that the developed script will successfully execute all the specified steps and update the database when run again, adding new data from the archive.
A script needs to be developed in the Python programming language to automate the following actions: downloading an archive containing JSON data, unpacking the archive, creating a PostgreSQL database, loading data from the archive into this database, and updating the database when the script is run again.
2. Task Description:
The script should perform the following steps:
2.1. Downloading the Archive:
The script should have the capability to download an archive containing JSON data from a specified link. It should handle possible errors and interruptions during the download to ensure successful retrieval.
2.2. Unpacking the Archive:
After downloading the archive, the script should unpack its contents into a specified directory. The archive may contain multiple JSON files.
2.3. Creating a Database:
The script should create a new PostgreSQL database or connect to an existing one. The database structure should be created based on the data from the JSON files. Each JSON file should be represented as a separate table, and the fields of the JSON objects should define the table structure.
2.4. Populating Data:
The script should analyze the unpacked JSON files and load data from them into the corresponding tables in the PostgreSQL database. It should handle duplicates and ensure data integrity.
2.5. Data Update:
When the script is run again, it should check for the existence of the database. If the database exists, the script should update the data from the archive, adding only new records and updating existing ones if necessary.
3. Script Requirements:
The script should be written in the Python programming language.
The script should use the psycopg2 library for working with the PostgreSQL database.
It should handle exceptions and errors during the download, unpacking, and data update processes.
The script should have parameters for configuring paths, links, database connection parameters, and other settings.
4. Expected Results:
It is expected that the developed script will successfully execute all the specified steps and update the database when run again, adding new data from the archive.