Snowflake CSV File - Table Creation
Budget: $10 – $30 USD
Let’s say that one of our stakeholders needs to load data from a number of CSV files into a particular table in a Snowflake database. The CSV files are between 50 and 100GB each. The first row of each CSV file contains column headers which match the names of the columns in the database table, though the columns in the CSV files are not necessarily in the same order as in the table. You can sign up for a free trial of Snowflake as well. Exercise:
1. Write a function in Python which takes as arguments a Snowflake connection, the fully qualified name of the table, and a list of the CSV file paths, and loads the data from those CSV files into the table in the Snowflake database. The required signature is as follows:
def load_csvs_to_snowflake_table(
conn: snowflake.connector.connection.SnowflakeConnection,
fully_qualified_table_name: str, csv_file_paths: List[str],
):
2. It’s always a good idea to get a second set of eyes on your code, so imagine you were going to submit this function for peer review. Write some comments to your peer reviewer that give some context on what you were trying to achieve, identify anything notable about your approach, and specify any open questions or concerns you might have.
3. The stakeholder would like to be able to use the function to upload CSVs from a folder on their computer. Write a script in a second file which calls this function, with placeholders where they can add their Snowflake credentials and the path to a folder on their computer containing the CSV files. Provide some instructions on how to use thescript, including any prerequisite steps such as installing libraries or packages. Assume the end user has Python 3.8 installed already.
4. In words, describe any changes you’d make to this function for use in production.
P.S. You might find most of the code on here.
https://github.com/daanalytics/Snowflake/blob/master/python/F1createDemo.py
https://medium.com/snowflake/from-csv-to-snowflake-46889eabbad6
1. Write a function in Python which takes as arguments a Snowflake connection, the fully qualified name of the table, and a list of the CSV file paths, and loads the data from those CSV files into the table in the Snowflake database. The required signature is as follows:
def load_csvs_to_snowflake_table(
conn: snowflake.connector.connection.SnowflakeConnection,
fully_qualified_table_name: str, csv_file_paths: List[str],
):
2. It’s always a good idea to get a second set of eyes on your code, so imagine you were going to submit this function for peer review. Write some comments to your peer reviewer that give some context on what you were trying to achieve, identify anything notable about your approach, and specify any open questions or concerns you might have.
3. The stakeholder would like to be able to use the function to upload CSVs from a folder on their computer. Write a script in a second file which calls this function, with placeholders where they can add their Snowflake credentials and the path to a folder on their computer containing the CSV files. Provide some instructions on how to use thescript, including any prerequisite steps such as installing libraries or packages. Assume the end user has Python 3.8 installed already.
4. In words, describe any changes you’d make to this function for use in production.
P.S. You might find most of the code on here.
https://github.com/daanalytics/Snowflake/blob/master/python/F1createDemo.py
https://medium.com/snowflake/from-csv-to-snowflake-46889eabbad6