A Pipeline to load parquet files into snoflake
Budget: $10 – $35 USD
Generally, if you can answer me below questions, I will give the project to the person who answers the requirements.
Using APIs is the usual way of sending data to customers, but one of my clients will also be sending some complementary data through files. In normal circumstances, the format of the files is either CSV or JSON that is ingested into Snowflake using a custom-made ELT application that runs on AWS Lambda. Still, in this case, the file format is parquet that the platform currently doesn’t have any support for.
How would you go about implementing support for ingesting these parquet files into
Snowflake?
Out clients has some more complex objects in their data that need to be unrolled and added on top of the data
model so that it’s available for the web application and analytics parts of the application.
How would you go about implementing these additional data fields in the model? Would you extend the different tables in the current model or create new tables specifically for these data fields?
Using APIs is the usual way of sending data to customers, but one of my clients will also be sending some complementary data through files. In normal circumstances, the format of the files is either CSV or JSON that is ingested into Snowflake using a custom-made ELT application that runs on AWS Lambda. Still, in this case, the file format is parquet that the platform currently doesn’t have any support for.
How would you go about implementing support for ingesting these parquet files into
Snowflake?
Out clients has some more complex objects in their data that need to be unrolled and added on top of the data
model so that it’s available for the web application and analytics parts of the application.
How would you go about implementing these additional data fields in the model? Would you extend the different tables in the current model or create new tables specifically for these data fields?