Databricks Native capabilities - Ingestion + LLM Integration
Budget: $30 – $250 AUD
I am setting up a fresh Databricks workspace and need a rock-solid, reusable ingestion pipeline for structured data.
Here’s how I picture the flow:
[Refer to the High Level Architecture.pdf]
• A Databricks app which talks to the Large Language Model (OpenAI GPT-4 or similar) model and identifies if the file is an existing or a new ingestion pattern
• If it is new pattern the app will ask for the configurations like bronze table name and silver table name and store the information in a config table and start the AutoLoader for ingestion
• If it is existing pattern a Databricks Auto Loader or Delta Live Tables pipeline pulls structured sources into a bronze layer, handles schema evolution, and lands everything in Delta format.
• A transformation step promotes the data to silver, where basic quality checks, column standardisation and lightweight enrichment happen.
• From there, a Large Language Model (OpenAI GPT-4 or similar) is invoked inside a notebook or a Spark UDF to generate metadata, summaries or semantic tags that will live alongside each table.
Deliverables
1. Databricks notebooks (or .dbc export) that stand up the bronze → silver flow, fully parameterised.
2. Sample code that calls an LLM -- Databricks’ ai_generate functions—adding output to a Delta column.
3. README or markdown walkthrough with setup steps, cluster/runtime specs, and how to swap in different API keys or models.
4. A short validation script that proves new structured files dropped in cloud storage are auto-ingested and annotated within one run.
Acceptance criteria
• Running the notebook end-to-end ingests a sample CSV, creates bronze and silver Delta tables, and appends an LLM-generated summary column.
• All jobs succeed on Databricks Runtime 13.x or above with no manual edits.
• Code is clean, commented, and uses native Databricks features wherever possible.
If this matches your expertise, let’s get the pipeline flowing.
Here’s how I picture the flow:
[Refer to the High Level Architecture.pdf]
• A Databricks app which talks to the Large Language Model (OpenAI GPT-4 or similar) model and identifies if the file is an existing or a new ingestion pattern
• If it is new pattern the app will ask for the configurations like bronze table name and silver table name and store the information in a config table and start the AutoLoader for ingestion
• If it is existing pattern a Databricks Auto Loader or Delta Live Tables pipeline pulls structured sources into a bronze layer, handles schema evolution, and lands everything in Delta format.
• A transformation step promotes the data to silver, where basic quality checks, column standardisation and lightweight enrichment happen.
• From there, a Large Language Model (OpenAI GPT-4 or similar) is invoked inside a notebook or a Spark UDF to generate metadata, summaries or semantic tags that will live alongside each table.
Deliverables
1. Databricks notebooks (or .dbc export) that stand up the bronze → silver flow, fully parameterised.
2. Sample code that calls an LLM -- Databricks’ ai_generate functions—adding output to a Delta column.
3. README or markdown walkthrough with setup steps, cluster/runtime specs, and how to swap in different API keys or models.
4. A short validation script that proves new structured files dropped in cloud storage are auto-ingested and annotated within one run.
Acceptance criteria
• Running the notebook end-to-end ingests a sample CSV, creates bronze and silver Delta tables, and appends an LLM-generated summary column.
• All jobs succeed on Databricks Runtime 13.x or above with no manual edits.
• Code is clean, commented, and uses native Databricks features wherever possible.
If this matches your expertise, let’s get the pipeline flowing.