Databricks LLM Data Ingestion App
Budget: $30 – $250 AUD
I need a concise Databricks-native app whose core job is data ingestion. The workflow is straightforward: a user drops one or more files—CSV, JSON, Parquet, or Excel—into the workspace, and the app auto-detects schema, validates basic data quality rules, and lands the clean result in Delta tables. All data will come in through file uploads; no databases or external APIs are involved at this stage.
I would like the code structured so that a large-language-model helper can be bolted on later (for example, to auto-generate column descriptions or propose transformation logic). That future extension means clean, well-commented notebooks, modular functions, and clear separation between ingest, validation, and storage components.
Deliverables:
• A Databricks notebook (Python / PySpark) or repos project that implements the upload using Autoloader, schema inference, validation, and Delta write steps.
• Lightweight UI or widgets so non-technical users can trigger the workflow without touching code.
• README inside the workspace outlining setup, cluster requirements, and how to plug in an LLM endpoint when ready.
• Brief hand-off session or video walk-through so I can maintain and extend the pipeline myself.
Acceptance criteria: I can upload a sample file, run the app with one click, and find a validated table in Delta Lake with the expected schema and row count, plus a confirmation log entry.
If you have recent experience with Databricks Repos, DBFS file handling, and Delta Lake best practices, I’d love to see your approach.
I would like the code structured so that a large-language-model helper can be bolted on later (for example, to auto-generate column descriptions or propose transformation logic). That future extension means clean, well-commented notebooks, modular functions, and clear separation between ingest, validation, and storage components.
Deliverables:
• A Databricks notebook (Python / PySpark) or repos project that implements the upload using Autoloader, schema inference, validation, and Delta write steps.
• Lightweight UI or widgets so non-technical users can trigger the workflow without touching code.
• README inside the workspace outlining setup, cluster requirements, and how to plug in an LLM endpoint when ready.
• Brief hand-off session or video walk-through so I can maintain and extend the pipeline myself.
Acceptance criteria: I can upload a sample file, run the app with one click, and find a validated table in Delta Lake with the expected schema and row count, plus a confirmation log entry.
If you have recent experience with Databricks Repos, DBFS file handling, and Delta Lake best practices, I’d love to see your approach.