Databricks & PySpark: SharePoint Data Loader

Job ID: 38803780

Budget: $30 – $250 USD

I'm looking for an experienced Databricks and PySpark expert to build a utility or function that can retrieve data from SharePoint and load it into a Databricks DataFrame. The function should take parameters such as SharePoint path, file name, and format, and return a DataFrame with the loaded data.

Key Requirements:
- The function should support loading files in CSV format.
- The connection between Azure Databricks and the SharePoint site must be configured correctly and documented in detail.
- Configuration of security, secrets, network settings, and/or service principles will be necessary.
- The function and its configurations must work seamlessly in my corporate environment.

Security Configuration:
- All configurations should utilize Service Principals for security or Oauth.

Network Settings:
- The function should be compatible with my current use of a Virtual Private Network (VPN).

Ideal skills for this job include:
- Extensive experience with Azure Databricks.
- Proficiency in using PySpark.
- Knowledge of SharePoint data retrieval.
- Ability to configure and document security settings using Service Principals.
- Understanding of working within a VPN.
Related categories: Cloud Computing Sharepoint Azure