Deploy Marqo AI Pipeline on Azure
Budget: £20 – £250 GBP
Seeking an experienced DevOps engineer or cloud architect to deploy a Marqo AI pipeline on Microsoft Azure. The project involves setting up a scalable infrastructure for the Marqo AI codebase, which includes a Vespa database and an embeddings pipeline, as outlined in the provided guide: Scalable Infrastructure for Marqo AI. The codebase to be used is available at Marqo AI GitHub.
The goal is to create a production-ready, scalable, and secure deployment on Azure, leveraging Kubernetes and other Azure services as recommended in the guide or based on best practices.
Scope of Work:
Configure and deploy the Marqo AI pipeline on Azure, ensuring compatibility with the Vespa database and embeddings pipeline.
Set up a Kubernetes-based infrastructure on Azure Kubernetes Service (AKS) following the suggested architecture in the provided guide.
Implement CI/CD pipelines for automated deployments and updates.
Ensure scalability, high availability, and security best practices (e.g., role-based access control, network policies, and monitoring).
Integrate necessary Azure services (e.g., Azure Blob Storage, Azure Monitor, or others) for optimal performance and observability.
Provide documentation for the deployment process and infrastructure setup.
Deliverables
Fully Deployed Marqo AI Pipeline on Azure:
A working instance of the Marqo AI pipeline, including the Vespa database and embeddings pipeline, deployed on Azure Kubernetes Service (AKS).
Verification that the deployment aligns with the architecture described in the provided guide.
CI/CD Pipeline Configuration:
A configured CI/CD pipeline (e.g., using GitHub Actions, or similar) for automated builds, testing, and deployments.
Scripts and configuration files for pipeline setup.
Infrastructure as Code (IaC):
IaC templates (e.g., Terraform) for provisioning and managing the Azure resources (AKS, networking, storage, etc.).
A reproducible setup for the entire infrastructure.
Security and Monitoring Setup:
Implementation of security best practices (e.g., RBAC, network security, and secrets management).
Integration with Azure Monitor or equivalent for logging, metrics, and alerts.
Documentation:
Detailed documentation covering the deployment process, infrastructure setup, and maintenance procedures.
Instructions for scaling the deployment and troubleshooting common issues.
Testing and Validation:
A test suite or validation process to confirm the pipeline’s functionality, including data ingestion, embedding generation, and query performance.
A report summarising the deployment’s performance and scalability.
The goal is to create a production-ready, scalable, and secure deployment on Azure, leveraging Kubernetes and other Azure services as recommended in the guide or based on best practices.
Scope of Work:
Configure and deploy the Marqo AI pipeline on Azure, ensuring compatibility with the Vespa database and embeddings pipeline.
Set up a Kubernetes-based infrastructure on Azure Kubernetes Service (AKS) following the suggested architecture in the provided guide.
Implement CI/CD pipelines for automated deployments and updates.
Ensure scalability, high availability, and security best practices (e.g., role-based access control, network policies, and monitoring).
Integrate necessary Azure services (e.g., Azure Blob Storage, Azure Monitor, or others) for optimal performance and observability.
Provide documentation for the deployment process and infrastructure setup.
Deliverables
Fully Deployed Marqo AI Pipeline on Azure:
A working instance of the Marqo AI pipeline, including the Vespa database and embeddings pipeline, deployed on Azure Kubernetes Service (AKS).
Verification that the deployment aligns with the architecture described in the provided guide.
CI/CD Pipeline Configuration:
A configured CI/CD pipeline (e.g., using GitHub Actions, or similar) for automated builds, testing, and deployments.
Scripts and configuration files for pipeline setup.
Infrastructure as Code (IaC):
IaC templates (e.g., Terraform) for provisioning and managing the Azure resources (AKS, networking, storage, etc.).
A reproducible setup for the entire infrastructure.
Security and Monitoring Setup:
Implementation of security best practices (e.g., RBAC, network security, and secrets management).
Integration with Azure Monitor or equivalent for logging, metrics, and alerts.
Documentation:
Detailed documentation covering the deployment process, infrastructure setup, and maintenance procedures.
Instructions for scaling the deployment and troubleshooting common issues.
Testing and Validation:
A test suite or validation process to confirm the pipeline’s functionality, including data ingestion, embedding generation, and query performance.
A report summarising the deployment’s performance and scalability.