Azure AI Data Engineer

Project description

Designed and implemented a scalable enterprise Lakehouse platform on Azure using Azure Data Factory, Delta Lake, ADLS Gen2, and Databricks. Migrated legacy ETL processes to cloud-native ELT pipelines supporting both batch and near real-time workloads. Established reusable data frameworks, metadata-driven orchestration, data quality validation, and automated CI/CD pipelines.

Responsibilities

• Designed and implemented scalable data pipelines using Azure Data Factory, Databricks, Delta Lake, Python, and SQL.Architected Lakehouse solutions supporting enterprise analytics, AI/ML, and Generative AI workloads.Developed structured and unstructured data pipelines for RAG, vector search, and AI-powered applications.Built reusable frameworks, improving code quality, scalability, and development productivity.Optimized ETL/ELT processes, reducing data latency and improving performance.Implemented CI/CD, automated testing, monitoring, and DataOps best practices.Established data quality, governance, security, and metadata management standards.Collaborated with cross-functional teams to deliver business-driven data and AI solutions.Mentored engineers and led technical design and architecture discussions.Drove cloud modernization initiatives, improving reliability, scalability, and operational efficiency.

SKILLS

Must have

• 5+ years of experienceAzure Data Factory (ADF)DatabricksDelta LakePythonSQL / PL-SQLPySparkAzure Data Lake (ADLS)CI/CD (Azure DevOps)Data ModelingETL/ELT Pipelines

Nice to have

Azure AI SearchMicrosoft FabricKafkaSnowflakeTerraformDocker / KubernetesMicrosoft Purview