Senior-level Lead Databricks Engineer

We are seeking a senior-level Lead Databricks Engineer to drive a large-scale Azure-to-Databricks transformation within a complex public sector environment. The role combines deep technical architecture capability with strong stakeholder engagement and governance oversight.

You will lead discovery, architecture design, migration planning, security hardening, and post-migration optimisation for an enterprise Azure Data Science Platform and Databricks-based MLS (Machine Learning Service) environment.

Strong stakeholder management is essential, this is a transformation journey, not just a technical implementation.

Discovery & Assessment

Produce a comprehensive Discovery and Assessment Report
Conduct Azure tenant and subscription reviews
Identify configuration drift, governance gaps, and security risks
Assess workloads, integrations, and dependencies

Architecture & Design

Develop a Target Architecture Design Document (Databricks-based MLS)
Define landing zones, network topology, and security architecture
Design governance model using Unity Catalog
Produce high- and low-level architecture documentation
Define runtime upgrade and environment standardisation strategy

Migration Planning & Execution

Deliver a Migration Plan and Runbook, Lead migration of;

• Azure SQL Databases, Storage Accounts, Virtual Machines (assessment and right-sizing), App Services and Function Apps, Azure Data Gateway/Power BI Gateway
• Plan and execute;
  Databricks workspace setup, Cluster and pool configuration, Unity Catalog implementation, Service principal reconfiguration,App registration provisioning (AD/Entra ID)

Cloud Infrastructure & Networking

Hands-on expertise required in;
Azure Portal environment review and governance setup, VNet architecture: Subnets, Peering, NSGs, Private Endpoints, Private DNS Zones

Security & Compliance

Monitoring, Optimisation & Governance

Databricks Expertise Required

Deep, hands-on experience with; Azure Databricks, Unity Catalog, Workspace, cluster, and pool management, Runtime upgrades and life cycle management, Private link connectivity, Secure enterprise integration,CI/CD integration (eg, Azure DevOps), Service principal RBAC elevation via automation Scripting

Azure Expertise Required

Advanced experience across; Microsoft Azure, Microsoft Entra ID (formerly Azure AD), Azure SQL Database migration, App Registrations and Service Principals, Key Vault (IAM-based model), Application Gateway & WAF, Azure Monitor & Log Analytics, Subscription and tenant-level governance

Stakeholder Management Expectations

• Engage senior technical and non-technical stakeholders

• Translate technical architecture into executive-level reporting

• Lead workshops and architecture reviews

• Manage transformation roadmaps

• Support procurement, risk, and compliance conversations

Public sector engagement experience strongly preferred.

Desirable Experience; AI/ML platform architecture within Databricks MLS, Public sector governance frameworks, Large-scale cloud modernisation programmes, Enterprise data platform design, Security-cleared environments

Security Clearance would be a distinct advantage.

Knowledge of AI and MLS with Databricks would make you a standout candidate.