Databricks Architect - SC Cleared

Databricks Architect - SC Cleared

We're seeking an experienced Databricks Architect to lead the design and implementation of a next-generation data platform within a secure government environment. The role focuses on architecting, migrating, and optimising data platforms using Databricks on AWS (Azure experience also valued).
You'll work closely with enterprise architecture, data engineering, and cloud infrastructure teams to design and deliver a scalable, compliant, and high-performing data ecosystem that supports advanced analytics and AI workloads.

Key Responsibilities

• Lead the architecture, design and implementation of Databricks-based data platforms in cloud environments (AWS or Azure).
• Define migration strategies from Legacy data environments to Databricks.
• Design data ingestion, transformation, and orchestration workflows to meet performance and security standards.
• Provide technical leadership on data lake, data warehouse, and streaming architecture best practices.
• Collaborate with Enterprise Architects, Data Engineers, and Security teams to align solutions with cloud and governance standards.
• Optimise performance, cost, and scalability of Databricks clusters and data pipelines.
• Contribute to architecture documentation, patterns, and reference models for reuse across the programme.

• Proven experience as a Data/Solution Architect specialising in Databricks (AWS preferred; Azure considered).
• Demonstrable experience in end-to-end implementation or migration to Databricks.
• Deep understanding of cloud-native data architectures, particularly in AWS (S3, Glue, EMR, Lambda) or Azure (ADF, Synapse, Data Lake).
• Strong experience with data modelling, ETL/ELT design, and orchestration tools (Airflow, DBT, or similar).
• Proficiency in Python, SQL, and Spark for data transformation and performance tuning.
• Experience with CI/CD pipelines and Infrastructure as Code (Terraform, CloudFormation, or Bicep).
• Understanding of data security, governance, and compliance frameworks within regulated or government environments.

• Databricks Certified Data Engineer/Architect or equivalent.
• Familiarity with Snowflake, Redshift, or Synapse.
• Experience supporting AI/ML platform integration and advanced analytics use cases.