Data Platform Architect - SC Cleared

Data Platform Architect - SC Cleared - London (1-2 days and roemte) - £640 - 682

Essential Skills & Experience
Minimum five years of experience in data platform or data architecture roles, with hands-on delivery of large-scale Databricks implementations.
Deep, demonstrable expertise in Databricks, including Lakehouse architecture, Delta Lake, Unity Catalog, Databricks Workflows, and cluster management.
Proven experience designing and delivering cloud-native data platforms on AWS, including S3, IAM, VPC, and integration with Databricks.
Strong understanding of medallion architecture, data lakehouse patterns, and the management of large-scale structured and semi-structured data.
Experience working within UK public sector organisations, with a clear understanding of the associated governance, compliance, and stakeholder complexities.
Familiarity with data security frameworks relevant to government, including Government Security Classifications and GDPR.
Strong grasp of platform engineering principles, including infrastructure-as-code, CI/CD, and DevSecOps practices applied to data platform delivery.
Hands-on experience with AWS infrastructure, with strong knowledge of the services commonly integrated with Databricks including S3, IAM, Glue, Kinesis, and CloudWatch.
Excellent communication skills, with the ability to engage both technical teams and senior stakeholders effectively.

Role

Platform Architecture & Design

• Own and lead the end-to-end technical architecture of the Databricks Lakehouse platform on AWS, from infrastructure design through to data access patterns.
• Define the foundational architectural components including landing zones, medallion architecture (Bronze, Silver, Gold), compute cluster strategy, and storage design using AWS S3.
• Establish platform design standards and patterns that enable scalable, consistent, and governed data engineering across all teams building on the platform.
• Ensure the platform architecture supports both batch and streaming workloads, with appropriate separation of concerns between ingestion, processing, and serving layers.

AWS Infrastructure & Integration

• Design and govern the AWS infrastructure underpinning the Databricks platform, including S3 storage architecture, IAM roles and policies, VPC configuration, and network security controls.
• Define integration patterns between Databricks and wider AWS services, including ingestion from source systems via AWS Glue, EventBridge, or Kinesis where applicable.
• Ensure infrastructure-as-code principles are applied throughout, with platform components managed and deployed via Terraform or equivalent tooling.
• Work closely with cloud infrastructure and DevOps teams to establish CI/CD pipelines for platform component deployment and configuration management.

Security, Governance & Compliance

• Embed security-by-design across all platform architecture decisions, ensuring data is protected at rest and in transit in line with public sector information assurance requirements.

Desirable Skills & Experience
Databricks Certified Data Engineer Professional or equivalent Databricks certification.
Experience with adjacent data tooling such as dbt, Apache Spark, or AWS Glue.
Familiarity with platform observability tooling and cost management practices for large-scale Databricks deployments.
Previous experience delivering a greenfield data platform build or Legacy migration within a government or regulated environment.
Understanding of UK public sector data and security requirements, including SC clearance processes and Government Security Classification handling.