Databricks Engineer (SC Cleared)

Job Title: Databricks Engineer (SC Cleared)

Location: London (Hybrid)

Contract: 6 months | £550-600/day (Inside IR35)

Start Date: ASAP (Dec 2025/Jan 2026)

About the Role

We are seeking an experienced Databricks Engineer with active SC Clearance to join a high profile GovTech project. You will play a key role in supporting a large scale data migration and developing a cutting edge Azure Databricks platform for economic data modelling, analysis, and forecasting. This is an exciting opportunity to work on a mission critical program that influences national economic insights.

Key Responsibilities

• Design, develop, and optimize data pipelines using Azure Databricks and PySpark.
• Support data migration activities from Legacy systems to modern cloud based architecture.
• Implement Delta Lake for efficient data storage and retrieval.
• Collaborate with data scientists and analysts to enable advanced economic modelling and forecasting.
• Ensure data security and compliance within a government environment.
• Troubleshoot performance issues and optimize Spark jobs for scalability.
• Contribute to CI/CD processes and automation for data workflows.

Required Skills & Experience

• Active SC Clearance (mandatory).
• Strong hands on experience with Azure Databricks and Spark.
• Proficiency in PySpark, SQL, and Python (Scala desirable).
• Experience with Azure services: Data Lake, Synapse, Key Vault.
• Proven track record in ETL pipeline development and data migration.
• Familiarity with Delta Lake and data governance best practices.
• Knowledge of CI/CD tools (eg, Azure DevOps, Git).

If this looks like the role for you, apply now.