Data Engineer - Python / Databricks

Role Summary

The role is for a hands-on Data Warehouse Python Developer within the Data Warehouse team. The individual will be responsible for the enhancement, optimisation, support, and modernisation of the Enterprise Data Warehouse platform. The role will focus on designing and delivering cloud-based data solutions using Azure Databricks, Microsoft Fabric, SQL Server, and Python while supporting the migration of the existing on-premise SQL DWH to a modern cloud-based Lakehouse architecture.

The role includes responsibility for production support, operational stability, automation, and continuous improvement of business-critical data platforms. The Enterprise Data Warehouse supports critical regulatory, finance, risk, operational, and management reporting requirements across the organisation.

The successful candidate will work closely with business stakeholders, business analysts, developers, infrastructure teams, and support teams to deliver scalable and reliable data solutions. The individual will be expected to actively participate in Agile delivery activities, production support, modernisation initiatives, and cloud transformation programmes. The role requires a strong technical background combined with the ability to understand business requirements and translate them into robust technical solutions.

Key Interfaces

• Data Warehouse Development Team
• Business Stakeholders
• Business Analysts
• Infrastructure and Cloud Engineering Teams
• Production Support Teams
• DBA Teams
• Architecture and Security Teams
• Risk, Finance, Operations, and Regulatory Reporting Functions

The Successful Candidate Will

• Design, develop, and support enterprise data solutions using Azure Databricks and Microsoft Fabric.
• Build, enhance, and optimise cloud-based ETL/ELT pipelines and data integration processes.
• Develop and maintain SQL Server databases, stored procedures, views, and performance-tuned solutions.
• Develop Python-based solutions for data engineering, automation, monitoring, orchestration, and data quality management.
• Drive the modernisation of the Enterprise Data Warehouse from traditional SQL Server-based architecture to a cloud-native Lakehouse architecture.
• Implement and support Medallion (Bronze, Silver, Gold) data models and best practices.
• Design, maintain, and support AutoSys job schedules, JIL scripting, workflow orchestration, and automation solutions.
• Participate in production support and on-call support activities for business-critical systems.
• Investigate, troubleshoot, and resolve production incidents and performance issues.
• Continuously improve operational processes, automation, monitoring, and platform performance.
• Collaborate with business users and technical teams to deliver scalable, resilient, and secure data solutions.
• Ensure adherence to data governance, security, audit, and operational standards.
• Contribute to architectural decisions and strategic cloud transformation initiatives.

Education

• Preferable degree educated in Computer Science, Software Engineering, Information Systems, Data Science, or a related technical discipline.
• Postgraduate qualification in a relevant technology discipline is desirable.

Experience

• Minimum 10 years' experience in Data Warehouse development and SQL Server development.
• Proven track record delivering enterprise-scale Data Warehouse and data engineering solutions.
• Extensive hands-on experience as a Data Warehouse Developer and Python Developer.
• Experience supporting large-scale production environments and business-critical systems.
• Demonstrable experience delivering Data Warehouse modernisation and cloud migration initiatives.

Technical Skills

• Strong hands-on experience with Azure Databricks.
• Strong experience designing and implementing solutions using Microsoft Fabric.
• Advanced SQL Server and T-SQL development skills.
• Strong Python programming and automation skills.
• Strong cloud data engineering and data warehousing experience.
• Experience designing and implementing Lakehouse architectures.
• Experience with Medallion (Bronze, Silver, Gold) data models.
• Experience with AutoSys workload automation, JIL scripting, job scheduling, and dependency management.
• Strong experience in ETL/ELT design and development.
• Strong data modelling and database design skills.
• Experience troubleshooting and tuning complex SQL and data processing workloads.
• Knowledge of CI/CD, DevOps, and automation best practices.

Desirable

• Experience within Financial Services, Banking, or Capital Markets.
• Experience with Power BI and enterprise analytics platforms.
• Experience with Azure Data Factory.
• Microsoft Fabric, Azure, Databricks, or r
• Ability to work independently and take ownership of technical delivery.
• Ability to balance project delivery, production support, and strategic initiatives.
• elated cloud certifications.