Data Engineer

About the role
We are looking for a hands-on Data Engineer to run and maintain production data pipelines on Databricks and AWS. You will keep scheduled processing reliable, resolve incidents, deliver permanent fixes and improve the performance of existing data services.
The role combines L2 and L3 production support with data engineering. You will work with service management, platform teams, data owners and downstream users to keep data accurate and available within agreed service levels and delivery deadlines.

Key responsibilities

• Monitor scheduled jobs, ingestion processes and downstream loads. Identify failures, delays and unusual data volumes, assess the business impact and prioritise action against delivery deadlines.
• Own incidents from investigation through recovery and closure. Diagnose issues in Python, PySpark, SQL, Databricks jobs and AWS integrations, and provide clear progress updates and timely escalation.
• Recover failed or incomplete processing through controlled retries, reruns and backfills. Check dependencies and reconcile outputs to prevent duplicate records, missing data or inconsistent downstream results.
• Investigate data quality issues, schema changes and late or missing source files. Trace problems through pipeline stages and agree corrective action with source owners and data consumers.
• Maintain and improve existing Python, PySpark and SQL code. Deliver defect fixes and small enhancements, using code review and appropriate tests to protect existing processing rules.
• Tune Spark jobs and SQL queries, including joins, partitioning, data skew and memory usage. Review Databricks compute usage and S3 file layouts to reduce processing time and unnecessary cost.
• Troubleshoot S3 access, IAM permissions, encrypted data access and connectivity with platform and security teams. Support runtime and dependency upgrades, controlled releases, rollback plans and post-release validation.
• Automate health checks, reconciliation and routine support tasks. Improve alerts and track job failures, data freshness, processing duration and recurring incidents to guide preventive maintenance.
• Maintain runbooks, dependency maps, root cause analyses and support handovers. Check that new or changed pipelines have monitoring, support ownership and recovery procedures before accepting them into service.

Essential skills and experience

• Proven experience supporting and maintaining production data pipelines, including incident investigation, safe recovery, root cause analysis and permanent remediation.
• Strong Python, PySpark and SQL skills, with the ability to understand unfamiliar code, diagnose defects and make maintainable changes.
• Practical Databricks experience covering notebooks, scheduled jobs, cluster configuration, driver and executor logs, Spark UI and performance troubleshooting.
• Experience with Spark SQL, Delta Lake, Parquet and Hive metastore tables, including schemas, partitions and the relationship between table metadata and underlying files.
• Hands-on AWS experience, particularly S3, IAM and CloudWatch, with an understanding of role-based access, KMS encryption and diagnosing data access failures.
• Understanding of batch and incremental processing, job dependencies, restartability, late-arriving data and safe reprocessing without duplicate results.
• Experience implementing data quality checks and source-to-target reconciliation, and investigating missing, duplicate or incorrect records in large datasets.
• Working knowledge of Git, code reviews, CI/CD and controlled production deployments, including testing, rollback and release validation.
• Experience working within incident, problem and change management processes. Able to prioritise by business impact, maintain clear support records and communicate with technical and non-technical colleagues.

Desirable experience

• AWS Glue, Lambda, Step Functions or other orchestration services; Linux and Shell Scripting; Terraform; and delivery tools such as GitLab CI or Jenkins.
• Monitoring and alerting automation, capacity planning, cloud cost optimisation and disaster recovery exercises for data services.
• Supporting healthcare, public sector or other regulated data environments, including access controls, audit trails and secure handling of sensitive information.
• Transitioning pipelines from project delivery into live support, with practical knowledge transfer and documented operational acceptance.