Validation Engineer - AI Infrastructure

Are you looking for an exciting new opportunity?

Join a fast-scaling UK AI infrastructure owner-operator building sovereign, renewable-powered GPU compute environments. By converting legacy brownfield and energy sites into modern data center facilities, the organization delivers high-density bare-metal APIs, Kubernetes clusters, and low-carbon AI capacity to enterprise, healthcare, financial, and public-sector clients across the UK.

The company is seeking a Validation Engineer to join its Product team and own validation end-to-end across Product, Engineering, and Operations. The successful candidate will design and build automated preflight suites and performance benchmarks for multi-node GPU clusters, validating platform reliability, hardware deployments, and infrastructure releases prior to enterprise customer deployment.

Responsibilities:

• Design and build zero-dependency, Python-based preflight tools that validate multi-node distributed initialisation, master-to-worker rendezvous routing and GPU-to-CPU process affinity before large training workloads launch
• Write and maintain Infrastructure as Code to provision, configure, test and tear down complex bare-metal and containerised compute environments
• Build adaptive test orchestration harnesses that autonomously detect environment drift and analyse the impact of platform changes
• Automate cluster-level and high-performance storage benchmarking, aggregating results to establish node-to-node throughput limits
• Script low-level checks for server topology, PCIe link speeds, HBM memory status, secure boot parameters and firmware performance via BMC, IPMI or Redfish
• Embed validation suites into CI/CD pipelines, build observability pipelines that isolate slow or degrading nodes, and lead root-cause analysis on platform regressions and interconnect bottlenecks alongside Platform, Network and Data Centre teams

Skills/Must have:

• Proven, hands-on validation or benchmarking experience in enterprise-scale environments. This is a specialist validation role, not a platform or systems engineering role with some testing exposure.
• Strong Python (zero-dependency tooling, test harnesses, low-level system checks), Infrastructure as Code for bare-metal and containerised environments, and Kubernetes
• Validating multi-node distributed systems; hardware interfaces (BMC, IPMI, Redfish); CI/CD integration of validation suites; observability pipelines for diagnostics and hardware metrics
• A rigorous, evidence-led mindset; the confidence to hold the line on release readiness; and a collaborative approach to cross-functional root-cause analysis

Desirable Skills:

• Validation of infrastructure optimised for large-scale model training, including GPU-to-CPU affinity and rendezvous routing
• Deep understanding of high-throughput network fabrics, traffic patterns and congestion control in multi-node GPU clusters
• Experience building cluster-level and storage benchmarking suites to isolate throughput limits
• Experience designing intelligent test systems that detect environment drift autonomously

Benefits:

• Performance related bonus

Salary:

• £100,000 to £110,000+ base (subject to client sign-off, see note below)