LLM Ops Engineer (UK)

As an LLMOps Engineer at Quantios, you will play a foundational role in building and operating the company's first generation of Large Language Model-powered agentic products. You will work closely with AI developers, architects, DevOps engineers, and Product Owners to design, deploy, monitor, and optimise LLM pipelines, RAG architectures, and agent-based systems. This is a hands on role suited to someone who enjoys solving complex technical problems, building scalable AI infrastructure, and shaping early stage best practices.

• Model, Data, and RAG Pipelines
  • Design, implement, and maintain ingestion pipelines for LLM training and retrieval augmented generation (RAG) datasets.
  • Develop and optimise chunking, embedding, enrichment, and indexing processes using LangChain or equivalent frameworks.
  • Manage the lifecycle of prompt templates, embedding models, LLM chains, evaluators, and model configurations.
  • Support experimentation, evaluation, and benchmarking of foundation models, prompts, and retrieval strategies.
• LLM Infrastructure & Operations
  • Build and operate infrastructure for AI components using Azure AI Foundry, Azure OpenAI, Azure App Services, and related cloud services.
  • Implement secure hosting for RAG applications, vector databases, and agent runtimes.
  • Define and maintain CI/CD pipelines for LLM artefacts (datasets, prompts, model configs, evaluation suites) using Azure DevOps.
  • Collaborate with DevOps engineers to support environment provisioning, scalability, reliability, and performance.
• Observability, Quality & Monitoring
  • Establish foundational observability for LLM based systems, including telemetry, latency tracking, cost visibility, and model diagnostics.
  • Monitor and surface signals such as hallucination rates, evaluation scores, retrieval quality, and content safety triggers.
  • Implement automated evaluation pipelines for prompts, responses, and RAG relevance metrics.
  • Ensure LLM quality gates are integrated into CI/CD workflows.
• Security, Governance & Compliance
  • Apply responsible AI principles in line with Quantios' AI and ISMS policies.
  • Ensure privacy, access control, and logging for all model interactions and vector index operations.
  • Support red team style penetration testing for prompt injection, leakage, and model based social engineering risks.
  • Contribute to documenting LLM pipelines, governance patterns, and internal standards.
• Collaboration & Delivery
  • Work with AI developers to integrate LLM and RAG components into product features.
  • Partner with Portfolio Architects to evaluate new AI technologies, patterns, and architectural approaches.
  • Collaborate with Product Owners to shape technical feasibility, performance considerations, and release planning for AI enabled features.
  • Participate in Agile ceremonies, contribute to estimation, and help the team deliver high quality AI capabilities.
• Continuous Improvement & Innovation
  • Stay up to date with emerging tools in LLMOps, RAG optimisation, evaluation methodologies, and vector search technologies.
  • Propose improvements to scalability, model performance, prompt engineering practices, and developer workflows.
  • Contribute to establishing early LLMOps best practices that will scale as the organisation's AI capability grows.

• Bachelor's degree in Computer Science, Software Engineering, Data Engineering, or a related field; or equivalent industry experience.
• 4+ years of experience in software engineering, data engineering, machine learning engineering, or DevOps-preferably within cloud environments.
• Hands on experience with Python and modern AI frameworks (e.g., LangChain, Semantic Kernel, MC based tools, or equivalent).
• Experience operating cloud based AI solutions using Azure AI Foundry, Azure OpenAI, Azure App Services, Azure Storage, or similar services.
• Familiarity with vector databases, embeddings, and retrieval pipelines (Azure AI Search, Pinecone, Chroma, Redis Vector, or similar).
• Strong understanding of CI/CD, version control, and environment management (Azure DevOps preferred).
• Experience with container orchestration using Kubernetes (AKS or equivalent) and containerized deployments.
• Experience with observability tooling and practices (Azure Monitor, logging, tracing, metrics).
• Knowledge of modern front end or service development technologies (React, TypeScript, C#, or equivalent) is beneficial.
• Strong problem solving, analytical, and debugging skills with a passion for building reliable AI driven systems.
• Excellent communication skills and ability to collaborate across multidisciplinary teams