Intermediate AI Quality Engineer - Agentic Systems, Stellenbosch

We're partnering with a global health‑tech scale‑up with roots in South Africa, delivering agentic AI systems into real production healthcare environments in the US. This is not an experimental role. These systems are live, scaling, and solving real clinical workflows now.

We're looking for a sharp AI Quality Engineer to own one of the hardest problems in applied AI: How do you make autonomous systems reliable enough for healthcare?

You’ll design the quality engine behind production AI systems — ensuring outputs are accurate, safe, measurable, and trusted by clinicians. If you’re excited by high‑stakes AI, complex systems, and real‑world impact, this is the seat.

WHAT YOU WILL BE BUILDING (AND OWN)

Prompt lifecycle systems, version control, testing, and structured output enforcement

Golden datasets and evaluation frameworks for LLM performance

Automated regression detection across prompts, models, and workflows

Production monitoring, observability, tracing, and root-cause analysis using Phoenix / OpenTelemetry

Human‑in‑the‑loop review pipelines, labelling workflows, and feedback systems

Statistical quality measurement to continuously improve system performance

OUR REQUIRED EXPERTISE

3–5 years software engineering experience in production environments

Exposure to AI / LLM systems deployed beyond experimentation

2+ years in QA, evaluation, reliability, or testing of AI systems

Strong Python skills (FastAPI / async preferred)

Experience with APIs, microservices, and distributed systems

Monitoring and observability experience in modern production environments

Strong analytical thinking with a focus on measurable quality

Master's Degree or PhD in Computer Science, Engineering, Artificial Intelligence, Data Science, or a related technical field

YOUR REWARD

Work at the intersection of AI and healthcare, where reliability truly matters

You'll help build systems clinicians and healthcare operators can trust, with direct impact in the US market

Join a greenfield environment with real ownership, complex technical challenges, and the chance to solve problems most companies have not figured out yet

Competitive salary + Perks

Hybrid/flexible working based out of Stellenbosch, Western Cape

#J-18808-Ljbffr