Lead AI Testing Engineer

Lead AI Testing Engineer
We are looking for a Lead AI Testing Engineer to drive the quality strategy for AI-driven and data migration initiatives spanning knowledge graphs, ontologies and RAG-based LLM features. You will define and execute QA practices across data ingestion, graph layers, rule evaluation engines and AI/LLM components while building automation frameworks that scale evaluation across complex, data-heavy systems.
Responsibilities
– Design and automate evaluation of RAG-based and LLM-driven features including grounding, answer accuracy, determinism/reproducibility, precision, recall and hallucination rate
– Build test harnesses to scale evaluation beyond human-in-the-loop processes
– Create data-driven test suites for underwriting rules and validate rule execution via SPARQL evaluator, arithmetic/threshold logic and multi-condition scenarios
– Verify ontology schema correctness and instance accuracy against source data, perform reasoner consistency checks and SHACL validation
– Define and drive end-to-end QA strategy covering data ingestion, ontology/graph layer, rule evaluation engine, AI/LLM layer and integration with underwriting solutions
– Establish quality gates, acceptance criteria and test coverage models
– Maintain automation test suites across knowledge graph, data ontology layer, backend services and frontend layers
– Validate end-to-end flows across ontology, rule engine and decision output, perform contract testing and cross-system data consistency validation
– Integrate test suites into CI/CD pipelines and define release quality gates
– Champion automation-first and shift-left quality practices
– Leverage agentic AI and Gen AI tooling in testing and framework development
Requirements
– 8+ years of experience in software development, testing automation and DevOps
– Proficiency in Python test automation using pytest or equivalent, scripting and AI/ML library integration
– Expertise in testing Knowledge Graph or Data Ontology solutions using OWL2, RDF and SPARQL
– Skills in SHACL validation and graph databases such as GraphDB, including ontology validation, entity/relationship integrity, reasoner consistency checks and graph query correctness
– Proven ability to define and drive QA strategy independently for complex data-heavy systems, including building automation frameworks and implementing quality gates
– Skills in data validation using SQL and graph query languages such as SPARQL and Cypher, along with understanding of deterministic vs probabilistic systems
– Background in backend and API testing (REST) covering data validation, integration and E2E testing
– Proficiency in frontend web application test automation using Playwright or equivalent Python-compatible framework
– Hands-on experience using coding agents and agentic development daily
– Demonstrated experience testing and evaluating RAG-based Gen AI / LLM applications including grounding, answer accuracy and hallucination/determinism checks
– Applied knowledge of Gen AI / LLM evaluation frameworks and metrics such as precision, recall, criteria recall and efficiency, with proven ability to automate Gen AI/RAG evaluation at scale
– Experience in Agile delivery using Git and Azure DevOps or JIRA
– English proficiency at B2 level or higher
Nice to have
– Skills in semantic search testing and evaluation
– Familiarity with Vector Database integration and retrieval validation
– Background in data science covering ML concepts, data pipelines or data engineering collaboration
– Proficiency in API tooling such as Postman and Bruno
– Experience with quality-gate implementation in CI/CD pipelines such as Azure DevOps or Jenkins
We offer
– We gather like-minded people:
– Engineering community of industry professionals
– Friendly team and enjoyable working environment
– Flexible schedule and opportunity to work remotely within Poland
– Chance to work abroad for up to 60 days annually
– Business-driven relocation opportunities
– We provide growth opportunities:
– Outstanding career roadmap
– Leadership development, career advising, soft skills, and well-being programs
– Certification (GCP, Azure, AWS)
– Unlimited access to LinkedIn Learning, Get Abstract, Cloud Guru
– English classes
– We cover it all:
– Stable income (Employment Contract or B2B)
– Participation in the Employee Stock Purchase Plan
– Benefits package (health insurance, multisport, shopping vouchers)
– Strategically located offices featuring entertainment and relaxation zones, table tennis and football, free snacks, fantastic coffee, and more
– Referral bonuses
– Corporate, social and well-being events
– Please, note:
– The set of bonuses might vary based on the role you apply for – specifics will be discussed with our recruiter during the general interview.
– We will reach out to selected candidates exclusively.
EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.