Senior QA & Evaluation Engineer

Job Title:

Senior QA & Evaluation Engineer

Key Skills:

Test Automation, .NET, API Testing, CI/CD, Automated Testing, LLM Evaluation, AI Testing, Test Frameworks

Experience:

4+ years of experience.

Location:

Costa Rica, Peru, Colombia, Bolivia.

Mode:

Remote.

We at Coforge are hiring Mid-Senior QA & Evaluation Engineer (N-5) with the following skill set.

Key Responsibilities

• · Design and maintain automated test suites covering unit, integration, API, and end-to-end testing.
• · Integrate automated testing into CI/CD pipelines and establish automated suites as release quality gates.
• · Build evaluation frameworks for AI capabilities, including scored rubrics, LLM-as-a-judge approaches, and normalized pass thresholds.
• · Build and maintain held-out evaluation datasets, measure AI behavior against ground truth, and report defect escape and evaluation results as part of delivery metrics.
• Required Skills & Qualifications
• · 4+ years of experience in Quality Engineering with a strong focus on test automation.
• · Hands-on experience with automated testing frameworks, preferably within the .NET ecosystem.
• · Strong experience with API testing and automated test development.
• · Experience integrating automated test suites into CI/CD pipelines.
• · Understanding of testing strategies for non-deterministic and probabilistic AI systems.
• · Ability to design evaluation approaches based on scored thresholds rather than exact-match assertions.
• · Experience working with test data, ground truth datasets, and automated validation.
• · Strong analytical and problem-solving skills.
• · Strong working proficiency in English.
• · Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field, or equivalent professional experience.
• Preferred Skills: · Direct experience with LLM evaluation, including rubric design, LLM-as-a-judge, golden/held-out datasets, or tools such as Azure AI Foundry Evaluators, promptfoo, Ragas, or DeepEval.
• · Experience testing information extraction or classification pipelines using accuracy-based measurements.
• · Experience with regulated-industry QA environments where test evidence and results must support formal review.
• · Familiarity with AI/ML testing methodologies and evaluation frameworks.
• Posted On: 24-09-2026
• At Coforge, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.