About the role

Test Lead / Test Manager

Experience Range: 9–12 years

Location: Pune or Bangalore

Role Type: Full-time

1. Position Overview

• Test Lead will drive QA strategy, governance, and delivery for an enterprise Agentic AI Platform supporting insurance workflows. This includes leading a team of testers, defining quality standards for LLM/agent behavior, overseeing safety and compliance, and ensuring predictable, reliable operation of multi agent systems and RAG pipelines.

• Experience in leading AI/ML testing initiatives, managing test teams, ensuring quality and enterprise AI deployments. The Test Lead will be responsible for strategic planning, governance, stakeholder management, and delivery oversight, ensuring that AI solutions are robust, ethical, and production ready.

• Experience in software testing, specializing in Artificial Intelligence (AI) systems and MLOps pipelines. The role involves validating AI/ML models, ensuring robustness of end-to-end machine learning workflows, and driving quality engineering practices across data, models, and deployment environments. The engineer will collaborate with data scientists, ML engineers, and DevOps teams to ensure trustworthy, scalable, and production-ready AI solutions.

• Strong understanding of both manual and automated testing methodologies, with the ability to design, develop, and execute test scripts.

• Responsible for the testing and delivery of parts of a product, in accordance with the customer’s requirements and interpersonal quality norms.

• Strong analytical and problem-solving skills, along with a proactive approach to identify and resolve issues, are essential.

• Responsible for ensuring that the testing process is efficient, effective, and aligned with project goals and timelines.

2. Responsibilities / Job Description

• Leadership & Team Management:

Lead and mentor a team of test engineers working on AI/ML projects.
Define roles, allocate tasks, and ensure timely delivery of testing milestones.
Drive knowledge sharing, training, and capability building in AI testing practices.
Lead a team of agentic AI test engineers and define role expectations.
Build capability in agent testing, adversarial testing, and evaluation techniques.

• Strategic Test Planning & Governance:

Establish quality KPIs and governance models for AI testing.
Ensure compliance with industry standards, data privacy, and AI ethics.
Represent testing in client meetings, providing thought leadership and strategic recommendations.
Define test strategies for agent workflows, tool interactions, and reasoning graphs.
Establish quality benchmarks (grounding %, task success rate, tool-call success rate).

• AI/ML Testing Oversight:

Ensure robustness of AI systems under adversarial and edge-case scenarios.
Govern regression testing for retrained models and drift detection mechanisms.
Oversee validation of agent behaviors, reasoning paths, and multi-agent coordination.
Ensure robust scenario-based, adversarial, and regression testing across releases.

• End-to-End Workflow Validation:

Ensure comprehensive testing of insurance use cases (FNOL, adjudication, underwriting).
Validate tool integrations, event-driven flows, and RAG pipelines.
Sign off on agent readiness for production deployment.

• Stakeholder & Client Engagement:

Collaborate with data scientists, ML engineers, DevOps, and business stakeholders.
Provide clear reporting on test progress, risks, and mitigation strategies.
Drive closure communications and ensure SLA adherence for defect management.

• Innovation & Continuous Improvement:

Introduce accelerators, reusable frameworks, and automation strategies for AI testing.
Evaluate emerging tools and practices in AI quality engineering and MLOps.
Contribute to organizational maturity assessments and transformation initiatives.

• Quality Governance & Compliance:

Ensure compliance with AI ethics, data privacy, and regulatory standards.
Establish KPIs for model quality (precision, recall, F1-score, drift detection).
Document test strategies, results, and provide audit-ready evidence.

• Collaboration & Leadership:

Work closely with data scientists, ML engineers, and DevOps teams to integrate testing into the AI lifecycle.
Mentor junior testers and contribute to building reusable test accelerators.
Represent the testing function in client discussions, providing insights on AI quality assurance.

Lead guardrail validation, safety policy testing, and privacy assessments.
Manage red-team exercises and ensure safe agent behaviors.
Oversee reliability, load, concurrency, and stress testing.
Ensure logging, tracing, and monitoring support auditability and debugging.
Define and implement AI testing frameworks and governance models.
Oversee performance and scalability testing of AI inference services.
Drive defect triage and SLA-based closure across AI projects.
Deliver client-facing reports, dashboards, and closure communications.
Create reusable test assets for AI projects (scripts, datasets, frameworks).
Provide detailed defect analysis and SLA-driven closure.

3. Required Skills

9–12 years of experience in software testing and quality engineering, with at least 3–4 years in AI testing.
Proven leadership in managing test teams and delivery programs.
Strong expertise in Python for automation and AI testing scripts.
Knowledge of Gen AI frameworks (Langchain, Langgraph, CrewAI).
Experience in CI/CD pipelines and DevOps practices.
Strong understanding of model evaluation metrics, drift detection, and monitoring.
Exposure to AI ethics and compliance testing.
Knowledge of cloud-native architectures (AWS, Azure, GCP).
Familiarity with containerization & orchestration (Docker, Kubernetes).