QA Engineer

• *Key Responsibilities**

• Design and execute comprehensive test strategies specifically for AI/ML models, LLM\-based applications, and data pipelines
• Develop automated test frameworks for model validation, regression testing, and performance benchmarking
• Evaluate model outputs for accuracy, consistency, relevance, hallucination, and bias across diverse inputs and use cases
• Test RAG (Retrieval\-Augmented Generation) pipelines, chatbots, recommendation systems, and other AI\-driven features
• Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds for AI systems
• Build and maintain evaluation datasets, ground truth sets, and adversarial test cases for comprehensive model validation
• Monitor models in production for drift, degradation, and anomalous behavior; implement monitoring solutions as needed
• Validate data quality, data pipelines, and feature stores that feed AI systems to ensure data integrity
• Document defects, edge cases, and failure patterns specific to AI behavior with actionable insights
• Ensure AI systems meet ethical, fairness, and compliance standards through bias audits and explainability checks

• *Required Skills \& Qualifications**

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field
• 3 \- 6 years of professional QA experience, with at least 1 \- 2 years in AI/ML quality assurance
• Strong proficiency in **Python** for test automation and data analysis
• Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith)
• Hands\-on experience with testing tools such as Pytest, Selenium, Postman, or similar platforms
• Solid understanding of the ML lifecycle — training, validation, deployment, and monitoring phases
• Knowledge of data quality tools and pipeline testing (e.g., Great Expectations, dbt tests)
• Strong analytical and inquisitive mindset with the ability to challenge model outputs critically
• Excellent documentation and communication skills with the ability to articulate complex technical concepts
• Collaborative approach and ability to work effectively with data science, engineering, and product teams

• *Nice to Have**

• Experience with prompt engineering and red\-teaming LLMs
• Familiarity with MLOps platforms such as MLflow, SageMaker, or Vertex AI
• Knowledge of vector databases and embedding quality evaluation
• Understanding of AI safety, responsible AI principles, and fairness frameworks
• Experience with A/B testing and shadow deployment strategies
• Knowledge of CI/CD pipelines and DevOps practices in ML environments

Application Question(s)

• Do you have a Bachelor's or Master's degree in Computer Science, Engineering, or a related field?
• Do you have at least 1\-2 years of professional experience specifically in AI/ML quality assurance?\*
• Are you proficient in Python for test automation and data analysis?
• Do you have hands\-on experience with at least one LLM evaluation framework such as RAGAS, DeepEval, Promptfoo, or LangSmith?
• Have you worked with data quality tools or pipeline testing tools such as Great Expectations or dbt tests?
• How many years of professional QA experience do you have?
• Which of the following testing tools have you used? (Select all that apply)\*

Pytest

Selenium

Postman

None of the above

• What is your experience level with the ML lifecycle including training, validation, deployment, and monitoring phases?
• How many times have you changed firms during your professional career?\*

0\-2 times

3\-5 times

More than 5 times

• What is your current location or relocation preference?\*

Mumbai\-based

Bangalore\-based

Open to relocation

Other location

• Salary budget is INR 2500000 \- 3000000Max, what are your expectations?

Work Location: In person