AI Quality Engineer

Acads360 India Private LimitedTS, IN18h ago

Required Skills

agileapiawsazureci/cddockergcpgithub actionsgraphqljavascriptjenkinsjirakubernetesllmmicroservicesnlpplaywrightpythonrestscrumsqltypescript

Job Description

*Role at a Glance****Role:** AI Quality Engineer (Playwright Automation)**Experience:** 3–6 Years**Engagement:** Full\-Time \| On\-Site / Hybrid**Domain:** AI\-First Enterprise Procurement SaaS**Core Stack:** Playwright · TypeScript/JavaScript · Python · REST APIs · SQL**Reporting To:** Head of Quality Engineering**Team Context:** Embedded in AI Product Squads **THE OPPORTUNITY**

We are looking for a seasoned AI Quality Engineer with 3–6 years of hands\-on experience to join our Quality Engineering team. This is not a conventional QA role. You will be embedded in our AI product squads, designing and executing intelligent test strategies that validate not just functionality, but the behaviour, accuracy, fairness, and reliability of AI\-driven features on a live enterprise SaaS platform. **You won't just test software. You'll safeguard the intelligence powering enterprise procurement for the world's leading organizations.** **KEY RESPONSIBILITIES****AI \& Intelligent Feature Testing*** Design and execute end\-to\-end test strategies for AI\-driven procurement features including supplier recommendations, spend classification, contract risk analysis, and demand forecasting.

Develop robust test suites using Playwright to validate AI model outputs, UI behaviours, and data flows across the full application stack.
Define and implement test cases for non\-deterministic AI features — including boundary testing, edge case exploration, hallucination detection, and bias assessment.
Build evaluation frameworks to measure and track model accuracy, consistency, recall, and precision across releases.
Collaborate with AI/ML engineers and data scientists to validate training data quality, model versioning, and inference pipeline correctness.

Similar Jobs

Browse all jobs

Upload resume for AI match score

Job Overview

Job type: Full-time
Work mode: On-site
Location: Hyderabad
Posted: 18h ago
Source: Indeed

LinkedIn 𝕏 / Twitter

*Automation Framework Engineering*** Architect, build, and scale a Playwright\-based automation framework (TypeScript/JavaScript) supporting UI, API, and component\-level testing.
Design Page Object Models (POM), fixtures, and reusable test utilities that enable rapid test authoring across multiple product areas.
Implement self\-healing selectors and intelligent wait strategies to handle AI\-driven dynamic content and async rendering.
Integrate test suites into CI/CD pipelines (GitHub Actions, Jenkins, or Azure DevOps) to enable shift\-left, continuous quality enforcement.
Maintain test result dashboards, flakiness tracking, and failure triage workflows to ensure signal\-to\-noise discipline in the automation suite.

*API \& Data Validation*** Validate REST and GraphQL API contracts through Playwright's API testing capabilities, ensuring AI service integrations behave correctly under all conditions.
Write SQL queries to verify data integrity across procurement entities — POs, invoices, supplier records, contracts — pre\- and post\-AI processing.
Design data\-driven test scenarios leveraging realistic procurement datasets to stress\-test AI features at scale.
Perform response schema validation, latency benchmarking, and error boundary testing on AI inference APIs.

*Performance, Security \& Reliability*** Lead performance testing initiatives to ensure AI\-powered features meet SLA requirements under realistic enterprise load conditions.
Collaborate with security teams to perform penetration and vulnerability testing on AI API endpoints and data pipelines.
Design and execute chaos and resilience tests to validate system behaviour during AI model degradation or outage scenarios.
Monitor production AI model drift and work with MLOps teams to define quality gates for model redeployment decisions.

*Quality Leadership \& Culture*** Champion a quality\-first engineering culture by embedding testing discipline into sprint ceremonies, design reviews, and definition of done.
Mentor junior QA engineers on Playwright, automation best practices, and AI\-specific testing methodologies.
Produce comprehensive test plans, risk assessments, and quality reports for product leadership and enterprise stakeholders.
Continuously evaluate and introduce next\-generation testing tools, AI\-assisted test generation platforms, and quality intelligence solutions.

*KEY SKILLS****Playwright \& Automation (Must\-Have)*** 3–6 years of hands\-on Playwright experience with TypeScript or JavaScript — portfolio or GitHub evidence strongly preferred.
Deep expertise in Playwright features: network interception, storage state, fixtures, parallel execution, trace viewer, and visual comparison.
Strong understanding of test design patterns: POM, screenplay pattern, data\-driven, and BDD with Cucumber or Gherkin.
Experience building and maintaining large\-scale automation suites (500\+ tests) with low flakiness and high maintainability.
Proficiency in CI/CD integration — configuring Playwright in GitHub Actions, Jenkins, Azure Pipelines, or equivalent.

*AI \& ML Quality Expertise (Must\-Have)*** Demonstrated experience testing AI/ML\-powered features — NLP, recommendation engines, classification models, or generative AI outputs.
Understanding of AI quality concepts: hallucination, model drift, data bias, fairness metrics, confidence scores, and output consistency.
Ability to design evaluation harnesses that objectively measure AI model quality against ground\-truth datasets.
Familiarity with prompt engineering and testing LLM\-based features for accuracy, robustness, and safety.

*API, Data \& Backend Testing*** Strong API testing skills using Playwright API project, Postman, or REST Assured — including auth flows, pagination, and schema validation.
Solid SQL proficiency for data validation, query\-based test assertions, and understanding procurement data models.
Experience with contract testing (Pact or similar) and microservices testing in cloud\-native environments.
Hands\-on with log analysis, distributed tracing, and monitoring tools (Datadog, Grafana, ELK Stack) for root cause analysis.

*Cloud, DevOps \& Tooling*** Experience testing SaaS applications deployed on AWS, Azure, or GCP — including cloud\-specific failure scenarios.
Comfort working within Agile / Scrum / SAFe delivery frameworks — sprint planning, refinement, and retrospectives.
Proficiency with bug tracking and test management tools: Jira, TestRail, Zephyr, or equivalent.
Exposure to containerised environments (Docker, Kubernetes) and how they impact test environment management.

*Soft Skills \& Professional Impact*** Exceptional analytical thinking — ability to decompose complex AI system behaviours into testable, measurable quality criteria.
Clear, confident communication — able to articulate quality risks, model limitations, and test coverage to engineering and business stakeholders.
Self\-directed and ownership\-driven — proactively identifies gaps, raises quality concerns early, and drives resolution.
Collaborative and empathetic team player who raises the quality consciousness of the entire product squad.

*QUALIFICATIONS*** Bachelor's or Master's degree in Computer Science, Information Technology, Software Engineering, or a related technical discipline.
3–6 years of progressive experience in software quality engineering, with a strong automation\-first track record.
Demonstrable, hands\-on Playwright experience across multiple projects — production\-grade automation portfolios are strongly valued.
Prior experience testing AI/ML\-powered products, SaaS platforms, or enterprise cloud applications is highly preferred.
Certifications in software testing (ISTQB), cloud platforms (AWS/Azure), or AI/ML fundamentals are a definite advantage.
A growth mindset — someone who actively stays current with the rapidly evolving landscape of AI testing tools, techniques, and standards.

*WHAT WE OFFER*** The rare opportunity to build quality engineering for an AI\-native product at the frontier of enterprise procurement technology.
Direct ownership of test strategy and automation architecture — this is not a maintenance role; it's a builder role.
A collaborative, intellectually rigorous engineering culture where quality is treated as a product feature, not a gate.
Competitive compensation commensurate with experience, performance\-linked growth, and comprehensive benefits.
Dedicated learning budget: certifications, conferences, AI research access, and internal AI upskilling programmes.
Exposure to cutting\-edge AI/ML engineering, cloud\-native architecture, and global enterprise customer contexts.
Flexible hybrid working arrangements with a focus on outcomes over hours.

AI Quality Engineer

Required Skills

Job Description

Similar Jobs

Job Overview

Share