AI - Quality Engineer

AI - Quality Engineer 
In our India organization, we are looking for a strongly motivated QA Engineer who will have a fundamental role in enabling business and industry innovators to collaboratively build and execute a successful plan, transforming market opportunities into marketplace advantages. As a Software Test Engineer, you will be participating in Qualification of Web and/or Client-Server Applications. These Applications are used by our customers to design, develop, produce and market their Products and/or Services to end consumers. You will work directly with Product Define, Development, Testing, Project Management teams to ensure we are delivering high-quality, innovative products with an emphasis on speed to market and sustainability. You will lead in execution and team adoption of testing, automation tools & best practices.

ROLES AND RESPONSIBILITIES

• Ensure reliability, safety, and performance of AI-driven applications, specifically those utilizing Large Language Models (LLMs).

• Develop and maintain frameworks to test AI applications, with a focus on AI/ML models.
• Design and implement tests for AI-powered applications, ensuring functionality, accuracy, and performance.
• Design & implement evaluation frameworks to benchmark model behavior, audit for hallucinations and bias, and integrate autonomous testing agents into CI/CD pipelines.
• Synthetics data generation using LLM/ Prompt Engineering for Data Generation
• Utilize frameworks like Pytest to ensure thorough and automated testing. Or identifying right framework for test,
• Collaborate with cross-functional teams to understand AI/ML models and apply appropriate testing techniques.
• Work with Python to build and integrate test solutions for AI-related features.

• Stay updated with emerging RAG (Retrieval-Augmented Generation) models, LLM-as-a-judge and best practices for AI testing.

   

Technologies you have a chance to work with:

• You will be developing AI test frameworks to automate the process of testing applications/software.
• Python (Must Have for AI/ML workflows)
• Good understanding of AI/LLM concepts (Deep understanding of Generative AI, RAG architectures, prompt engineering, and vector databases.)
• Evaluation Frameworks: Hands-on experience with tools like Ragas, DeepEval, Promptflow, LangSmith, or Gantry
• Statistical Thinking: Ability to apply scoring rubrics and understand variance, distributions, and confidence intervals in probabilistic outputs (Good to have)
• Basic JavaScript (Good to have)