AI QA & Evaluation Lead

Jodayn is seeking an AI QA & Evaluation Lead to establish and manage enterprise-grade quality assurance, validation, and evaluation processes for AI-powered systems and intelligent automation platforms.

The successful candidate will be responsible for ensuring the reliability, consistency, security, explainability, and governance alignment of AI-generated outputs and AI-driven workflows.

This role is critical in maintaining enterprise trust, operational quality, and compliance standards across AI initiatives delivered to government and enterprise clients.

The ideal candidate must possess strong experience in enterprise quality assurance, automation testing, AI evaluation methodologies, and governance-oriented validation frameworks.

Requirements

AI Evaluation & Validation

• Design and maintain AI testing and evaluation frameworks
• Develop benchmarking datasets and output validation methodologies
• Evaluate AI-generated responses for consistency, accuracy, and reliability
• Define confidence scoring and quality measurement approaches

AI Quality Assurance

• Implement automated regression testing for AI workflows
• Monitor prompt performance and behavioural consistency
• Detect model drift and output degradation
• Validate multilingual AI outputs including Arabic and English content

Governance & Compliance

• Ensure AI outputs align with enterprise governance standards
• Support auditability and traceability of AI-generated results
• Implement review and approval workflows for AI systems
• Support compliance validation activities for enterprise deployments

Security & Risk Testing

• Conduct AI security and prompt injection testing
• Validate protections against unsafe outputs and misuse scenarios
• Support secure AI deployment and operational practices

Reporting & Continuous Improvement

• Produce quality assurance reports and evaluation summaries
• Recommend improvements to prompts, workflows, and AI models
• Support continuous optimisation of AI solution quality

Required Qualifications

Education

• Bachelor’s degree in Computer Science, Information Systems, Software Engineering, or related field

Experience

• Minimum 5 years experience in enterprise quality assurance or software testing
• Experience with automation testing frameworks
• Experience evaluating AI-powered systems preferred
• Experience in enterprise or regulated environments preferred

Required Technical Skills

Quality Assurance

• Enterprise QA methodologies
• Automation testing frameworks
• API testing
• Regression testing methodologies
• Test case development

AI Evaluation

• Large language model evaluation
• Prompt engineering concepts
• AI benchmarking methodologies
• AI governance and validation concepts

Security & Governance

• Risk assessment concepts
• AI security testing
• Auditability and traceability principles

Preferred Skills

• Experience in enterprise AI initiatives
• Exposure to regulated industries
• Experience with bilingual AI validation
• Familiarity with enterprise governance frameworks

Core Competencies

• Exceptional attention to detail
• Strong analytical thinking
• Structured problem-solving capability
• High quality and governance awareness
• Strong documentation and reporting capability