Job ID: JOB_ID_2735
Role Overview
We are seeking a highly skilled and detail-oriented AI Tester to join our innovative team in Cincinnati, OH. As AI technologies continue to reshape the enterprise landscape, the need for rigorous validation of machine learning models and generative AI applications has never been greater. This role is pivotal in ensuring that our AI-enabled applications are not only functional but also safe, reliable, and ethically sound. You will be responsible for testing probabilistic outputs, identifying hallucinations in LLMs, and ensuring that our solutions meet the highest standards of responsible AI usage.
Key Responsibilities
- AI & Functional Testing: Design and execute comprehensive test scenarios specifically tailored for AI-driven application features. You will validate AI outputs against complex business rules and acceptance criteria, ensuring that the system behaves as expected across a wide range of inputs, including edge cases and ambiguous scenarios.
- Generative AI Validation: Evaluate responses generated by Large Language Models (LLMs) for accuracy, relevance, consistency, and stability. A critical part of this role involves identifying hallucinations, incorrect reasoning, and undesired outputs that could impact user trust or safety.
- Prompt Engineering & Context Handling: Validate prompt behavior and context window management. You will analyze how different prompt structures affect response variability and ensure the AI maintains context throughout multi-turn conversations.
- Data & Model Testing: Verify the quality and coverage of test data used for AI validation. You will support the validation of model performance metrics such as accuracy, precision, recall, and F1 scores, working with both structured and unstructured data.
- Regression & Drift Monitoring: Perform regression testing to ensure AI behavior remains acceptable after model updates or fine-tuning. You will identify model or output drift in production-like environments and raise risks when behavior changes beyond acceptable thresholds.
- Responsible AI & Ethics: Validate AI systems for fairness, explainability, and bias risks. You will ensure that all outputs comply with security, privacy, and ethical usage expectations, supporting human-in-the-loop review processes where necessary.
Technical Requirements
- 4-10 years of experience in software testing, quality engineering, or validation roles.
- Strong foundation in software testing methodologies and the defect lifecycle.
- Working knowledge of AI and machine learning concepts, including the difference between deterministic and probabilistic systems.
- Experience with generative AI characteristics, such as non-deterministic outputs and context sensitivity.
- Ability to work with structured and unstructured data sets for validation purposes.
- Exposure to test automation concepts, scripting, and API testing (Postman, RestAssured).
- Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or a related field.
Professional & Behavioral Skills
- Strong attention to detail and critical-thinking skills.
- Ability to challenge AI outputs constructively and objectively.
- Clear written and verbal communication skills for documenting defects and findings.
- Comfort working in evolving and ambiguous problem spaces typical of AI development.
- Strong sense of ownership for quality and risk management.
Special Requirements
Visa status and details required; 6+ Months Contract-to-Hire (C2H).
Compensation & Location
Salary: $135,000 – $185,000 per year (Estimated)
Location: Cincinnati, OH
Recruiter / Company – Contact Information
Recruiter / Employer: Siri InfoSolutions Inc
Email: avinash.singh@siriinfoinc.com
Recruiter Notice:
To remove this job posting, please send an email from
avinash.singh@siriinfoinc.com with the subject:
DELETE_JOB_ID_2735