Senior Quality Engineer - AI

About StarRez StarRez is the global leader in student housing software, providing innovative solutions for on and off-campus housing management, resident wellness and experience, and revenue generation. Trusted by 1,400+ clients across 25+ countries, StarRez supports more than 4 million beds annually with its user-friendly, all-in-one platform, delivering seamless experiences for students and administrators. With offices in the United States, Australia, the UK, and India, StarRez blends the robust capabilities of a global organization with the personalized care and service of a trusted partner. The Role: We are looking for a Senior Quality Engineer - AI to help shape how StarRez evaluates, validates, and improves AI-powered product experiences. You will play a pivotal role in elevating our quality practices for AI-powered product experiences and tooling. You will be a product expert within our engineering organization, deeply understanding workflows and customer outcomes. This role will balance hands-on individual contributor responsibilities with leading, influencing, and coaching your peers. You’re someone who is passionate about seeing systems holistically and is eager to drive the highest standards of accuracy, safety, usefulness, and reliability. You will help define what "good" AI output means for StarRez, build repeatable evaluation systems, coach reviewers and subject matter experts, and turn subjective feedback into measurable product improvement. You will work at the intersection of AI engineering, product, and domain knowledge — partnering with engineers, product managers, and SMEs to raise the bar on how StarRez evaluates, monitors, and improves AI tools. Key Responsibilities: Lead or contributed to end-to-end AI quality and evaluation strategy for product experiences involving LLMs, RAG, prompts, tool-use, or agent workflows. Provided quality-focused input during ticket grooming, discovery, and feature discussions, with clear guidance on testability, observability, scalability, reliability, product resilience, customer usability, and operational readiness. Championed shift-left quality practices by influencing or mentoring developers and cross-functional partners on evaluation-driven development, clear acceptance criteria, early validation, and fast feedback loops. Coached or calibrated human reviewers, subject matter experts, or product teams by running calibration sessions, reviewing patterns, improving rubric clarity, and converting domain knowledge into reliable evaluation criteria. Contributed to the evolution of development or quality pipelines, including automated and human evaluation workflows, regression monitoring after model upgrades, prompt changes, RAG updates, or agent workflow changes. Define practical AI quality standards, rubrics, and evaluation criteria that help teams assess AI outputs consistently across StarRez product workflows. Partner with engineering and product teams to identify AI quality risks early, including risks related to grounding, hallucination, tool-use behaviour, retrieval quality, customer impact, and safe escalation. Build or support repeatable evaluation approaches that help teams compare prompt, model, RAG, data, or workflow changes before release. Translate AI failure patterns into actionable product and engineering improvements, such as prompt changes, retrieval improvements, source-data fixes, tool safeguards, UX changes, or escalation-flow updates. Use metrics, reviewer feedback, quality trends, and operational signals to support release readiness and ongoing product improvement. Required Experience: Quality Assurance or Quality Engineering experience in a SaaS B2B or B2B2C company. Proven track record as a senior individual contributor with a strong quality mindset and the ability to influence technical and product direction. Excellent technical communication and collaboration skills, with the ability to guide engineering peers, product partners, and non-technical sta…