Senior Staff Machine Learning Engineer, Data & Eval

About the Role

In this Senior Staff Machine Learning Engineering role, you will set technical direction and lead execution for ML evaluation and the end-to-end data flywheel powering Airbnb Assistance Engineering (e.g., assistive agents, issue resolution, and tooling). Your work will define how quality is measured, how feedback turns into learning signals, and how models and products are continuously improved safely and efficiently. You will partner closely with product, engineering, design, and operations to build evaluation systems connecting offline metrics to online outcomes.

Responsibilities

• Define evaluation strategy and success metrics for GenAI systems, aligning offline evaluation with online business and customer experience outcomes.
• Build and scale evaluation frameworks (golden sets, synthetic data, automated regressions, rubric-based grading, LLM-as-judge) with controls for bias, drift, and reliability.
• Design the data flywheel: instrumentation, feedback collection, data quality checks, labeling strategy, dataset versioning, and governance for continuous improvement.
• Lead cross-functional quality initiatives across product, ops, and engineering, establishing quality standards and actionable evaluation workflows.
• Develop and productionize pipelines for dataset creation, model monitoring, evaluation-at-scale, and continuous testing (pre-deploy and post-deploy).
• Drive technical decisions and architecture for evaluation and data infrastructure, balancing speed, rigor, cost, and safety.

Qualifications

Minimum Qualifications

• PhD in Computer Science, Mathematics, Statistics, or related technical field (or equivalent practical experience).
• 10+ years of experience building, testing, and shipping ML/AI systems end-to-end; including 2+ years of experience with GenAI/LLM systems in production.
• 5+ years leading large, ambiguous technical initiatives as a senior individual contributor, influencing roadmap and engineering/science direction across teams.
• Deep expertise in evaluation methodology (offline/online alignment, metric design, human-in-the-loop evaluation, A/B testing, power analysis, regression testing).
• Hands-on experience with GenAI systems, including orchestration, retrieval, tool calling, and memory.
• Experience building data pipelines and quality systems (labeling workflows, dataset curation, versioning, monitoring, and governance).
• Solid ML fundamentals and best practices (model selection, training/serving, monitoring, reliability, and model lifecycle management).

Preferred Qualifications

• Experience applying ML/AI to customer support workflows (e.g., agent assist, classification/routing, resolution recommendation, QA).
• Experience building robust evaluation platforms for agent behavior validation, safety/guardrails, and continuous improvement.
• Proven ability to take evaluation and data flywheel work from incubation to production, iterating quickly while maintaining scientific rigor.