Employer listing
Lead Product Data Scientist, Agents
Klaviyo
Boston, Ma · on site · full time · Lead
$172,000 - $258,000
About the Team
Klaviyo's Product Data Science team shapes how we understand, measure, and improve product and business performance. We partner closely with Product, Engineering, Finance, and Go-to-Market teams to turn complex data into actionable insights that inform product strategy, customer experience, and long-term growth.
As Klaviyo continues to expand its AI-powered capabilities, intelligent agents are becoming central to how businesses engage with their customers:
- Composer: Klaviyo's AI marketing agent that helps marketers analyze, build, and execute marketing strategies.
- Customer Agent: Enables brands to engage shoppers and answer questions across chat, email, SMS, and WhatsApp.
About the Role
We're looking for a Lead Product Data Scientist, Agents to own and advance Klaviyo's measurement and data science strategy across Composer and Customer Agent. In this hands-on technical leadership role, you will define how we measure the adoption, engagement, quality, cost, and business impact of our AI agents, establishing the analytical foundations that create meaningful customer value.
Responsibilities
- Own agent measurement end to end across Composer and Customer Agent, connecting adoption, engagement, agent performance, customer value, and revenue impact.
- Partner with Product and Engineering to establish meaningful measures of agent effectiveness, task completion, customer outcomes, failure rates, and quality vs. cost trade-offs.
- Build scalable agent evaluation frameworks combining conversation data, agent traces, tool-call outcomes, human evaluation, and LLM-based classification.
- Measure causal impact and incremental value using experimentation and advanced quasi-experimental methods (difference-in-differences, matched comparisons, staged rollouts, synthetic controls).
- Partner with Finance and Product teams to evaluate usage-based pricing, packaging, credit economics, model costs, and unit economics.
- Establish trusted metrics, consistent measurement methodologies, and governed data models.
- Leverage coding agents, LLMs, and modern tools to accelerate SQL development, data modeling, dashboard creation, and analytical workflows.
- Build self-service analytics capabilities empowering cross-functional stakeholders.
- Partner with Engineering and Analytics Engineering to build instrumentation, data pipelines, and dbt models.
- Lead technical direction of agent analytics, review methodologies, and mentor other data scientists.
- Influence executive decisions by translating complex analytical findings into clear, actionable recommendations.
Qualifications
Required
- 10+ years of experience in data science, product analytics, applied statistics, or a related quantitative discipline, including recent hands-on experience measuring AI, machine learning, or LLM-powered products in production.
- Deep expertise in product measurement, experimentation, statistical modeling, and causal inference (including quasi-experimental design).
- Experience evaluating AI systems in production (eval frameworks, conversation analysis, human labeling, LLM-as-judge methodologies).
- High proficiency in SQL and Python or R, with experience developing production-quality analytical solutions and familiarity with dbt.
- Proven technical leadership setting direction on complex analytical initiatives and mentoring data scientists.
- Strong product and business intuition regarding customer engagement, usage-based pricing, and unit economics.
- Excellent communication skills for technical and non-technical audiences.
- Degree in Statistics, Mathematics, Computer Science, Economics, Finance, Engineering, another quantitative discipline, or an MBA with a quantitative emphasis (or equivalent hands-on experience in applied data science).
Nice to Have
- Experience measuring and optimizing AI agents, conversational AI, or LLM-powered products in B2B SaaS.
- Hands-on experience with agent observability and evaluation platforms such as Braintrust, LangSmith, or Arize.
- Experience with consumption-based pricing, credit models, AI infrastructure costs, or LLM unit economics.
- Experience building governed metrics, semantic layers, metric trees, or standardized measurement frameworks.
- Experience applying classification or clustering to large volumes of unstructured text or customer conversations.
- Familiarity with marketing attribution or marketing technology / ecommerce SaaS.
Benefits
Total compensation package includes participation in the company's annual cash bonus plan, equity, sign-on payments, and a comprehensive range of health, welfare, and wellbeing benefits based on eligibility.