Staff Software Engineer, Backend (Observability)

About the Role

The Observability team builds the platform every Affirm engineer relies on to understand the behavior and performance of our systems. Our platform ingests 200+ TB/day of metrics, logs, and traces from hundreds of services across 80+ Kubernetes clusters, and when a user's checkout fails, our data is how engineers find out why.

We’re looking for a Staff Software Engineer to be the technical lead for Observability. You’ll spend most of your time hands-on while defining the roadmap, driving cross-functional projects, mentoring a team of six engineers, and partnering closely with infrastructure and product teams.

What You'll Do

• Own the architecture of Affirm’s telemetry pipeline, balancing reliability, query performance, and cost at 200+ TB/day.
• Design log sampling strategies that significantly reduce volume and cost without losing the signal engineers need during incidents.
• Shape how observability fits into agentic development, so that AI coding agents and code review catch instrumentation gaps before code ships.
• Set technical strategy for your team on a year-long time scale, connecting it with critical, business-impacting projects.
• Collaborate across teams in the product development lifecycle with product management, design, and analytics to manage technical sustainability, risks, and trade-offs.
• Act as a force-multiplier for your team through definition and advocacy of technical solutions and operational processes.
• Take ownership of your team’s operations and availability by ensuring appropriate monitoring, triage rotations, playbooks, policies, testing, and alerting are in place.
• Foster a culture of quality and ownership by setting code review and design standards and advocating for them through writing and tech talks.
• Help develop talent on your team by providing feedback, guidance, and leading by example.

What We Look For

• 8+ years of experience designing, developing, and launching backend systems at scale using languages like Python or Kotlin.
• Extensive track record of developing highly available distributed systems using technologies like AWS, MySQL, Flink, and Kubernetes.
• Experience building or operating high-volume telemetry systems (metrics, logs, and/or traces) using technologies like OpenTelemetry, Kafka, and Terraform.
• A well-informed point of view on modern observability practices such as structured and wide events, sampling, cardinality management, and SLOs.
• Proven track record of driving adoption of platform standards across many engineering teams via tooling and engagement.
• Experience delivering major features, system components, or deprecating functionality through technical and execution planning.
• Comfort moving between low-level language idioms and large-scale system architecture.
• Demonstrated ability to gather and iterate on feedback from engineering and cross-functional peers.
• Strong verbal and written communication skills.
• Bachelor’s degree in a related field or equivalent practical experience.