Senior Software Engineer, Observability

About the Team

The Observability team is part of Roku's Cloud Technology Infrastructure organization, ensuring the services and workloads running on our platform have robust observability. Our mission is to advance Roku's observability platform, which ingests terabytes of data daily and manages hundreds of millions of active series. We build our own metrics stack in Go and Rust on top of open-source technologies, tailored to Roku's scale and needs.

We collaborate closely with the Cloud Platform team to build and scale foundational infrastructure (Kubernetes, Istio, Envoy, operators) and the observability layer built on CNCF-supported projects. We integrate AI tools and agents across our engineering lifecycle to design, write, review, and test code, automate operational tasks, and accelerate troubleshooting and incident response.

About the Role

As a Senior Software Engineer on the Observability team, you will help design, develop, and scale the platform used across Roku engineering teams to alert and monitor thousands of services. You will write Go and Rust to build and evolve our metrics stack on top of open-source technologies and work hands-on with tools such as Prometheus, Grafana, Thanos, Loki, Tempo, and OpenTelemetry.

What You'll Be Doing

• Building Our Metrics Stack: Design and build high-performance components of our metrics stack in Go and Rust to handle terabytes of data daily and hundreds of millions of active series.
• AI-Driven Engineering: Use AI tools and agents for daily development and operational tasks (code writing, reviewing, testing, operational automation, troubleshooting, incident response).
• Scaling and Innovating: Enhance and optimize Roku's observability stack while introducing Roku-specific features and AI-assisted capabilities.
• Tooling and Standardization: Partner with the Platform team to standardize and improve observability tools into a cohesive solution.
• Stability and Performance: Ensure high stability, performance, and reliability across the platform.
• Automation and Usability: Build automation around observability tools to simplify onboarding (dashboards, alerts, traces).
• Open Source Contributions: Contribute new tools and improvements back to the CNCF ecosystem.
• Collaboration and Engagement: Partner with engineering teams across Roku to improve the developer experience.
• Technical Leadership: Lead design discussions, mentor engineers and interns, and help guide technical strategy.

Qualifications

• Significant experience (typically around 8 years) in Infrastructure Engineering and/or DevOps, with a focus on observability platforms.
• Hands-on experience using AI tools and agents in daily development/operations, and practical experience building with Large Language Models (tool/function calling, agents, MCP, RAG, evaluating LLM output).
• Solid programming experience (typically around 4 years) in Go and/or Rust, with an interest or background in performance-sensitive systems.
• Proven experience designing and operating large-scale observability platforms handling metrics, logs, and traces.
• Expertise in deploying, operating, and extending open-source observability tools at scale (Prometheus, Grafana, Thanos, Loki, Tempo, Cortex, Mimir, or ELK).
• Familiarity with open standards such as OpenTelemetry and OpenMetrics.
• Deep understanding of Kubernetes and Istio architectures.
• Hands-on experience with cloud providers such as AWS, GCP, and/or Azure.
• Strong ability to engage with stakeholders to deliver tailored observability solutions.
• B.S. or M.S. degree in Computer Science, Engineering, or equivalent experience.