Senior Software Engineer, Observability

About the Team

The Observability team is part of Roku's Cloud Technology Infrastructure organization. We ensure the services and workloads running on our platform have robust observability. We build our own world-class metrics stack in Go and Rust on top of open-source technologies, processing terabytes of data daily and supporting hundreds of millions of active series. We work closely with the Cloud Platform team to build and scale foundational infrastructure (Kubernetes, Istio, Envoy) and the observability layer.

AI is integrated throughout our engineering lifecycle to design, write, review, and test code, automate operational work, and speed up troubleshooting and incident response.

About the Role

As a Senior Software Engineer on the Observability team, you'll help design, develop, and scale the platform used across Roku engineering teams to alert and monitor thousands of services. You'll write Go and Rust to build and evolve our metrics stack on top of open-source technologies, optimize tools like Prometheus, Grafana, Thanos, Loki, Tempo, and OpenTelemetry, and use AI tools and agents daily.

What You'll Be Doing

• Building Our Metrics Stack: Design and build high-performance components of our metrics stack in Go and Rust to handle terabytes of data daily and hundreds of millions of active series.
• AI-Driven Engineering: Use AI tools and agents as part of your daily development and operations work, and help shape how the team works with AI.
• Scaling and Innovating: Enhance and optimize Roku's observability stack, introducing Roku-specific features including AI-assisted capabilities.
• Tooling and Standardization: Work with the Platform team to standardize and improve observability tools.
• Stability and Performance: Ensure the platform remains stable, performant, and reliable.
• Automation and Usability: Build automation around observability tools to simplify onboarding (dashboards, alerts, traces).
• Open Source Contributions: Contribute new tools and improvements back to the CNCF ecosystem.
• Collaboration and Engagement: Partner with engineering teams across Roku to deliver out-of-the-box experiences.
• Technical Leadership: Lead design discussions, mentor engineers and interns, and help shape the platform's technical direction.

We're Excited If You Have

• Significant experience (typically around 8 years) in Infrastructure Engineering and/or DevOps, with a focus on observability platforms.
• Hands-on experience using AI tools and agents, and practical experience building with Large Language Models (tool/function calling, agents, MCP, RAG, evaluating LLM output).
• Solid programming experience (typically around 4 years) in Go and/or Rust, especially with performance-sensitive systems.
• Proven experience designing and operating large-scale observability platforms handling metrics, logs, and traces.
• Expertise in deploying, operating, and extending open-source observability tools (Prometheus, Grafana, Thanos, Loki, Tempo, Cortex, Mimir, or ELK).
• Familiarity with open standards such as OpenTelemetry and OpenMetrics.
• Deep understanding of Kubernetes and Istio architectures.
• Hands-on experience with AWS, GCP, and/or Azure.
• Strong ability to engage with customers and stakeholders to deliver tailored solutions.
• B.S. or M.S. degree in Computer Science, Engineering, or equivalent experience.

Hybrid Work Approach

Roku teams generally work in the office Monday through Thursday, with Fridays flexible for remote work.