Senior Software Engineer, Observability

About the Team

The Observability team is part of Roku's Cloud Technology Infrastructure organization. We ensure the services and workloads running on our platform have robust observability. Our mission is to advance Roku's observability platform, which ingests terabytes of data daily and manages hundreds of millions of active series.

We build our own metrics stack on top of open-source technologies using Go and Rust, tailored to Roku's scale and needs. We work closely with the Cloud Platform team to support Roku's shift toward a unified, cloud-agnostic infrastructure (Kubernetes, Istio, Envoy, operators, and OSS/CNCF observability projects). We actively integrate AI tools and agents across our engineering lifecycle and into platform capabilities.

About the Role

As a Senior Software Engineer on the Observability team, you'll help design, develop, and scale the platform used across Roku engineering teams to alert and monitor thousands of services. You'll write Go and Rust to build and evolve our metrics stack on top of open-source technologies, working hands-on with tools like Prometheus, Grafana, Thanos, Loki, Tempo, and OpenTelemetry. You will also use AI tools and agents daily to build, test, and operate systems while contributing improvements back to the open-source community.

Responsibilities

• Building Our Metrics Stack: Design and build high-performance components in Go and Rust to handle terabytes of data daily and hundreds of millions of active series.
• AI-Driven Engineering: Use AI tools and agents daily for writing, reviewing, and testing code, automating operational tasks, and accelerating troubleshooting and incident response.
• Scaling and Innovating: Enhance and optimize Roku's observability stack while introducing Roku-specific features and AI-assisted capabilities.
• Tooling and Standardization: Standardize and improve observability tools as part of a cohesive solution in collaboration with the Platform team.
• Stability and Performance: Ensure the observability platform is stable, performant, and reliable for all users.
• Automation and Usability: Build automation around observability tools to simplify onboarding (dashboards, alerts, traces, etc.).
• Open Source Contributions: Contribute new tools and improvements back to the CNCF ecosystem.
• Collaboration and Engagement: Partner with engineering teams to build high-impact solutions.
• Technical Leadership: Lead design discussions, mentor engineers and interns, and shape technical direction.

Qualifications

• Around 8 years of experience in Infrastructure Engineering and/or DevOps, with a focus on observability platforms.
• Hands-on experience using AI tools/agents in daily engineering and practical experience building with LLMs (tool/function calling, agents, MCP, RAG, evaluating LLM output).
• Around 4 years of solid programming experience in Go and/or Rust, especially with performance-sensitive systems.
• Proven experience designing and operating observability platforms processing massive amounts of data (metrics, logs, traces).
• Expertise deploying, operating, and extending open-source observability tools at scale (Prometheus, Grafana, Thanos, Loki, Tempo, Cortex, Mimir, or ELK stack).
• Familiarity with open standards such as OpenTelemetry and OpenMetrics.
• Deep understanding of Kubernetes and Istio architectures.
• Hands-on experience with AWS, GCP, and/or Azure.
• Strong ability to engage with customers and stakeholders.
• B.S. or M.S. degree in Computer Science, Engineering, or equivalent experience.