Senior Software Engineer, GPU Infrastructure (HPC)

About the Role

The internal infrastructure team is responsible for building world-class infrastructure and tools used to train, evaluate, and serve Cohere's foundational models. In this role, you will work in close collaboration with AI researchers to support cutting-edge AI workloads, focusing on stability, scalability, and observability while building and operating superclusters across multiple clouds.

*Please Note:* All infrastructure roles require participating in a compensated 24x7 on-call rotation.

Responsibilities

• Build and scale ML-optimized HPC infrastructure: Deploy and manage Kubernetes-based GPU/TPU superclusters across multiple clouds, ensuring high throughput and low-latency performance for AI workloads.
• Optimize for AI/ML training: Collaborate with cloud providers to fine-tune infrastructure for cost efficiency, reliability, and performance using technologies like RDMA, NCCL, and high-speed interconnects.
• Troubleshoot and resolve complex issues: Proactively identify and resolve infrastructure bottlenecks, performance degradation, and system failures to ensure minimal disruption to AI/ML workflows.
• Enable researchers with self-service tools: Design intuitive interfaces and workflows that allow researchers to monitor, debug, and optimize their training jobs independently.
• Drive innovation in ML infrastructure: Work closely with AI researchers to understand emerging needs (e.g., JAX, PyTorch, distributed training) and translate them into robust, scalable infrastructure solutions.
• Champion best practices: Advocate for observability, automation, and infrastructure-as-code (IaC) across the organization, ensuring systems are maintainable and resilient.
• Mentorship and collaboration: Share expertise through code reviews, documentation, and cross-team collaboration, fostering engineering excellence.

Qualifications

• Deep expertise with GPU/TPU clusters, distributed training frameworks (JAX, PyTorch, TensorFlow), and high-performance computing (HPC) environments.
• Proven ability to deploy, manage, and troubleshoot cloud-native Kubernetes clusters at scale for AI workloads.
• Strong proficiency in Python (for ML tooling) and Go (for systems engineering).
• Low-level systems knowledge, including familiarity with Linux internals, RDMA networking, and performance optimization for ML workloads.
• Track record of collaborating closely with AI researchers or ML engineers to solve infrastructure challenges.
• Demonstrated ability to independently identify bottlenecks, propose solutions, and drive impact in a fast-paced environment.