Employer listing
Principal Software Engineer
Crusoe
San Francisco, Ca · on site · full time · Principal
$285,000 - $335,000
About Crusoe
Crusoe is on a mission to accelerate the abundance of energy and intelligence. As the only vertically integrated AI infrastructure company built from the ground up, we own and operate each layer of the stack — from electrons to tokens — to power the world's most ambitious AI workloads.
About This Role
As a Principal Software Engineer for Infrastructure, you will partner with teams across the company to help shape the technical direction of the compute, storage, and networking platforms that power Crusoe’s AI-first cloud. This is a high-leverage leadership role spanning the full infrastructure stack: you will work alongside engineers and stakeholders on the systems that help massive GPU clusters run AI and HPC workloads at extreme throughput, low latency, and high reliability. Your work spans the Linux kernel, hypervisors and virtualization, distributed storage, network fabrics, and the control planes that tie them together.
What You’ll Be Working On
- Infrastructure Strategy: Partner with engineering, product, and leadership to help shape the long-term direction for compute, storage, and networking, turning direction into practical plans and milestones.
- Architecture & Design: Collaborate with teams on architecture reviews, design documents, and code reviews for critical infrastructure components and services.
- Compute & Virtualization: Work with compute teams on virtualization and orchestration layers delivering bare-metal performance to GPU workloads (hypervisor tuning, SR-IOV, vDPA, virtio, device passthrough, NUMA-aware scheduling).
- Storage Systems: Work on high-throughput, low-latency storage for training and inference pipelines (distributed block/file/object storage, NVMe/NVMe-oF, RDMA data paths, checkpointing).
- Networking & Fabric: Partner on architectures for GPU-to-GPU fabrics, high-speed Ethernet, and specialized AI clusters (XDP, eBPF, DPDK, OVS/OVN, BGP, EVPN, VXLAN).
- Kernel & Driver Work: Contribute to Linux kernel modules and high-performance drivers across the network, storage, and device stack.
- Hardware Integration: Work on accelerators and offload devices (GPUs, SmartNICs, DPUs, IPUs, NVMe) from qualification through bring-up.
- Cross-Stack Performance & Reliability: Troubleshoot complex production issues across compute, storage, and network boundaries; improve performance, observability, capacity planning, and failure-domain design.
- Technology Evaluation: Explore emerging infrastructure technologies, contribute to open-source projects, and share learnings.
- Mentorship & Collaboration: Mentor and support engineers across teams to foster engineering excellence.
What You’ll Bring to the Team
- Large-Scale Infrastructure Mastery: 10+ years of experience building and operating large-scale infrastructure systems in production, ideally in cloud or hyperscale environments.
- Cross-Domain Breadth: Hands-on experience across compute, storage, and networking, with a track record of solving problems crossing domain boundaries.
- Systems Programming Excellence: Strong systems programming skills in C, C++, Rust, or Go, working on performance-critical services and control planes.
- Linux Internals Depth: Extensive knowledge of Linux kernel architecture, memory management, scheduler, I/O and network stacks, and device driver development.
- Virtualization & Compute Literacy: Experience with KVM/QEMU or similar hypervisors, container runtimes, and orchestration platforms such as Kubernetes; understanding of GPU, CPU, and NUMA topology.
- Storage Literacy: Familiarity with distributed storage architectures (block, file, or object), NVMe/NVMe-oF, RDMA, and performance tradeoffs.
- Network & Fabric Literacy: Familiarity with data center fabrics (Spine-Leaf, BGP, EVPN, VXLAN), network virtualization (OVS, OVN), and high-speed packet processing (XDP, eBPF, DPDK).
- Accelerator Knowledge: Familiarity with SmartNICs, DPUs, and accelerators (e.g., NVIDIA ConnectX-6/7, BlueField-3, Intel IPUs).
- Production Debugging: Experience diagnosing complex issues across hardware, firmware, kernel, and control-plane software.
- Strategic Influence: Track record of leading through influence, strong written communication skills, and a mentoring mindset.
- Education: Bachelor’s degree in Computer Science, Computer Engineering, or related analytical field (or equivalent professional experience).
Bonus Points
- Active contributions to open-source projects such as Linux kernel, KVM/QEMU, Kubernetes, Ceph, OVS, OVN, Cilium, or Multus.
- Experience with infrastructure automation, fleet management, or complex orchestration tooling.
- Prior experience building compute, storage, or networking stacks for multi-node GPU training at scale.
- Experience working with hardware teams on platform design, qualification, and firmware-to-software integration.
Benefits
Comprehensive health, dental, and vision insurance; HSA employer contributions; 401(k) with company match up to 4%; equity/RSUs; paid time off, holidays, and parental leave; commuter benefits; cell phone stipend; daily meals allowance; professional development and tuition reimbursement.