Software Engineer – Query Engines

Overview

Palantir builds software for data-driven decisions and operations. As part of the query engines software engineering team, you will apply open-source data technologies to data processing infrastructure underpinning Palantir Foundry.

You will work on how computations are represented, how query plans are optimized, how operators execute, and how data is read and written efficiently. You will build on and extend Apache Spark, Apache DataFusion, and Apache Iceberg, owning improvements from design through production.

Core Responsibilities

• Designing and implementing query planning and optimization capabilities that turn complex computations into efficient execution plans.
• Extending execution engines with new capabilities and improving query operators, parallelism, memory management, and data movement.
• Developing Foundry’s Iceberg catalog and engine integrations, including table metadata, transactions, and efficient reads and writes.
• Building shared transformation semantics and execution interfaces for workloads across Foundry’s products and platform services.
• Improving incremental processing so pipelines can reuse previous results and process new data efficiently while preserving correctness.
• Evaluating and integrating advances in open-source data systems, validating behavior and performance against real-world workloads.
• Investigating correctness and performance issues across planning, execution, and storage; building tests and benchmarks that prevent regressions.
• Collaborating with product teams and customers to translate operational needs into engine capabilities that integrate with Foundry’s security, data management, and build infrastructure.

Qualifications

• Experience: 4+ years of professional software engineering experience developing database engines, distributed data processing systems, storage systems, or comparable infrastructure, with depth in query planning, execution, or performance optimization. (Operating on top of these technologies alone does not qualify.)
• Experience diagnosing correctness and performance problems in complex systems.
• Strong written and verbal communication skills and ability to work effectively across teams.

Technologies Used

• Java, Scala, Rust, Python
• Apache Spark, Apache DataFusion, Apache Comet, Velox
• Apache Iceberg
• Apache Arrow, Apache Parquet
• Gradle, Cargo, GitHub