- Spending time learning about the Apache Arrow compute primitives, compute intermediate representation, compute engine, and other foundational components.
- Familiarizing yourself with the different partners for compute kernels and the query execution engine on Apache Arrow.
- Learning and embracing the Apache development process.
Within a month:
- Becoming familiar with our SQL parser and query optimizer.
- Benchmarking queries and exploring the effects of different query optimization techniques using our query optimizer.
- Making changes and improvements to the existing query optimization rules and how it creates physical execution plans for the execution engines under development.
Within 6 months:
- Adding new query optimization rules.
- Making improvements to decision making in the cost based optimization based on metadata availability for the tables being queried.
- Integrating non-SQL operations to the optimization framework.
- Working with client interfacing engineers to understand performance bottlenecks in customer queries.
Within 12 months:
- Proposing and implementing improvements to the query parsing and optimization framework.
- Integrating with a stateful inter-query query engine context to optimize the reusability of compute stages across queries that query the same data in similar ways.
Previous experience that could be helpful:
- Building and/or using open source query optimization frameworks like Apache Calcite, Apache Spark Catalyst, Postgres Query Optimizer, and/or others
- Developing in C++, especially using modern C++
- Utilizing serialization libraries like FlatBuffers, Protobuf, Thrift, MessagePack, and/or others
- Working on non-SQL systems and non-SQL computational abstractions
For NYC-based applicants, the expected salary range is $175k to $210K + equity + benefits.
*Note: Disclosure as required by NYC Pay Transparency Law
Actual starting pay will be based on job-related factors, including exact work location, experience, training, and skill level, so may be higher or lower than what is shown on this posting.