About the Role
If you’re passionate about building large scale data processing systems, and you are motivated to make an impact in creating a robust and scalable data platform used by every team, come join us. You will jump into an early stage team that builds and manage a workflow orchestration service. You will help shape the vision and architecture of WeWork's next generation data infrastructure, making it easy for developers to build data-driven products and features. You are responsible for developing a reliable infrastructure that scales with the company’s incredible growth. Your efforts will allow accessibility to business and user behavior insights, using huge amounts of WeWork data to fuel several teams such as Analytics, Data Science, Sales, Revenue, Product, Growth and many others as well as empowering them to depend on each other reliably. You will be a part of an experienced engineering team and work with passionate leaders on challenging distributed systems problems.
About the Mission
Data is at the core of our business, providing insights into the effectiveness of our products and enabling the technology that powers them. The mission of Developer Platform is to empower WeWork to build better products faster. We’re building an ecosystem where consumers and producers of data can depend on each other safely. We thrive to build high-quality systems we can be proud to open source and an amazing experience for our users and ourselves. We regard culture and trust highly and are looking forward to welcoming your contribution to the team.
- Build and maintain high-performance, fault-tolerant, secure, and scalable fully managed workflow orchestration service built on top of Apache Airflow that empowers data platform across the entire company.
- Lead development of high leverage projects and capabilities of the platform
- Partner with engineers and product leaders to design and build robust features on Airflow to support all data engineering teams across WeWork.
- Thinking through long-term impacts of key design decisions and handling failure scenarios
- Form a holistic understanding of tools, key business concepts (data tables), and the data dependencies and team dependencies
- Building self-service platforms to power WeWork’s Technology
- 7-9+ years of experience
- When thinking about Big Data workflow engines you are not just experienced with Apache Airflow. You know under the hood implementations on this platform.
- Strong Python and Unix development experience
- Solid experience with Docker, Kubernetes or other container technologies
- Write server-side code to power the Airflow platform with Python
- Build, configure and implement features against other open-source infrastructure components that empower Airflow platform, such observability(Prometheus and Grafana) and Kubernetes
- Experience shipping several high quality of complex software's release.
- Strong troubleshooting and performance tuning skills.
- Experience in development for cloud-based systems, ideally AWS
- Strong foundation in algorithms and data structures and their real-world use cases.
- Excellent communication (verbal and written) and interpersonal skills and an ability to effectively communicate with both business and technical teams
- Strong understanding of distributed systems concepts and principles (consistency and availability, liveness and safety, durability, reliability, fault-tolerance, consensus algorithms)
- Strong Experience with one or more of the following technologies:
- Workflow management - Luigi, Apache Oozie or Azkaban
- Distributed logging systems (Kafka, Pulsar, Kinesis, etc)
- MPP databases (Redshift, Vertica, …)
- Cloud storage: S3, GCS
Nice to Have
- 3+ years experience in leadership roles or engineering management.
- Experience in open source development
- Data warehouse tooling
WeWork Technology is bridging the gap between physical and digital platforms, providing a delightful, flawless & powerful experience for members and employees. We build software and hardware that enables our members to connect with each other and the space around them like never before.
We augment our community and culture teams through the tools we build. We believe there’s a macro shift toward a new way of working—one focused on a movement towards meaning and purpose.
WeWork Technology is proud to be shaping this movement. We are a team of passionate, fearless and collaborative problem-solvers distributed globally with one goal in mind - to humanize technology across the world.
We are an equal opportunity employer and value diversity in our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.