Senior Backend Software Engineer, Data Pipelines (REMOTE - Palo Alto, CA)

About Skyflow:

Skyflow is a data privacy vault company built to radically simplify how companies isolate, protect, and govern their customers’ most sensitive data. With its global network of data privacy vaults, Skyflow is also a comprehensive solution for companies around the world looking to meet complex data localization requirements. Skyflow currently supports a diverse customer base that spans verticals like fintech, retail, travel, and healthcare.

Skyflow is headquartered in Palo Alto, California and was founded in 2019. For more information, visit www.skyflow.com or follow on X (formerly Twitter) and LinkedIn.

About the role:

As a Senior Backend Software Engineer, Data Pipelines, you will be responsible for the design and development of complex data processing workflows and for performing batch processing of large amounts of structured and semi-structured data in cloud-native environments. You will do this by leveraging Kafka for asynchronous queues, Docker for containerization, and Kubernetes for orchestration, to achieve high levels of efficiency, scalability, and reliability.

Desired Qualifications:

Solid experience with Golang (Must-Have) developing highly-available and scalable production-level code to build microservices, batch architectures, and lambda expressions
Excellent understanding of data structures; You’re not challenged by keeping a flow of data structures in-memory
Experience working with multiple file formats (CSV, JSON, Parquet, Avro, Delta Lake et al.)
Knowledge of data warehouse technical architectures, Docker and Kubernetes infrastructure components, and how to develop secure ETL pipelines
Experience in pub/sub modes like Kafka
Experience working in a Big Data environment (Hadoop, Spark, Hive, Redshift et al.)
Experience with relational and non-relational databases
Experience building real-time streaming data pipelines is a plus

Responsibilities:

Containerize each component of the data pipeline (like ETL processes, databases, data processing applications) using Docker to create Dockerfiles, and Docker Images
Set up Kubernetes clusters to manage and orchestrate Docker Containers and deploy Pods, as well as create services and load balancing policies
Use Kubernetes Volumes for managing data and stateful applications, ensuring that data persists beyond the lifespan of individual Pods
Configure Kafka for scalability, ensuring it can handle high volumes of data streams efficiently.
Configure Kafka brokers, topics, producers, and consumers, as well as use Kafka Connect to integrate with external databases, systems, or other data sources/sinks
Implement logging and monitoring solutions to keep track of the health and performance of your data pipelines
Troubleshoot connectivity issues to common datastores such as Amazon S3 and Azure Data Lake
Implement network policies in Kubernetes for secure communication between different services
Follow best practices for security, such as securing Kafka clusters and implementing proper access controls

Benefits:

Work from home expense (U.S., Canada, and Australia)
Excellent Health, Dental, and Vision Insurance Options (Varies by Country)
Vanguard 401k
Very generous PTO
Flexible Hours
Generous Equity

Pay:

A base salary range of $150,000 - $200,000 can be expected for this role in the San Francisco/Bay Area. You could also be entitled to receive an additional incentive bonus or variable pay, equity, and benefits.

Skyflow operates from a place of high trust and transparency; we are happy to disclose the pay range for our open roles that best align with your needs. Exact compensation may vary based on skills, experience, education, and location.

At Skyflow, we believe that diverse teams are the strongest teams. We invite applicants of all genders, races, ethnicities, nationalities, ages, religions, sexual orientations, disability statuses, educational experiences, family situations, and socio-economic backgrounds.

Apply for this Job

* Required

First Name *

Last Name *

Email *

Phone *

Resume/CV *

(File types: pdf, doc, docx, txt, rtf)

Cover Letter

(File types: pdf, doc, docx, txt, rtf)

When autocomplete results are available use up and down arrows to review

+ Add another education

LinkedIn Profile

1. How many years of experience do you have writing production-level code using Golang? Can you please describe this experience in detail? (as we know that the number of years of experience does not necessarily correlate with level of experience or aptitude) *

2. When developing ETL pipelines using Golang, can you please provide some details on the stack used (i.e. Overall DW technical architectures; Docker (Dockerfiles, Docker Image, Docker registry such as Docker Hub or a Private Registry); Kubernetes (Setting up clusters, Deployments and Pods, Service & Load Balancing, Volume Mgmt. for data storage, Automation & Scaling, Monitoring & Logging); Configuration Mgmt. and Implementing CI/CD pipelines; Implementing Network Policies and Security Measures, and adhering to any Compliance standards)?

This question is optional, however we feel that those candidates who have a depth and breadth of experience in creating efficient, scalable, reliable, and secure data pipelines may want to more fully elaborate on their experience, or call out something which they are particularly proud of with regard to this topic (if it is not already detailed on their resume)

Voluntary Self-Identification

For government reporting purposes, we ask candidates to respond to the below self-identification survey. Completion of the form is entirely voluntary. Whatever your decision, it will not be considered in the hiring process or thereafter. Any information that you do provide will be recorded and maintained in a confidential file.

As set forth in Skyflow’s Equal Employment Opportunity policy, we do not discriminate on the basis of any protected group status under any applicable law.

Gender

Are you Hispanic/Latino?

Please identify your race

Race & Ethnicity Definitions

If you believe you belong to any of the categories of protected veterans listed below, please indicate by making the appropriate selection. As a government contractor subject to the Vietnam Era Veterans Readjustment Assistance Act (VEVRAA), we request this information in order to measure the effectiveness of the outreach and positive recruitment efforts we undertake pursuant to VEVRAA. Classification of protected categories is as follows:

A "disabled veteran" is one of the following: a veteran of the U.S. military, ground, naval or air service who is entitled to compensation (or who but for the receipt of military retired pay would be entitled to compensation) under laws administered by the Secretary of Veterans Affairs; or a person who was discharged or released from active duty because of a service-connected disability.

A "recently separated veteran" means any veteran during the three-year period beginning on the date of such veteran's discharge or release from active duty in the U.S. military, ground, naval, or air service.

An "active duty wartime or campaign badge veteran" means a veteran who served on active duty in the U.S. military, ground, naval or air service during a war, or in a campaign or expedition for which a campaign badge has been authorized under the laws administered by the Department of Defense.

An "Armed forces service medal veteran" means a veteran who, while serving on active duty in the U.S. military, ground, naval or air service, participated in a United States military operation for which an Armed Forces service medal was awarded pursuant to Executive Order 12985.

Veteran Status

Voluntary Self-Identification of Disability

Form CC-305

Page 1 of 1

OMB Control Number 1250-0005

Expires 04/30/2026

Why are you being asked to complete this form?

We are a federal contractor or subcontractor. The law requires us to provide equal employment opportunity to qualified people with disabilities. We have a goal of having at least 7% of our workers as people with disabilities. The law says we must measure our progress towards this goal. To do this, we must ask applicants and employees if they have a disability or have ever had one. People can become disabled, so we need to ask this question at least every five years.

Completing this form is voluntary, and we hope that you will choose to do so. Your answer is confidential. No one who makes hiring decisions will see it. Your decision to complete the form and your answer will not harm you in any way. If you want to learn more about the law or this form, visit the U.S. Department of Labor’s Office of Federal Contract Compliance Programs (OFCCP) website at www.dol.gov/ofccp.

How do you know if you have a disability?

A disability is a condition that substantially limits one or more of your “major life activities.” If you have or have ever had such a condition, you are a person with a disability. Disabilities include, but are not limited to:

Alcohol or other substance use disorder (not currently using drugs illegally)
Autoimmune disorder, for example, lupus, fibromyalgia, rheumatoid arthritis, HIV/AIDS
Blind or low vision
Cancer (past or present)
Cardiovascular or heart disease
Celiac disease
Cerebral palsy
Deaf or serious difficulty hearing
Diabetes
Disfigurement, for example, disfigurement caused by burns, wounds, accidents, or congenital disorders
Epilepsy or other seizure disorder
Gastrointestinal disorders, for example, Crohn's Disease, irritable bowel syndrome
Intellectual or developmental disability
Mental health conditions, for example, depression, bipolar disorder, anxiety disorder, schizophrenia, PTSD
Missing limbs or partially missing limbs
Mobility impairment, benefiting from the use of a wheelchair, scooter, walker, leg brace(s) and/or other supports
Nervous system condition, for example, migraine headaches, Parkinson’s disease, multiple sclerosis (MS)
Neurodivergence, for example, attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorder, dyslexia, dyspraxia, other learning disabilities
Partial or complete paralysis (any cause)
Pulmonary or respiratory conditions, for example, tuberculosis, asthma, emphysema
Short stature (dwarfism)
Traumatic brain injury

Disability Status

PUBLIC BURDEN STATEMENT: According to the Paperwork Reduction Act of 1995 no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. This survey should take about 5 minutes to complete.

Our system has flagged this application as potentially being associated with bot traffic. Please turn off any VPNs, clear your browser cache and cookies, or try submitting your application in a different browser. If this issue persists, please reach out to our support team via our help center.

Please complete the reCAPTCHA above.