Data Engineer II

Primary Responsibilities & Expectations:

• Designing and Developing Data Pipelines and ETLs: You will play a crucial role in designing, building, and maintaining data pipelines and ETLs/ELTs (using tools like AWS DMS, Airflow and other AWS services). 
• Manage ingestion of raw transactional and non-transactional data into an S3 data lake. 
• Ensure efficient data flow across various systems within AstroPay, enabling data-driven decision-making.
• Data Lake and Warehouse integration: As a Lead / Senior Data Engineer, you will be responsible for managing and enhancing our data architecture combining and integrating a data lake and a relational data warehouse. 
• Ensure seamless integration of raw data into the S3 data lake before applying necessary transformations and delivering clean, structured data to downstream systems. We need to handle mostly batch processing but also there are some requirements for streaming.
• Data Modeling: Apply transformation, enrichment, and data processing using the most appropriate tools for the workload. We follow medallion architecture principles. 
• Apply best practices in data engineering, including version control, testing, and documentation.
• Collaboration with Cross-Functional Teams: Collaborate closely with engineers, data analysts, infrastructure and other stakeholders to understand their data needs and deliver solutions that enable data-driven decision-making across the organization. 
• Data Governance and Security:  Ensure and implement data governance best practices. Establish and maintain a secure data environment to comply with regulations in different countries.
• Create Reusable Data Assets: Develop and implement strategies for creating reusable data assets, promoting efficiency, and reducing redundancy in data processing.
• Build Trust in Our Data: Implement measures to enhance data quality and build trust in the accuracy and reliability of our data. This includes data validation, quality checks, and documentation.
  
  

Core Competencies and Skills:

Must-Have:

• Expertise in Data Pipelines: Proven experience designing, developing, and maintaining data pipelines for diverse workloads using AWS services like DMS, Airflow, Lambda, and Glue.
• Data  Lakes and Warehouse Architecture: Proven experience managing data lakes and relational data warehouses. Familiarity with Apache Iceberg, S3-based data lakes, and Redshift.
• AWS Cloud Knowledge: Extensive experience with AWS services, including Redshift, Lambda, Airflow, DMS, and S3. You should be comfortable working with these services to build and maintain data solutions.
• Collaborative Problem Solver: Strong communication skills to collaborate effectively with cross-functional teams and proactively own data challenges.
• Software engineering background: Solid foundation in software engineering principles, to apply software engineering best practices to our data architecture and processes. Some Python experience is also required.
• English: to interact with the rest of the company.

Nice to Have:

• Familiarity with Database Administration to fine-tune database performance and ensure data integrity.
• Familiarity with Infrastructure Management and other AWS services like CodeCommit, SageMaker, CloudWatch, StepFunctions, Athena, and Glue, can be a valuable addition to your skill set.
• Familiarity with BI Tools:  Looker or other similar BI tools.
• Build data models: Experience building data models, understanding of data catalogs, data governance, and data lineage.
• Fintech: Prior experience working in the fintech ecosystem will be considered an advantage.