As a Data Engineer, under the leadership of a team manager, you’ll be responsible for gathering and collecting data, storing it, doing batch processing or real-time processing on it, and serving it to a data scientist and data analyst who can easily query it
We are looking for a savvy data engineer who eagerness to continually improve skills and learn new technologies, willingness to take initiative to contribute beyond basic responsibilities
- Understanding data source (10%)
- Develop data processes for construction, mining, and modeling that are delivered to the data analyst/science team (30%)
- Create large data warehouses by running some ETL (Extract, Transform and Load) that is used for analysis by the analyst/scientists (30%)
- Install continuous pipelines of huge pools of filtered information so that data analyst/scientists can pull relevant data sets for their analyses (20%)
- Develop, construct, test and maintain architectures such as databases and large-scale data processing systems (10%)
- BS in Computer Science or equivalent experience
- 1+ years of day-to-day working experience as data engineer
- Experience with object-oriented and functional programming languages: Python, Scala etc.
- Experience with RDBMS (MS-SQL, MYSQL etc.) and NoSQL (MongoDB, DynamoDB etc.)
- Experience with ETL and data integration tool : Fluentd, Spark, Sqoop, Airflow, Informatica
- Experience with AWS cloud services: EC2, S3, EMR, Glue, RedShift, Athena etc.
- Experience with Big Data querying tools: Hue, Hive, HiveQL, Pig and Impala etc.
- Proficiency in general database administration concepts and efficient query writing
- Excellent verbal and written communication skills
- Experience with data visualization software like Tableau