Senior Lead Data Engineer - Nairobi County
Only for registered members Nairobi County, Kenya
3 weeks ago
* This salary range is an estimation made by beBee
Job summary
Design and develop pipelines using Python, PySpark, and SQL. Utilize S3 buckets for storing large volumes of raw and processed data. Implement and manage complex data workflows using Apache Airflow (MWAA) to orchestrate tasks.
Responsibilities
- Design and develop pipelines using Python, PySpark, and SQL
- Use GitLab as the versioning control system
- Utilize S3 buckets for storing large volumes of raw and processed data
- Implement and manage complex data workflows using Apache Airflow (MWAA) to orchestrate tasks
- Utilize Apache Iceberg (or similar) for managing and organizing data in the data lake
- Create and maintain data catalogs using AWS Glue Catalog to organize metadata
- AWS Athena for interactive querying
- Familiarize with data modeling techniques to support analytics and reporting requirements, as well as knowledge of the data journey stages within a data lake (Medallion Architecture)
Qualifications
- Ideally, a degree in Information Technology, Computer Science, or a related field
- Ideally, +5 years of experience within the Data Engineering landscape
- Strong expertise in Python, PySpark, SQL, and the overall AWS data ecosystem
- Strong problem-solving and analytical skills
- Ability to explain technical concepts to non-technical users
- Proficiency to work with GitHub
- Terraform and CICD pipelines are a great 'nice-to-have'
Job description
Lorem ipsum dolor sit amet
, consectetur adipiscing elit. Nullam tempor vestibulum ex, eget consequat quam pellentesque vel. Etiam congue sed elit nec elementum. Morbi diam metus, rutrum id eleifend ac, porta in lectus. Sed scelerisque a augue et ornare.
Donec lacinia nisi nec odio ultricies imperdiet.
Morbi a dolor dignissim, tristique enim et, semper lacus. Morbi laoreet sollicitudin justo eget eleifend. Donec felis augue, accumsan in dapibus a, mattis sed ligula.
Vestibulum at aliquet erat. Curabitur rhoncus urna vitae quam suscipit
, at pulvinar turpis lacinia. Mauris magna sem, dignissim finibus fermentum ac, placerat at ex. Pellentesque aliquet, lorem pulvinar mollis ornare, orci turpis fermentum urna, non ullamcorper ligula enim a ante. Duis dolor est, consectetur ut sapien lacinia, tempor condimentum purus.
, consectetur adipiscing elit. Nullam tempor vestibulum ex, eget consequat quam pellentesque vel. Etiam congue sed elit nec elementum. Morbi diam metus, rutrum id eleifend ac, porta in lectus. Sed scelerisque a augue et ornare.
Donec lacinia nisi nec odio ultricies imperdiet.
Morbi a dolor dignissim, tristique enim et, semper lacus. Morbi laoreet sollicitudin justo eget eleifend. Donec felis augue, accumsan in dapibus a, mattis sed ligula.
Vestibulum at aliquet erat. Curabitur rhoncus urna vitae quam suscipit
, at pulvinar turpis lacinia. Mauris magna sem, dignissim finibus fermentum ac, placerat at ex. Pellentesque aliquet, lorem pulvinar mollis ornare, orci turpis fermentum urna, non ullamcorper ligula enim a ante. Duis dolor est, consectetur ut sapien lacinia, tempor condimentum purus.
Get full access
Access all high-level positions and get the job of your dreams.