Data Engineer

Data Engineer

Roles & Responsibilities

Key Responsibilities

• Build Scalable Data Infrastructure: Design, develop, and optimize ETL/ELT pipelines and data warehouses (e.g., ClickHouse) to support analytics, business intelligence, and AI/ML workloads, ensuring scalability, performance, and cost-efficiency.
• Ensure Data Quality and Governance: Implement proactive quality frameworks (e.g., dbt tests, Great Expectations) with automated validation, anomaly detection, and alerts to ensure 99.99% data accuracy, completeness, and timeliness. Enforce data masking and compliance with local regulations (e.g., Kenya’s Data Protection Act).
• Support AI and Self-Service Data Platform: Collaborate with the Senior Data Scientist to build and optimize data pipelines that power the AI-led self-service data platform, ensuring high-quality data for generative AI, LLMs, and predictive models like Credit Scoring.
• Drive AI Innovation: Proactively explore and implement AI-driven use cases in data engineering, such as automated data cleansing or real-time anomaly detection, to enhance operational efficiency and support business processes.
• Data Cataloging and Lineage: Implement data cataloging tools and metadata frameworks to enable efficient data discovery, traceability, and compliance, supporting issue resolution and stakeholder needs.
• Data Access Management: Implement and manage role-based access control (RBAC) for data pipelines and warehouses, ensuring appropriate permissions for stakeholders without over-provisioning. Monitor usage and streamline access to maintain security and compliance.
• Monitoring and Optimization: Build and maintain dashboards (e.g., Grafana) and alerts to monitor pipeline health, uptime, and quality metrics. Optimize query performance and resource utilization for AI and analytics workloads.
• Root Cause Analysis and Ad Hoc Support: Investigate data quality issues raised by Data Business Partners or business users, perform root cause analysis, and implement sustainable fixes. Address ad hoc requests, such as data extracts or one-off dashboards, to support the BI team.
• Continuous Improvement: Driven by curiosity and innovation, identify and implement enhancements to data engineering processes (e.g., optimized pipelines, streamlined data extraction). Take ownership of initiatives, ensuring follow-through to deliver reliable, efficient solutions.

Qualifications

• Bachelor’s degree in Data Science, Computer Science, Statistics, Engineering, or a related field.
• 2–5 years of professional experience in data engineering, with a proven track record in building and optimizing scalable data pipelines.
• Demonstrated expertise in designing ETL/ELT workflows and data warehouses for analytics and AI applications.
• Strong interpersonal and communication skills, with the ability to convey technical concepts to non-technical stakeholders and collaborate across teams.
• Strategic thinker with an innovative mindset, driven by curiosity to uncover new opportunities and a strong sense of ownership to ensure high-quality, reliable data infrastructure.
• Cultural adaptability to align data solutions with local business contexts in Kenya, Uganda, or Côte d'Ivoire, ensuring compliance with regional regulations.
• Creative problem-solver who thrives in dynamic environments, prioritizes data accuracy, and drives process improvements.

Required Skills

• Data Engineering: Proficiency in ETL/ELT tools (e.g., Dagster, Apache Airflow, dbt, Apache Beam, Google Dataflow) for building and optimizing scalable data pipelines.
• Programming: Strong skills in Python, Bash, and JavaScript; experience with Node.js Express or FastAPI for API development.
• Data Quality and Governance: Expertise in data quality tools (e.g., dbt tests, Great Expectations) for validation, anomaly detection, and compliance.
• Cloud Platforms: Experience with cloud data services (e.g., AWS S3, GCP BigQuery, Azure Data Lake) for storage, processing, and analytics.
• Version Control and Deployment: Proficiency in Git/GitHub for collaborative development and Docker for containerizing data applications.
• Monitoring and Visualization: Experience with monitoring tools (e.g., Grafana) and BI tools (e.g., Power BI, Tableau) for pipeline health and analytics support.
• Collaboration: Strong ability to work with Senior Data Scientists, BI Analysts, and Data Business Partners to deliver trusted data solutions.

Preferred Skills

• Experience in credit, lending, or financial services industries, particularly supporting predictive models like Credit Scoring.
• Professional Data Engineer Certification (e.g., Google Cloud, AWS).
• Familiarity with AI-driven data platforms or MLOps.

Success Measures (KPIs)

• Successful delivery of scalable, high-performance data pipelines with 99.99% accuracy, supporting analytics, AI models, and the self-service data platform.
• Reduction in data-related issues through proactive monitoring, anomaly detection, and automated quality checks.
• Positive stakeholder feedback on data pipeline reliability, accessibility, and support for AI and analytics workloads.
• High satisfaction scores from BI Analysts, Senior Data Scientists, and Data Business Partners on data usability and collaboration.
• Proactive identification and implementation of innovative process improvements (e.g., AI-driven pipeline optimization) that enhance efficiency and deliver measurable value.
• Compliance with ethical data standards and local regulations, ensuring trust and security in region-specific data use cases.