Intermediate Data Engineer, Cape Town

We are seeking an exceptionally talented hands-on Intermediate Data Engineer who combines deep technical expertise in data pipelines, data modelling, and integration.

This role is accountable for designing and delivering scalable, reliable, and high-performance data solutions that support both analytics and application use cases. The successful candidate will play a key role in shaping the organization's data architecture, ensuring that data is structured, governed, and accessible in a way that supports business decision-making and product capabilities.

The ideal candidate is a self-starter who values clean, maintainable data solutions, understands trade-offs between normalized and analytical models, and continuously improves the quality and performance of the data platform.

What you'll do:

• Data Pipeline Development & Integration
  • Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF), including pipelines, data flows, triggers, and parameterization.
  • Integrate data from APIs, flat files, databases, and cloud/on-prem systems.
  • Implement robust ingestion patterns for structured and semi-structured data (JSON, XML, CSV).
  • Ensure reliable, efficient, and secure movement of data across systems.
• Data Modelling & Transformation
  • Design and maintain both normalized (OLTP-aligned) and denormalized (analytical / reporting) data models.
  • Apply best practices in dimensional modelling (fact/dimension tables) as well as normalized relational design.
  • Implement transformations using SQL (T-SQL), stored procedures, and data flows to prepare analytics-ready datasets.
  • Ensure data models are scalable, reusable, and aligned with business requirements.
  • Manage historical data tracking, including slowly changing dimensions and auditability.
• Performance, Reliability & Scalability
  • Optimize SQL queries, ETL pipelines, and data storage for large datasets (millions+ rows).
  • Implement indexing strategies, partitioning, and efficient data access patterns.
  • Ensure pipelines are resilient with proper error handling, retry logic, and monitoring.
  • Design solutions that minimize impact on transactional systems (clear separation of OLTP and reporting workloads).
  • Proactively identify and resolve performance bottlenecks.
• Application & API Integration
  • Collaborate closely with backend (.NET) teams to support data access patterns and integration with application services.
  • Design and deliver aggregated datasets and data structures optimized for API consumption.
  • Support frontend (e.g., Vue.js) data requirements by enabling efficient querying, filtering, and pagination.
  • Contribute to embedded analytics and application-driven reporting use cases.
• Collaboration & Continuous Improvement
  • Work closely with BI developers, analysts, and stakeholders to translate data requirements into scalable solutions.
  • Continuously improve data platform architecture, tooling, and processes.
  • Support CI/CD practices for data pipelines and deployments.
  • Stay current with evolving data engineering tools, patterns, and best practices.

Your Expertise:

• 5+ years in data engineering, ETL development, or related roles.
• Azure Data Factory (ADF): Strong hands-on experience with pipeline orchestration, data flows, triggers, parameterization, and monitoring.
• SQL / T-SQL: Advanced querying, performance tuning, indexing strategies, and stored procedure development.
• Data Modelling:
  • Strong experience with both normalized (3NF) and denormalized (star/snowflake) data models
  • Understanding of when to apply each approach
  • Experience designing scalable and maintainable data schemas
• Data Platforms: Experience with Azure SQL, Synapse Analytics, or Data Lake architectures.
• ETL / ELT:
  • Strong understanding of data pipeline design, incremental loading, and transformation strategies.
  • Exposure to SSIS, Informatica, Talend, dbt, or similar tools.
• Data Warehousing: Solid knowledge of dimensional modelling (star/snowflake schemas) and data lifecycle management.
• Performance & Scalability:
  • Experience working with large-scale datasets and high-volume data pipelines.
  • Strong understanding of indexing, partitioning, and query optimization techniques.
  • Experience designing solutions that separate transactional and analytical workloads.
• Data Governance & Quality:
  • Experience implementing data validation, reconciliation, and quality controls.
  • Strong understanding of data lineage, auditability, and consistency.
• Integration: Experience working with APIs and handling JSON/XML data formats.
• Familiarity with Power BI, Tableau, or similar platforms.
• Experience with Azure data services (e.g., Synapse, Fabric) is advantageous.
• Azure DevOps, Git, and CI/CD experience is a plus.
• Demonstrated experience delivering end-to-end data engineering solutions in production environments.
• Proven experience contributing to code reviews, enforcing standards, and improving engineering practices.

Education & Qualifications:

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience.