Senior Data Engineer (AI-Driven Engineer) (CPT - Hybrid), Cape Town
Senior Data Engineer (AI-Driven Engineer) (CPT - Hybrid), Cape Town
-
Cape Town, South Africa
-
Posted: less than a week ago
-
Save
Description
Environment Our client is transforming the way global shipping contracts are created, executed, and fulfilled. As the leading digital contracting platform for the ocean freight industry, they enable shippers and carriers to improve performance, reduce friction, and increase trust, all powered by data. They are looking for a Senior Data Engineer to help them scale their data platform, support product innovation, and enable advanced AI, analytics, and compliance initiatives. As a Senior Data Engineer, you’ll design and build highly scalable data pipelines, architect foundational data systems, and support machine learning and GenAI capabilities. You’ll also contribute to the backend service layer, working with Java, Python and microservices to ensure seamless data integration between internal systems and their broader platform.
Duties Platform&Infrastructure Engineering
Build and maintain robust
data pipelines
(batch and streaming) using
Airflow, AWS Glue, Step Functions, Lambda , and more
Develop
microservices and data-centric APIs
in
Java , with clean modular architecture and secure data access patterns
Deploy and monitor services in AWS with infrastructure-as-code tools like
Terraform
and
Docker
Data Modelling, Observability&Lineage
Design and implement reliable
data models
to support analytics, data products, and AI workloads
Establish data lineage, quality monitoring, and testing frameworks using tools like
Great Expectations ,
Marquez , or
Monte Carlo
Maintain metadata management and documentation for compliance and discoverability
Data Science&GenAI Enablement
Collaborate with data scientists to provision
training datasets ,
feature stores , and
model pipelines
Build orchestration and evaluation workflows to support
LLM and GenAI development
(e.g., RAG pipelines, embedding search, document intelligence)
Integrate unstructured data (PDFs, documents, messages) into structured datasets for analytics and AI
Security&Compliance
Implement best practices aligned with
SOC 2 ,
GDPR , and internal infosec standards
Ensure secure access controls, audit logging, and encrypted storage for sensitive data
Work with cybersecurity and infrastructure teams to ensure end-to-end data governance
Cross-functional Collaboration
Partner with engineering, product, analytics, and operations teams to support cross-cutting data initiatives
Collaborate closely with
backend and DevOps engineers
to align services, APIs, and deployment patterns.
Requirements
7+ years of experience in data engineering or backend software development
Proficiency in Java and Python, with experience developing microservices and scalable APIs
Strong expertise in SQL, data modelling, and building reliable ETL/ELT pipelines
Deep familiarity with AWS services (Step Functions, Lambda, Glue, S3, Redshift)
Hands-on experience with Airflow, dbt, or similar orchestration and transformation tools
Knowledge of data lineage, quality frameworks, and monitoring systems
Prior experience working alongside data scientists or ML engineers
It’s a plus if you have:
Experience with AIOps or GenAI systems.
Familiarity with real-time streaming (e.g., Kafka, Kinesis) and event-driven architectures.
Exposure to data privacy regulations and SOC 2 compliance.
Background in logistics, supply chain, or a data-rich SaaS environment is a plus.
#J-18808-Ljbffr
Duties Platform&Infrastructure Engineering
Build and maintain robust
data pipelines
(batch and streaming) using
Airflow, AWS Glue, Step Functions, Lambda , and more
Develop
microservices and data-centric APIs
in
Java , with clean modular architecture and secure data access patterns
Deploy and monitor services in AWS with infrastructure-as-code tools like
Terraform
and
Docker
Data Modelling, Observability&Lineage
Design and implement reliable
data models
to support analytics, data products, and AI workloads
Establish data lineage, quality monitoring, and testing frameworks using tools like
Great Expectations ,
Marquez , or
Monte Carlo
Maintain metadata management and documentation for compliance and discoverability
Data Science&GenAI Enablement
Collaborate with data scientists to provision
training datasets ,
feature stores , and
model pipelines
Build orchestration and evaluation workflows to support
LLM and GenAI development
(e.g., RAG pipelines, embedding search, document intelligence)
Integrate unstructured data (PDFs, documents, messages) into structured datasets for analytics and AI
Security&Compliance
Implement best practices aligned with
SOC 2 ,
GDPR , and internal infosec standards
Ensure secure access controls, audit logging, and encrypted storage for sensitive data
Work with cybersecurity and infrastructure teams to ensure end-to-end data governance
Cross-functional Collaboration
Partner with engineering, product, analytics, and operations teams to support cross-cutting data initiatives
Collaborate closely with
backend and DevOps engineers
to align services, APIs, and deployment patterns.
Requirements
7+ years of experience in data engineering or backend software development
Proficiency in Java and Python, with experience developing microservices and scalable APIs
Strong expertise in SQL, data modelling, and building reliable ETL/ELT pipelines
Deep familiarity with AWS services (Step Functions, Lambda, Glue, S3, Redshift)
Hands-on experience with Airflow, dbt, or similar orchestration and transformation tools
Knowledge of data lineage, quality frameworks, and monitoring systems
Prior experience working alongside data scientists or ML engineers
It’s a plus if you have:
Experience with AIOps or GenAI systems.
Familiarity with real-time streaming (e.g., Kafka, Kinesis) and event-driven architectures.
Exposure to data privacy regulations and SOC 2 compliance.
Background in logistics, supply chain, or a data-rich SaaS environment is a plus.
#J-18808-Ljbffr
Highlights
-
Company nameDataFin Recruitment
-
Job positionSenior Data Engineer (AI-Driven Engineer) (CPT - Hybrid)
Safety Tips
Report any suspicious ads or messages.
More info about this ad
Senior Data Engineer (AI-Driven Engineer) (CPT - Hybrid) has been posted in the Cape Town Information Technology category on Locanto.
If you’re looking for something similar, check out Automation Developer X2 - Contract, Cape Town, PHP Developer, Cape Town or New Business Development Consultant in Cape Town, also posted in Information Technology. Currently, there are 345 ads posted in the Information Technology category in Cape Town.
There are more ads within a 15 km radius for this category. If you want to view those ads, click here.