Data Engineer (m/f/d)

Job description

Design, further development, and practical implementation of a scalable data lakehouse and data platform architecture for very large, heterogeneous data sets Developing a long-term data and storage strategy that takes into account performance, scalability, data lifecycle, costs, availability, and operational requirements Designing suitable storage and access concepts for large volumes of image, video, sensor, geospatial, and other unstructured and semi-structured data, including metadata, partitioning, indexing, compression, and archiving

Building and operating robust batch and streaming pipelines for the ingestion, processing, transformation, and delivery of data from heterogeneous sources Co-design and implementation of a semantic data model or ontology that enables data objects, entities, relationships, and context to be described and linked across systems Creating a resilient data foundation for machine learning, AI-based applications, and autonomous systems, including traceable data provenance and reproducible training and validation datasets

Ensuring data lineage, data versioning, data quality, schema management, and appropriate governance and access concepts throughout the entire data lifecycle Development and operation of the data platform in large, distributed Kubernetes-based infrastructures, as well as integration into Infrastructure-as-Code and CI/CD/DevSecOps processes Analyzing and optimizing data layouts, data locality, compute proximity, and data movements for data-intensive applications

Evaluation of suitable open-source and commercial technologies, as well as implementation ranging from proofs of concept to production-ready, scalable solutions Close collaboration with AI/ML, software, platform, and cybersecurity teams on architectural decisions, interfaces, data products, and operational implementation