Senior Data Engineer - AWS

Job description

Core Responsibilities

• Architect and implement scalable data lake and lakehouse solutions on AWS for telemetry, analytics, and AI workloads.
• Design and manage secure data ingestion pipelines from edge devices and external systems into AWS S3 and structured storage.
• Develop and optimize ETL/ELT workflows using AWS Glue, Lambda, and Kinesis for structured, semi-structured, and unstructured data.
• Integrate LLM-based smart assistants with vector databases for contextual search and retrieval.
• Implement data partitioning, versioning, and metadata management strategies for high-volume datasets.
• Ensure data governance, audit logging, and compliance using Lake Formation, CloudTrail, and encryption mechanisms.
• Collaborate with platform, operations, and domain teams to curate domain-specific corpora and manage ingestion/index refresh jobs.

AWS Core Services

• S3, Lambda, DynamoDB, RDS (PostgreSQL), API Gateway, CloudFormation, CloudWatch.

Data Engineering Tools

• AWS Glue, Athena, Redshift, Kinesis, EventBridge, SQS, Lake Formation.

Security & Compliance

• SSE-KMS, IAM, VPC endpoints, S3 Object Lock, CloudTrail, encryption policies.

IoT Integration

• AWS IoT Core, Device Defender, fleet provisioning templates, telemetry routing.

Containerization & Orchestration

• Amazon EKS, Kubernetes, Argo CD.

LLM & AI Pipelines

• SageMaker, RAG architecture, prompt/response logging, evaluation metrics.

Data Modeling & Cataloging

• Data lake schema design, metadata tagging, data quality management. ________________________________________ Preferred Experience
• Building data hubs for IoT platforms and predictive maintenance systems.
• Working with equipment metadata and technical documentation for industrial workflows.
• Experience with multi-modal data (structured logs, telemetry, vector embeddings).
• Familiarity with Generative AI integration in engineering or manufacturing contexts.
• Exposure to Infrastructure-as-Code (IaC) tools like Terraform, AIAC CLI for AWS provisioning.