Senior Data Engineer – Databricks & Data Modeling/Architecture

Job description

Description and Requirements

Key Responsibilities

• Design, develop, test, and deploy scalable pipelines using Databricks, PySpark, Spark SQL, and Delta Lake, spanning Bronze/Silver/Gold (medallion) layers for HR data integration and analytics.
• Build data models – logical, physical, dimensional, and integration – and maintain source-to-target mappings for workforce, organization, employee, position, job, and related HR domains.
• Migrate legacy GDC integrations by re-engineering Informatica, SQL, and stored-procedure logic into modern Databricks patterns, including inbound Oracle HCM feeds and outbound HR data files.
• Validate output parity through reconciliation, parallel runs, and source-to-target checks, and build automated data quality, exception handling, and audit controls into pipelines.
• Apply enterprise governance, security, privacy, and role-based access expectations to sensitive HR data, capturing lineage and metadata where applicable.
• Produce production-ready documentation – technical designs, runbooks, data dictionaries, and support handoffs – for maintainable, observable, cost-aware solutions.
• Collaborate within a distributed delivery team, participate in agile ceremonies, mentor junior engineers, and communicate progress, blockers, and risks clearly.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field, with 7+ years of data engineering, ETL, or data platform experience.

Azure Databricks & Apache Spark

• 3+ years hands-on building Delta Lake pipelines with PySpark, Spark SQL, and SQL across a medallion architecture.

Data modeling & integration

• Strong logical/physical/dimensional modeling plus source-to-target mapping, reconciliation, and production cutover support.

AI-assisted engineering

• Proficiency using AI/Copilot tooling to accelerate development, code conversion, and documentation.
• Practical, delivery-focused work style, comfortable turning ambiguous legacy logic into clean target-state designs within a highly governed environment.
• Strong written and verbal communication, with the ability to overlap U.S. Eastern Time for key meetings, design reviews, and issue resolution.

Preferred Qualifications

• Suggested 3+ years modernizing legacy ETL (Informatica-to-Databricks) or building metadata-driven, configuration-driven data frameworks.

Azure ecosystem

• Data Lake Storage, Data Factory, DevOps, Azure SQL/Synapse, Unity Catalog, and Git-based CI/CD.

HR domain data

• Oracle HCM Cloud extracts/PVOs/HDL, workforce analytics, and consumption via Power BI, Crunchr, or similar.
• Experience supporting regulated, sensitive employee data in insurance, financial services, or HR technology.
• Detail-oriented, collaborative, and comfortable working independently against offshore delivery expectations while staying aligned to MetLife leads.