Senior Data Engineer

Job description

How you'll make an impact Design and build production pipelines on Databricks using Spark Declarative Pipelines (SDP) and PySpark, from raw ingestion through business-ready data products. Define all pipelines, jobs, and schedules as code in Databricks Asset Bundles, deployed to every environment through automated CI/CD. Build data quality, monitoring, and lineage into pipelines so issues are caught and diagnosed before they reach consumers.

Turn recurring solutions into reusable frameworks, standards, and shared libraries that raise delivery speed across the team. Own your pipelines in production — performance, cost, reliability, and incident response. Partner with Digital Product Managers, architects, and business stakeholders to translate requirements into technical designs, and mentor engineers newer to the platform. What you'll need (Required) Bachelor's degree in computer science, engineering, or a related technical field, plus five or more years of data engineering experience, including hands-on production experience on Databricks.

Demonstrated experience with Spark Declarative Pipelines (SDP / Delta Live Tables) — streaming tables, materialized views, expectations, Auto Loader, and CDC patterns. Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments. Strong Spark and PySpark skills, including performance tuning, and production-quality Python beyond notebook scripting. Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL.

Experience with Git-based CI/CD in a shared repository — code review, automated validation, and promotion across environments. Demonstrated ability to take ambiguous requirements through design to production independently, and to make and defend sound technical decisions. "Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication." "Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads." What else we look for (Preferred) Hands-on experience with Spark Declarative Pipelines (SDP / Delta Live Tables), including streaming tables, materialized views, expectations, Auto Loader, and CDC patterns.

Strong Spark and PySpark development skills, including performance tuning, and production-quality Python beyond notebook scripting. Experience with Git-based CI/CD for data platforms, including code review, automated validation, and promotion across environments. Experience with cloud data platforms on AWS, metadata-driven ingestion frameworks, and infrastructure-as-code. Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication.

Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads. Familiarity with the broader modern data ecosystem, such as Snowflake, dbt, Kafka, and Airflow. Experience integrating enterprise and clinical source systems, such as Epic, SAP, or Salesforce, or migrating workloads from legacy ETL platforms onto a lakehouse. Practical understanding of governance, quality, security, validation, and support expectations in a regulated enterprise environment.

Ability to provide technical guidance, coach team members, and contribute to reusable standards, documentation, and delivery practices.