Senior Data Platform / Data Product Engineering Lead

Job description

Senior Data Platform / Data Product Engineering Lead Must Have Technical/Functional Skills Job Description: Senior Data Platform / Data Product Engineering Lead

Role Overview

We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development. The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams. ________________________________________ Required

Skills & Experience

• 10+ years of experience in Data Engineering / Data Platform roles
• Strong hands-on expertise in:
• Databricks (Delta Lake, workflows, DAG)
• Apache Airflow / Astronomer
• Python, SQL, DBT ,AWS
• Proven experience implementing CI/CD frameworks (Harness, GitHub Actions, Azure DevOps)
• Deep understanding of:
• Data governance (catalogs, lineage, contracts, metadata)
• Data quality and masking techniques
• Enterprise data platforms and marketplace ecosystems
• Experience with API-based integrations (e.g., entitlement systems like AccessCentral)
• Monitoring/observability tools (e.g., Datadog)

Job Description

Senior Data Platform / Data Product Engineering Lead

Role Overview

The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams. ________________________________________ Core Responsibilities

• Platform Strategy & Self-Service Enablement
• Define and implement a self-service data platform strategy to reduce onboarding friction.
• Lead automated provisioning of:
• Databricks workspaces (via DevHub)
• Airflow/Astronomer environments
• Access and entitlements (AccessCentral APIs)
• Establish isolated, stable development environments for federated teams.
• Drive platform observability by integrating metrics into tools lik e Datadog. ________________________________________
• Data Discovery, Access & Governance
• Architect and implement enterprise-wide data discovery and marketplace enablement.
• Drive adoption of:
• Data contracts
• Metadata standards
• Domain-aligned catalogs (Unity Catalog)
• Enable secure access to curated, masked datasets in dev and production environments.
• Implement tagging, access patterns, and entitlement automation.
• Partner with risk/compliance teams to enforce regulatory and governance controls (BFSI-aligned). ________________________________________
• Data Engineering, Curation & Orchestration
• Lead design of scalable data ingestion, curation, and transformation frameworks.

Build and standardize modular, reusable frameworks

• LaunchLake templates
• Airflow DAG libraries o DBT-based transformation models

Ensure

• Data quality and consistency
• Embedded governance and compliance policies
• Enable concurrent development using standardized patterns and environments. ________________________________________
• CI/CD, Automation & Deployment
• Define and enforce standard CI/CD pipelines across data products:
• Harness (or equivalent)
• Databricks Asset Bundles (DAB)

Automate

• o DAG deployments (Airflow/Astronomer) o DBT pipeline releases
• Reduce manual interventions and ensure consistent, repeatable deployments.
• Improve release reliability with feedback loops, notifications, and monitoring. ________________________________________
• Data Product Publishing & Marketplace Enablement
• Drive publishing of data products to:
• Unity Catalog
• Enterprise Data Marketplace

Define and enforce

• Documentation standards
• Data ownership models
• Versioning and contract management
• Enable cross-domain data sharing with embedded governance and access controls. ________________________________________
• Operations, Observability & Reliability
• Establish a scalable operating model for data product support.

Implement

• Monitoring dashboards (Datadog)
• Data quality frameworks
• Usage and performance metrics tracking

Improve visibility into

• Pipeline health
• Data lineage
• Access and consumption patterns
• Lead incident management, root cause analysis, and escalation processes. ________________________________________
• Transformation, Roadmap & Innovation
• Drive execution of platform priorities such as:
• Data contract activation strategy
• Domain catalog integration
• Data masking in development environments
• Data quality frameworks o DBT adoption and POCs

Lead maturity uplift from

• Manual, fragmented workflows → standardized, automated paved paths
• Champion continuous improvement and innovation in developer experience. ________________________________________ Salary Range- $100,000-$120,000 a year </div