Lead Data Engineer

Job description

Position Summary

We are seeking an experienced Senior Data Engineer to drive the performance, governance, and AI-native maturity of our enterprise Data Platform in Databricks. This is a Databricks-focused Data Engineering role with a working understanding of DevOps practices — designing scalable pipelines, tuning workloads for performance and cost, and operationalizing modern data and AI capabilities on Lakehouse. The ideal candidate has deep, hands-on Databricks expertise, a strong performance-engineering instinct, and a builder's mindset for AI-assisted operations.

You'll own the Databricks performance and governance standards for the platform, mentor engineers, and shape the direction for AI-native operations.

Key Responsibilities

Design and develop scalable data pipelines and Lakehouse solutions on Databricks. Tune Databricks workloads for performance and cost, including cluster sizing, query optimization, and Delta Lake table design. Establish and enforce best practices for partitioning, clustering, and workload isolation. Track performance trends, identify high-cost queries, and partner with source teams and end users to resolve long-running loads.

Design and operationalize Unity Catalog for data governance — access control, lineage, and security. Build monitoring and self-healing automation using Databricks-native AI and agentic capabilities. Drive CI/CD workflows for Databricks assets, setting DevOps best practices for deployment and release management. Lead design reviews and mentor Data Engineers on Databricks best practices and AI-native features. Own Databricks vendor coordination — case management, escalations, and release adoption strategy.

What Success Looks Like (First 6–12 Months) Within 6–12 months, you'll define the platform's tuning and governance standards, lead design reviews, mentor junior engineers, and shape the AI-native operations roadmap.

Required Qualifications

Bachelor's or Master's degree in Computer Science, Information Technology, or equivalent relevant experience. 6+ years of data engineering experience with 2+ years hands-on Databricks in enterprise settings. Deep understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and Workflow orchestration. Proven ability to tune Spark workloads for cost and performance at production scale. Advanced Python (PySpark) and SQL skills.

Working knowledge of CI/CD practices and DevOps principles applied to data workloads. Experience with observability tooling for Databricks.

Preferred Qualifications

Experience with Databricks-native AI capabilities and agentic frameworks. Familiarity with Databricks Serverless Compute and DBSQL performance tuning. A Databricks Certified Professional. Exposure to Infrastructure-as-Code is a plus.

Competencies

Performance-engineering mindset — measures, tunes, and re-measures. Curiosity for AI-native operations and continuous automation. Strong sense of platform ownership — quality, cost, and reliability. Effective communication with engineering peers, vendors, and business stakeholders. Influence outcomes across source teams, vendors, and business stakeholders without direct authority. More information about NXP in India... #LI-7013