Specialist Software Engineer - Data Engineer Databricks pySpark

Job description

Specialist Software Engineer – Data Engineer (Databricks-pySpark)

Responsibilities

Understand business and technical requirements and translate them into scalable, maintainable data engineering solutions. Own the design and development of robust, high‑performance data pipelines using Databricks(pySpark) . Build and optimize data flows within the Hadoop ecosystem (Hive, HDFS, Starburst, Oozie). Ensure performance tuning, resource optimization, and efficient cluster usage on Databricks platform. Contribute to data architecture decisions related to storage, processing, security, and exposure.

Apply industrialization best practices including CI/CD, unit testing, integration testing, and pipeline validation. Model data based on business needs and maintain shared data components, dictionaries, and governance rules. Perform functional and technical analysis, document specifications, and validate implemented solutions. Participate in Agile ceremonies and collaborate with Product Owners, architecture, business, and Ops teams.

Support production processes, troubleshoot issues, update runbooks, and maintain system stability. Ensure compliance with data quality, traceability, and security standards across the data lifecycle. Engage in continuous improvement activities, proposing upgrades, refactoring, and automation opportunities. Work closely with global teams and support cross‑location coordination when required. Demonstrate ownership during key delivery phases, including planned on‑call or deployment activities.