Azure PySpark Data Engineer

Job description

Develop and maintain ETL pipelines using PySpark / Spark SQL in Azure Databricks . Process and transform large-scale financial data for various business and compliance applications. Implement data pipelines ensuring data quality, governance, and lineage tracking . Integrate data from different systems such as core banking, payments, and trading platforms . Optimize Spark jobs for high performance and cost efficiency . Collaborate with business teams and compliance departments to align on project requirements.

Apply error handling, logging, and monitoring techniques during pipeline development. Perform advanced data transformations and develop user-defined functions (UDFs) where necessary. Use Unix scripting, scheduling tools , and maintain generic process development and performance tuning . Support audit requirements and ensure compliance with regulatory standards </ul

Qualifications

BACHELOR OF ENGINEERING