Lead Data Platform Engineer - AWS, Spark

Job description

Experian Credit Services & Verifications are looking for a Lead Data Platform Engineer to provide hands-on technical leadership for a new fraud data capability that will connect fraud-related data assets into Experian's Data Analytics Platform. Reporting into our Engineering Director, you will lead a technical team that supports innovation, learning and collaboration. Ensuring that the team follows the highest technical standards.

You will work closely with the central Data Analytics Platform team, data owners, security, compliance, platform and delivery stakeholders to agree practical implementation approaches, align with established platform patterns, and ensure the team delivers secure, scalable and supportable solutions. This is a hands-on leadership role suited to someone who can combine strong data engineering depth with cloud platform experience, engineering discipline and team leadership.

You will apply effective DevSecOps practices and contribute to technical roadmap discussions where there are shared or overlapping platform requirements, while working collaboratively with central platform teams where broader architectural ownership sits outside the local team. This is a Hybrid, Nottingham based role (40% in the office). Key responsibilities Lead and support an agile technical team delivering data platform solutions for fraud-related data assets.

Help guide the delivery of AWS-based data pipelines, data lake components and producer-side data platform capability. Support the handover, ownership, maintenance and enhancement of pipelines and cloud components initially developed by the central DAP team. Lead the team in building, testing and maintaining high-quality data pipelines from source systems such as Snowflake and internal product/data platforms. Ensure metadata is prepared and published accurately so assets can be surfaced through the DAP catalogue.

Champion DevSecOps practices, CI/CD, Infrastructure as Code and cloud-native tooling to improve reliability, security and delivery efficiency. Mentor and coach senior and junior engineers, supporting innovation, learning and collaboration within the team. Promote modern engineering practices, including automation and AI-assisted development where appropriate, to improve delivery speed, quality and repeatability.

Qualifications

Strong experience building data pipelines, data lake solutions or data platform capabilities. Strong experience with AWS cloud services relevant to data engineering and platform delivery, including services such as AWS Glue, S3, Step Functions, IAM, Lake Formation, EC2, Lambda, EventBridge and SQS. Strong experience with DevOps and Infrastructure as Code tooling, including Terraform, CI/CD pipelines, Git and Jenkins. Strong experience developing on Apache Spark or Hadoop.

Significant programming experience using Scala or Python. Experience with Configuration as Code principles and API integration. Strong understanding of cloud security, data privacy, access control and governance principles in data platform environments Desired skills AWS Developer Certification or other relevant AWS certification. Strong expertise in Terraform or CloudFormation. Experience with OpenShift or Kubernetes. Experience with automated testing tools such as pytest or Scalatest.

Familiarity with code quality and security scanning tools such as SonarQube and Veracode. Knowledge of RESTful APIs and microservice architectures. Experience working with sensitive, governed or regulated data environments. Familiarity with data catalogue, metadata management, data marketplace or federated data mesh concepts.