Senior Data Engineer, Data Lake

Job description

About Us

Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be paid. At Visa, you'll have the opportunity to create impact at scale — tackling meaningful challenges, growing your skills and seeing your contributions impact lives around the world.

Join Visa and do work that matters – to you, to your community, and to the world. Progress starts with you.

Job Description

Summary We're looking for a Data Engineer at the Analyst level to join our Data Lake team. You'll build and maintain data ingestion and transformation pipelines using Spark, Databricks, and Airflow, contributing to the reliability and quality of the corporate data lake. This is an intermediate role where you'll work under moderate supervision on defined tasks while building deeper expertise in lakehouse architecture. At Pismo, the Data Lake team is responsible for centralizing and organizing data into a single, trusted platform that supports decision-making across the company and for external clients.

We work on challenges such as scaling global data infrastructure, delivering high-quality reporting, and enabling secure, self-service access to data—helping teams move faster while avoiding information silos.

What You'll Do

Develop, test, and maintain data pipelines (ingestion, transformation, quality checks) using PySpark/SparkSQL on Databricks . Build and modify Airflow DAGs (MWAA) for pipeline orchestration. Write and optimize SQL queries for data transformation and validation. Support data quality by implementing and monitoring quality checks (Great Expectations or equivalent). Participate in code reviews — both giving and receiving feedback.

Investigate pipeline failures and data quality issues with guidance from senior engineers. Write and maintain documentation for datasets and pipelines you build. Participate in sprint planning, estimation, and retrospectives. Follow CI/CD, testing, and governance standards defined by the team. This is a remote position. A remote position does not require job duties be performed within proximity of a Visa office location. Remote positions may be required to be present at a Visa office with scheduled notice. #LI‑Remote Qualifications For this role, you must be based in Brazil Language Skills Proficiency in English at B2 level or above (Upper-Intermediate)

Basic Qualifications

3+ years of relevant work experience with a Bachelor's or Associate’s Degree OR 5+ years of relevant work experience. Python for automation and data processing Data pipeline concepts (batch, ETL/ELT patterns)

Technical Skills

Apache Spark basics (PySpark or SparkSQL) Databricks (jobs, workflows, cluster management, tuning) SQL (intermediate–advanced: joins, window functions, CTEs) Amazon S3 data lake design (partitioning, layout, lifecycle) Basic cloud concepts (AWS: S3, IAM, CloudWatch) Data modeling (dimensional / Kimball, medallion layers)

Git/GitHub workflow and code review Preferred Qualifications

Delta Lake (ACID tables, OPTIMIZE, VACUUM, schema evolution, MERGE) Apache Airflow / MWAA (DAG design, retries, backfills, SLAs) CDC patterns (DMS, incremental processing, MERGE upserts) Data quality concepts (validation, testing) Terraform basics BI integration (Superset, dashboarding) Visa is an EEO Employer Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status.

Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.