This is a hands-on engineering role. You will build and support production data pipelines in Databricks, work with the broader delivery team to understand requirements, and contribute to the development and quality of the platform.
What you'll do
- Build and tune ingestion and transformation pipelines in Databricks using PySpark, SQL, and Delta Lake.
- Work with the delivery team and client data owners to understand source systems, data quality issues, and business rules.
- Translate defined requirements into clear, testable data engineering tasks.
- Build and maintain medallion (bronze, silver, gold) data layers and associated quality checks.
- Set up and support jobs, orchestration, monitoring, and CI/CD processes.
- Troubleshoot data quality, pipeline performance, and processing issues.
- Participate in code reviews and collaborate with engineers across U.S. and offshore teams.
What you bring
- 3–5 years of data engineering experience, including hands-on experience building production pipelines in Databricks.
- Strong PySpark and SQL skills.
- Hands-on experience with Delta Lake and exposure to Unity Catalog.
- Experience working in AWS or Azure environments.
- Experience working within distributed or global engineering teams.
Nice to have
- Asset management or investment data experience, such as holdings, positions, transactions, or security reference data.
- Databricks Data Engineer certification.
- Experience with dbt, Airflow, or Kafka.