Senior Lead Data Engineer

About the Role

We are looking for an experienced Senior Lead Data Engineer to join a high-impact data engineering team supporting a leading financial services organization.

The role involves designing and delivering enterprise-scale data platforms and data pipelines for complex banking and card-related data environments. The ideal candidate will have strong hands-on expertise in Databricks, PySpark, AWS Glue, and AWS data services, along with proven experience in data engineering architecture and technical leadership.

You will work closely with senior stakeholders and technical leaders across India and the US to design scalable, secure, and highly reliable data solutions.

Key Responsibilities

• Lead the design and development of scalable, enterprise-grade data engineering solutions on AWS and Databricks.
• Design and optimize high-volume ETL/ELT pipelines using PySpark, Databricks, AWS Glue, and related AWS services.
• Define and implement data lake/lakehouse architectures, including Medallion architecture and scalable data processing frameworks.
• Drive technical architecture, design decisions, coding standards, performance optimization, and engineering best practices.
• Develop reliable batch and incremental data pipelines with appropriate validation, monitoring, error handling, and recovery mechanisms.
• Work with large-scale financial and transactional datasets, ensuring performance, quality, security, and reliability.
• Optimize Spark workloads using techniques such as partitioning, broadcast joins, caching, AQE, skew handling, file optimization, and query tuning.
• Work with services such as Amazon S3, AWS Glue, Athena, EMR, Redshift, Lambda, and related AWS technologies.
• Implement data governance, data quality, lineage, auditability, and data retention practices.
• Collaborate with product, business, architecture, and technical stakeholders to translate requirements into scalable data solutions.
• Provide technical guidance and mentorship to junior and mid-level data engineers.
• Participate in architecture reviews, technical discussions, code reviews, production support, and root-cause analysis.
• Work closely with US-based technical and product stakeholders and contribute to cross-functional delivery.

Required Technical Skills

• 8–12 years of overall experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with Databricks.
• Strong hands-on experience with PySpark / Apache Spark.
• Strong experience with AWS Glue and AWS data engineering services.
• Strong understanding of Data Engineering Architecture and enterprise data platforms.
• Strong experience designing and implementing Data Lakes / Lakehouse architectures.
• Strong programming experience in Python and SQL.
• Experience with large-scale ETL/ELT pipelines and distributed data processing.
• Experience with Delta Lake / Parquet and data storage optimization.
• Strong understanding of data pipeline performance optimization.
• Experience with cloud data services, preferably AWS.
• Proven experience providing technical leadership, mentoring engineers, and driving technical decisions.

Preferred / Good-to-Have Skills

• Experience in Banking / BFSI / Financial Services.
• Experience working with Credit Card / Card Platforms / Card Transactions / Payments.
• Experience with Amazon Athena.
• Experience with Parquet optimization and large-scale file optimization.
• Knowledge of data retention, data governance, data security, and compliance.
• Understanding of symmetric and asymmetric cryptography.
• Experience with transaction, payment, fraud, or other high-volume financial datasets.
• Experience working with distributed/global teams and US-based stakeholders.

What We Are Looking For

The ideal candidate will be someone who:

• Can independently own complex data engineering architecture and technical decisions.
• Is comfortable working hands-on with Databricks, PySpark, AWS Glue, and AWS.
• Has experience leading engineering teams or acting as a Technical Lead / Data Engineering Lead.
• Understands performance, scalability, reliability, security, and governance at an enterprise level.
• Can communicate effectively with technical and business stakeholders.
• Has experience working on complex financial-services or transaction-data platforms.

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.

Why Join

• Opportunity to work on a large-scale financial services data platform.
• Exposure to complex banking and card-related data engineering challenges.
• Work with modern technologies including Databricks, PySpark, AWS Glue, AWS and Lakehouse architectures.
• Collaborate with experienced technical and business leaders across India and the US.
• Opportunity to take ownership of architecture and technical leadership responsibilities.