Lead Software Engineer

Lead Data Engineer

Location: Singapore

Mode of Work: Initial 2 Months work from VISA Client Office.

Role Overview

The Lead Data Engineer will be responsible for designing, building, and optimizing scalable data pipelines. The role requires strong hands-on experience in data engineering and distributed data processing.

Key Responsibilities

• Design, build, and maintain ETL/ELT pipelines.
• Develop and optimize data ingestion frameworks, data transformations, and end-to-end workflows for batch and streaming use cases.
• Work with stakeholders to understand data requirements and translate them into scalable data engineering solutions.
• Manage and optimize codes.
• Ensure data quality, reliability, and observability through validation frameworks and monitoring.
• Contribute to data modelling, metadata management, and best practices within the data platform.
• Collaborate closely with data scientists, analysts, and business teams to support analytics and ML workloads.

Must have Skills & Experience

• 10+ years of experience in data engineering development.
• Hands-on experience with 5+ years in PySpark distributed data processing.
• 3+ years of in-depth hands-on experience in Python.
• At least 3+ years of experience in Hadoop.
• 3+ years of experience as a Data Engineer in AWS or Azure.
• Solid SQL knowledge and experience working with large-scale datasets.
• Good understanding of Delta Lake, medallion architecture, and scalable lakehouse patterns.
• Good understanding of CI/CD, Git, and modern DevOps practices for data pipelines.
• Familiarity with structured/unstructured data, data quality frameworks, and performance tuning.