Developer

Role: Developer

Job Level:  >10 years of relevant experience

Location: Singapore

Key Responsibilities:

·        Develop robust data ingestion, transformation, and loading processes across batch and near-real-time workflows.

·        Implement distributed processing with Apache Spar k (PySpark/Scala) for large-scale data transformations and analytics.

·        Create and maintain logical and physical data models (dimensional/star schemas, data vault, or wide tables) optimized for analytics and reporting.

·        Write optimized SQL and HiveQL queries; manage tables, partitions, and storage formats.

·        Schedule and monitor pipelines using ControlM ensuring SLA adherence and timely delivery.

·        Tune Spark jobs, SQL/Hive queries, and storage strategies for scalability and cost efficiency.

·        Implement validation, reconciliation, and lineage using checks, unit tests, and metadata frameworks.

·        Build operational dashboards and alerts; diagnose failures; drive root-cause analysis and remediation.

·        Maintain clear runbooks, architecture diagrams, data dictionaries, and coding standards.

·        Apply best practices for data privacy, access control as applicable.

·        Execute continuous service improvement and process improvement plans.

·        Prepare Unit test cases and work closely with Testing team during SIT and UAT.

·        Build package and migrating the code drop through environments (e.g., DEV → QA → PROD), with audit trails and workflow governance

Key Requirements:

Education:

·        Bachelor’s degree/University degree in Computer Science, Engineering, or equivalent experience

Essential:

·        Adopt an uncompromising attitude when it comes to quality and help raise bar of products and team members

·        Be a team player who communicates effectively and professionally with both internal and external customers

·        Identify ideas to improve system performance and impact availability

·        Embrace tackling and resolving complex technical design issues

·        Possess strong problem solving and decision-making skills while exercising good judgment

·        Strong analytical and problem-solving skills

·        Ability to work on multiple projects at a time

·        Be able to work under pressure and manage deadlines or unexpected changes in expectations or requirements

·        Good communication skills - ability to convey technical information to non-technical audience

·        Ability to understand the big picture

·        Ability to develop long lasting relationships with all levels

·        Deep understanding and experience in software development cycle, including Agile based rapid delivery

·        Collaborate with business and IT to analyse, elicit and review business requirements

·        Facilitate communication between vendor, project team, business stakeholders and internal IT team

·        Ability to work in a team distributed across multiple locations

Technical Skills:

·        Programming Languages: Proficiency in Python for data processing, automation, and unix shell scripting.

·        Big Data Frameworks: Hands-on experience with Apache Spark for distributed data processing and analytics.

·        Database & Querying: Strong knowledge of SQL for relational databases and Hive for querying large datasets in Hadoop ecosystems.

·        ETL Development: Expertise in designing and implementing ETL pipelines for data ingestion, transformation, and loading.

·        Workflow Orchestration: Familiarity with Control-M or similar scheduling tools for batch job automation and monitoring.

·        Data Warehousing: Understanding of data modeling and optimization techniques for large-scale data storage and retrieval.

·        Performance Tuning: Ability to optimize queries and jobs for efficiency and scalability.

·        Version Control & CI/CD: Experience with Git and deployment pipelines for data engineering workflows.

·        BI/Analytics Integration: Familiarity with how downstream tools (Power BI/Tableau) consume curated datasets.

·        Security: IAM, secrets management, encryption at rest/in transit, PII handling.

·        Languages: Python (PySpark), SQL, Unix Shell Scripting

·        Frameworks: Spark, Hive, Sqoop

·        Orchestration: ControlM

·        Storage & Files: HDFS, Parquet, ORC

·        Version Control & CI/CD: Git, GitHub/GitLab

·        Release and Deployment: Aldon, Jenkins

·        Issue Tracking: Jira

·        Documentation: Confluence/Wiki

·        Optional: QLK sense, Tableau or any reporting dashboard