About the role

Senior Data Engineer / Platform Re-Engineering Lead (Azure Synapse & Databricks Migration)

Job Summary:

We are seeking a highly experienced Senior Data Engineer to spearhead the re-engineering of our existing enterprise data platform utilizing Azure Synapse Analytics. This pivotal role involves auditing, understanding, and validating complex data architectures, ultimately guiding the migration of workloads to Databricks while ensuring data integrity and governance.

Key Responsibilities:

Lead the technical assessment and re-engineering of the enterprise data platform, covering all layers from source ingestion to data consumption.
Reverse-engineer, document, and validate current pipeline logic, data models, transformation frameworks, and data governance controls.
Identify gaps, defects, and technical debt, remediating any incorrect or sub-optimal implementations.
Ensure accuracy in data processing patterns, including change data capture, slowly changing dimensions, deduplication, and business reconciliation.
Design and implement target-state architectures aligned with modern lakehouse principles, maintaining business logic integrity during transitions.
Manage platform evolution initiatives, validating output consistency during cutovers between parallel implementations.
Define and execute migration strategies for existing workloads, ensuring adherence to governance and control frameworks.

Requirements:

10+ years of experience in Data Engineering, particularly in complex platform migration or re-engineering projects.
Strong hands-on experience with Azure Synapse Analytics, including Pipelines, Spark Pool, and Dedicated SQL Pool.
Proficiency with Azure Data Lake Storage Gen2 and Delta Lake, employing Synapse Lakehouse patterns.
Experience with integrating real-time data sources using Oracle Golden Gate Replication.
Demonstrated knowledge of medallion architecture and data processing patterns such as SCD, CDC, and quality validation.
Strong programming skills in Python and SQL for ETL/ELT processes at scale.
Experience in managing CI/CD pipelines and utilizing Infrastructure as Code methodologies.

Preferred Qualifications:

Hands-on experience with Azure Databricks, particularly Delta Live Tables and Unity Catalog.
Familiarity with real-time streaming pipelines like Event Hub, Kafka, or Kinesis.
Exposure to monitoring and observability tools such as Dynatrace.
Experience in integrating with BI tools like Power BI or Tableau.
Knowledge of GenAI or machine learning platform integration and MLOps.

Benefits:

Competitive salary with performance-based bonuses.
Comprehensive health, dental, and vision insurance plans.
401(k) retirement plan with company matching.
Flexible work arrangements, including remote work options.
Opportunities for professional development and certifications.
Generous paid time off policy and holidays.
A collaborative and inclusive work environment that values diversity.