Data Engineering Manager (15-21y)

Overview EXL   Business Consulting and Services 

Choosing a digital partner is about more than capabilities — it’s about collaboration and character. Unrealistic overhauls and off-the-shelf products ignore what matters most — your unique needs, culture, goals, and your legacy data and technology environments. At EXL, our collaboration is built on ongoing listening and learning to adapt our methodologies. We’re your business evolution partner—tailoring solutions that make the most of data to make better business decisions and drive more intelligence into your increasingly digital operations.

Role Summary 
We are looking for a senior Azure Databricks Lead / Architect to own the architecture, design, and technical delivery of enterprise-scale data platforms on Azure Databricks . The role requires strong hands-on expertise along with architecture ownership, technical leadership, stakeholder management, and mentoring of engineering teams.

Key Responsibilities

• Lead end-to-end architecture and solution design for Azure Databricks-based data platforms.
• Design scalable Lakehouse, Medallion (Bronze/Silver/Gold), Delta Lake and data engineering architectures . 
• Architect Unity Catalog for data governance, security, access control, lineage, and data discovery. 
• Design data ingestion and processing frameworks using PySpark, Spark SQL, Databricks Workflows, Auto Loader and Delta Lake .
• Define architecture for batch, streaming and CDC workloads.
• Drive integration with ADLS Gen2, Azure Data Factory, Azure Synapse, Azure SQL and other Azure services .
• Establish best practices for performance optimization, cluster/workload design, cost optimization and scalability .
• Define CI/CD, DevOps and Infrastructure-as-Code practices using Azure DevOps, Git and Terraform.
• Ensure enterprise-grade security, governance, networking, monitoring and disaster recovery .
• Provide technical direction to Data Engineers/Tech Leads and conduct architecture/design/code reviews.
• Engage with senior stakeholders, architects and business teams to translate requirements into scalable technical solutions.
• Evaluate new Databricks and Azure capabilities and drive platform modernization.
• Establish reusable frameworks, standards and engineering best practices across projects.

Must-Have Skills

• 15–21 years overall experience in Data Engineering / Data Architecture.
• Strong Azure + Azure Databricks architecture experience.
• Expert-level Databricks, Apache Spark, PySpark and SQL .
• Strong Delta Lake + Medallion Architecture experience.
• Strong Unity Catalog, governance, security and lineage experience.
• ADLS Gen2 + Azure Data Factory .
• Experience designing enterprise-scale Lakehouse/Data Platform architectures .
• Experience with CI/CD, Azure DevOps, Git and Terraform/IaC .
• Strong understanding of data modelling, data quality, performance tuning and optimization .
• Proven experience leading technical teams and mentoring senior engineers .
• Strong communication and stakeholder management skills.

Good to Have

• Databricks certifications such as Databricks Certified Data Engineer / Data Engineer Professional .
• Experience with Azure Purview/Microsoft Purview .
• Experience with Databricks SQL, Photon and serverless workloads .
• Exposure to ML/AI/GenAI platforms on Databricks .
• Multi-cloud or cross-platform data architecture exposure.
• Experience with enterprise data governance and DataOps .