databricks Tech Lead

Databricks Tech Lead

Company: EXL Services
Role: Databricks Tech Lead
Experience: 8+ Years
Location: All EXL Locations (India)
Work Mode: Hybrid

About EXL Services

EXL Services is a global leader in data analytics, AI, digital operations, and technology solutions. With deep domain expertise across Banking, Financial Services, Insurance, Healthcare, Retail, and Analytics, EXL helps organizations accelerate digital transformation through cloud, data engineering, artificial intelligence, and advanced analytics. At EXL, you'll work on cutting-edge technologies, solve complex business challenges, and collaborate with global teams to deliver impactful solutions.

Job Summary

We are looking for an experienced Databricks Tech Lead with 8+ years of experience in Data Engineering and Cloud Technologies. The ideal candidate should possess strong expertise in Azure Databricks, Azure Cloud, Python, PySpark, SQL , and modern data engineering practices. Experience with AI/Generative AI solutions and leading technical teams is highly desirable. The candidate will be responsible for designing scalable data platforms, leading development teams, and delivering enterprise-grade cloud-based data solutions.

Key Responsibilities

• Lead the design, development, and implementation of enterprise-scale data engineering solutions using Azure Databricks.
• Design and optimize scalable batch and real-time data pipelines on Azure Cloud.
• Build high-performance ETL/ELT pipelines using Python, PySpark, and SQL.
• Architect and implement modern Lakehouse and Medallion architecture solutions.
• Lead technical discussions, architecture reviews, and solution design sessions.
• Collaborate with business stakeholders to translate business requirements into scalable technical solutions.
• Drive performance tuning, code optimization, and cost optimization across Databricks workloads.
• Mentor and guide a team of data engineers, ensuring adherence to coding standards and best practices.
• Implement CI/CD, DevOps, and automation practices for data engineering solutions.
• Ensure data quality, governance, security, and reliability across enterprise data platforms.
• Evaluate and integrate AI/ML and Generative AI capabilities into modern data platforms where applicable.
• Work closely with cross-functional teams including Data Scientists, Analysts, Architects, and Product Owners.

Required Skills

• 8+ years of experience in Data Engineering and Big Data technologies.
• Strong hands-on expertise in Azure Databricks .
• Extensive experience with Azure Cloud Services .
• Strong programming skills in Python .
• Hands-on experience with PySpark for distributed data processing.
• Advanced proficiency in SQL and database optimization.
• Experience designing and developing scalable ETL/ELT pipelines.
• Strong understanding of Data Lake, Lakehouse, and Medallion Architecture.
• Experience with Delta Lake and performance optimization techniques.
• Good understanding of Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), and Azure Synapse Analytics.
• Experience working with Git, Azure DevOps, CI/CD pipelines, and Agile methodologies.
• Strong leadership, stakeholder management, and communication skills.

AI & Advanced Analytics Skills

• Experience integrating AI capabilities into enterprise data platforms.
• Exposure to Generative AI , Azure OpenAI , or Azure AI Services .
• Experience building AI-powered data solutions, intelligent data pipelines, or Retrieval-Augmented Generation (RAG) solutions is an added advantage.
• Understanding of Vector Databases, LLM integrations, Prompt Engineering, or AI orchestration frameworks is preferred.

Preferred Skills

• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS Gen2)
• Delta Lake
• Azure Synapse Analytics
• Git & Azure DevOps
• Data Modeling & Data Warehousing
• Spark Performance Tuning
• Streaming with Structured Streaming or Kafka
• AI/ML integration within modern data platforms

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• Microsoft Azure or Databricks certifications are preferred.

Why Join EXL?

• Work on large-scale cloud, AI, and data transformation programs for global enterprises.
• Collaborate with highly skilled engineering and analytics teams.
• Gain exposure to cutting-edge Azure, Databricks, and AI technologies.
• Competitive compensation, excellent career growth opportunities, and a collaborative hybrid work environment.