Azure Databricks Engineer

Key Responsibilities
 

• Design, build, and maintain ETL/ELT pipelines on Databricks using PySpark , Spark SQL , and Delta Lake .
   
• Develop and optimize data ingestion frameworks, data transformations, and end‑to‑end workflows for batch and streaming use cases.
   
• Implement Delta Lake‑based architectures, including versioning, schema evolution, and ACID‑compliant pipelines.
   
• Work with stakeholders to understand data requirements and translate them into scalable data engineering solutions.
   
• Manage and optimize Databricks clusters, jobs, and notebooks for performance and cost efficiency.
   
• Integrate Databricks pipelines with cloud services (Azure Data Lake / AWS S3, Event Hub/Kafka, Synapse/Redshift, etc.).
   
• Ensure data quality , reliability, and observability through validation frameworks and monitoring.
   
• Contribute to data modeling , metadata management, and best practices within the data platform.
   
• Collaborate closely with data scientists, analysts, and business teams to support analytics and ML workloads.
   

 
Must have Skills & Experience
 

• 5 -10 years of experience in data engineering with strong expertise in Databricks .
   
• Hands‑on experience with Spark (PySpark/Spark SQL) and distributed data processing.
   
• Solid SQL knowledge and experience working with large-scale datasets
   
• Strong understanding of Delta Lake , medallion architecture, and scalable lakehouse patterns.
   
• Good understanding of CI/CD, Git, and modern DevOps practices for data pipelines.
   
• Familiarity with structured/unstructured data, data quality frameworks, and performance tuning.