Loading open roles
Vrinda Global
Databricks Lead
Vrinda Global · posted 5 months ago
Gurugram
About the role
Key Responsibilities
-
Design, build, and maintain ETL/ELT pipelines on Databricks using PySpark,
Spark SQL, and Delta Lake.
-
Develop and optimize data ingestion frameworks, data transformations, and
end‑to‑end workflows for batch and streaming use cases.
-
Implement Delta Lake‑based architectures, including versioning, schema
evolution, and ACID‑compliant pipelines.
-
Work with stakeholders to understand data requirements and translate them
into scalable data engineering solutions.
-
Manage and optimize Databricks clusters, jobs, and notebooks for performance
and cost efficiency.
-
Integrate Databricks pipelines with cloud services (Azure Data Lake / AWS
S3, Event Hub/Kafka, Synapse/Redshift, etc.).
-
Ensure
data quality
, reliability, and observability through validation frameworks and
monitoring.
-
Contribute to
data modeling
, metadata management, and best practices within the data platform.
-
Collaborate closely with data scientists, analysts, and business teams to
support analytics and ML workloads.
Must have Skills & Experience
-
8-12 years of experience in data engineering with strong expertise in
Databricks
.
-
Hands‑on experience with
Spark
(PySpark/Spark SQL) and distributed data processing.
-
Solid SQL knowledge and experience working with large-scale datasets
-
Strong understanding of
Delta Lake
, medallion architecture, and scalable lakehouse patterns.
-
Good understanding of CI/CD, Git, and modern DevOps practices for data
pipelines.
- Familiarity with structured/unstructured data, data quality frameworks, and performance tuning