Databricks

Data Engineer

Job Summary:

The Data Engineer will play a crucial role in managing and optimizing our data pipeline, ensuring data integrity, and supporting the analytics team with high-quality datasets. This position is key to transforming raw data into actionable insights that drive strategic business decisions.

Key Responsibilities:

• Design and develop scalable architectures for data processing and storage using Databricks.
• Implement ETL pipelines to ingest data from various sources into data warehouses and lakes.
• Monitor and optimize existing data pipelines for performance and reliability.
• Collaborate with data scientists and analysts to understand data needs and provide high-quality datasets.
• Ensure data quality and integrity through testing and validation processes.
• Maintain documentation of data engineering processes, data schemas, and ETL workflows.

Requirements:

• Bachelor's degree in Computer Science, Information Technology, or related field.
• Proven experience in data engineering or software development roles.
• Strong proficiency in SQL and programming languages such as Python or Scala.
• Demonstrated expertise in using Databricks for data engineering tasks.
• Experience with cloud platforms like AWS, Azure, or Google Cloud Platform.
• Databricks certification is mandatory.

Preferred Qualifications:

• Experience with machine learning frameworks and data analytics tools.
• Knowledge of data modeling, data warehousing, and database design concepts.
• Familiarity with Apache Spark and its ecosystem.
• Strong analytical skills with the ability to solve complex data problems.