About the role

Data Engineer

Job Summary:

The Data Engineer will play a vital role in our organization by designing and implementing scalable data solutions that support analytics and business intelligence initiatives. This position focuses on optimizing data pipelines and driving data integration across various platforms to enhance decision-making capabilities within the company.

Key Responsibilities:

Design and develop robust data pipelines using Python, SQL, and Apache Spark to process large datasets efficiently.
Implement data storage solutions utilizing Delta Lake and Azure Synapse Analytics for seamless data management and retrieval.
Integrate event-driven architectures using Azure Event Hubs to enable real-time data processing and analytics.
Automate workflows and CI/CD pipelines using Git, Terraform, and Infrastructure as Code (IaC) practices to ensure code quality and deployment efficiency.
Create and manage Databricks Workflows and Azure Data Factory (ADF) to orchestrate data movement and transformation tasks across the data ecosystem.
Monitor job scheduling and performance metrics to ensure reliability and timely data availability for analytics applications.
Collaborate with data scientists, analysts, and business stakeholders to align data solutions with business needs and insights.

Requirements:

Bachelor’s degree in Computer Science, Information Technology, or a related field.
Proven experience as a Data Engineer or in a similar role with expertise in Python and SQL.
Strong understanding of Apache Spark (RDDs and DataFrames) and data processing frameworks.
Hands-on experience with Delta Lake and Azure Synapse Analytics for data storage and analysis.
Familiarity with cloud platforms, specifically Azure, including Azure Event Hubs and Azure Data Factory.
Knowledge of CI/CD practices and tools, including Git and Terraform for efficient deployment.
Excellent problem-solving skills and the ability to work independently and in a team-oriented environment.

Preferred Qualifications:

Experience with Microsoft Fabric for data integration and analytics.
Familiarity with job scheduling tools and techniques to optimize data workflows.
Proficiency in other programming languages such as Scala or Java.
Understanding of data governance, data quality, and data security best practices.
Certifications related to Azure or data engineering are highly desirable.

Benefits:

Competitive salary and performance-based bonus structure.
Comprehensive health, dental, and vision insurance plans.
Retirement savings plan with company matching contributions.
Flexible working hours and remote work options to support work-life balance.
Opportunities for professional development and career advancement.
Access to cutting-edge technology and tools for data engineering.
Collaborative and inclusive company culture promoting innovation and teamwork.