Data Engineering Lead

Job Summary

Amara Raja Batteries is seeking an innovative and detail-oriented Data Engineering Lead to design, develop, and optimize data infrastructure across business domains. This role involves managing data pipelines, integrating multimedia data into a structured medallion data lake-house. The ideal candidate will collaborate with cross-functional teams to ensure reliable and scalable data architecture on Azure platform to support data scientists, analysts and Ai engineers.

About the Company

<Team to add>

Key Responsibilities

Key Accountabilities Areas

Key Activities

Data Pipeline Ownership

·          Design, build, and maintain scalable data pipelines to integrate data from multiple systems.

·          Enable efficient data flow for real-time monitoring and predictive analytics aligned with business objectives.

Collaboration with Data Teams

·          Partner with data analysts and data scientists to provide clean, structured, and accessible data for advanced analytics.

·          Collaborate to build transformation Silver pipelines and semantic Gold layer for business analytics.

IoT and Manufacturing Data Integration

·          Integrate IoT-generated data with manufacturing systems to enable real-time insights and decision-making.

·          Drive IT-OT convergence to enhance connectivity and visibility across digital and physical manufacturing environments.

Data Infrastructure Management

·          Optimize data storage and processing systems to support AI and digital use cases.

·          Develop scalable architectures to handle large volumes of high frequency transactional and slow moving enterprise data.

ETL & Workflow Automation

·          Develop and automate ETL (Extract, Transform, Load) processes to ensure accurate and timely data integration.

·          Implement frameworks to maintain data accuracy, integrity, and security in compliance with ISO 9001 and other industry standards.

·          Ensure adherence to data governance.

Required skills

·          Proficiency in tools like SQL, Python, or ETL frameworks for managing and processing data.

·          Azure tools like Databricks, Azure Data Factory and Fabric.

·          Understanding of IoT protocols (e.g., MQTT, OPC-UA) and ability to process IoT data for real-time insights.

·          Expertise in relational databases (MySQL, PostgreSQL) and knowledge of non-relational databases as needed.

·          Proven ability to design, automate, and optimize ETL pipelines for manufacturing data.

·          Broad expertise in at least one Cloud platform, preferably Azure.

·          Working knowledge of automation and deployment tools.

Data Governance & Security

·          Knowledge of data governance principles and compliance with industry regulations like ISO 9001 and GDPR.

Nice to Have

·          Understanding of smart manufacturing concepts, including predictive maintenance and digital twins.

Technical Proficiency

• Good understanding of DevOps, CI/CD pipelines, orchestration, and containerization tools like Docker and Kubernetes.

Qualifications

Work Experience

·          5-8 years of experience in data engineering, with a strong focus on Azure platform.

·          Proven track record of integrating data from MES, ERP, CRM and manufacturing systems.

Educational Background

·          Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related field.

·          Coursework or specialization in data architecture, analytics in Azure platform (preferred).

Preferred Industry Experience

·          Manufacturing, energy, or related industries with exposure to Industry 4.0 .

Technical Expertise

·          Expertise in developing and optimizing data pipelines, ETL processes, and managing large-scale data systems in Azure (preferred).

·          Familiarity with tools and platforms like SQL, Python, Databricks, Airflow, and IoT protocols (e.g., MQTT, OPC-UA).

Hiring Process and Next Steps

Application Process

•     <Team to add>