Sr/Lead Data Engineer

Senior/Lead Data Engineer

Position Summary:

We are looking for a hands-on Senior Data Engineer with a strong DevOps mindset to design, build, and operate reliable, scalable, and observable data pipelines that power business functions across the enterprise. This is a senior individual-contributor role — you'll independently own the delivery of complex pipelines, uphold engineering standards, deploy via CI/CD, support the operational health of the platform, and mentor junior engineers through reviews and collaboration.

Core Skills:

Databricks · Python (PySpark) · SQL · Data Pipelines · CI/CD

Key Responsibilities:

Engineering & Delivery:

• Independently design, build, and maintain complex, production-grade data pipelines on Databricks.
• Develop efficient ETL/ELT processes with a strong focus on data quality, consistency, and scalability.
• Build reusable frameworks for ingestion, transformation, and reconciliation across enterprise source systems.
• Apply and help improve engineering standards — pipeline architecture, coding standards, and ETL/ELT best practices.

Technical Mentorship:

• Mentor junior engineers through code reviews, design reviews, and pair-programming on complex problems.
• Share best practices in Databricks/PySpark, coding standards, and engineering discipline.
• Contribute to a culture of ownership, automation, and continuous improvement.

Operations & DevOps:

• Deploy changes through CI/CD and the Change Request (CR) lifecycle, including validation, release management, and ticket closure.
• Participate in problem management and root-cause analysis — driving permanent fixes and automation over recurring firefighting.
• Support the operational health of business-critical data workloads — monitoring, alerting, and incident response.

Collaboration:

• Partner with Reporting, Visualization, Platform, and Business teams to expose curated datasets for downstream analytics consumers.
• Communicate technical trade-offs, progress, and risks clearly to technical and non-technical stakeholders across geographies.
• Document workflows, standards, and runbooks to ensure reproducibility and knowledge continuity.

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Information Technology, or equivalent relevant experience.
• 4-8 years of experience in data engineering.
• Strong hands-on background in Databricks, Python (PySpark), and SQL for large-scale data processing.
• Proven experience designing and delivering production data pipelines (ETL/ELT) at enterprise scale.
• Working knowledge of CI/CD pipelines, Git-based branching strategies, and DevOps practices.
• Experience with cloud platforms (AWS preferred) and core data services.
• Experience supporting production data pipelines, including monitoring, alerting, and incident response.
• Strong communication skills across engineering and business audiences.

Preferred Qualifications:

• Experience with orchestration frameworks and streaming technologies.
• Exposure to Infrastructure-as-Code and modern deployment tooling.
• Familiarity with observability tooling for data platforms.
• Background in semiconductor manufacturing or large-scale industrial data processing.
• Databricks Certified Data Engineer Associate or Professional certification is a plus.