Senior Data Platform Engineer

Data Platform Engineer

---

About the Role

We are looking for a Data Platform Engineer to design, build, and optimize scalable data platform infrastructure and pipelines. The ideal candidate will have deep expertise in Terraform, CI/CD and DevOps practices on platforms like AWS and Snowflake. While building the platform the candidate is expected to maintain and troubleshoot data pipelines, hence knowledge of SQL and Python is necessary.

---

Key Responsibilities

• Platform Design and Development: Work with the lead to architect and build the next generation of our data platform, leveraging cloud technologies like AWS and Snowflake.
• Infrastructure as Code (IaC): Lead the implementation and maintenance of our data platform infrastructure using Terraform for automation and reliability.
• CI/CD and DevOps: Implement and manage CI/CD pipelines for data infrastructure automation and apply DevOps principles to the entire data platform lifecycle.
• Troubleshooting: Proactively monitor and troubleshoot complex data pipelines, jobs, and cloud infrastructure to ensure high availability and meet SLAs.
• Performance and Cost Optimization: Take the lead on optimizing query performance, storage costs, and compute resources within Snowflake and the AWS ecosystem.
• Mentorship and Collaboration: Collaborate closely with the Data Tech Lead and other engineers, providing guidance and technical leadership to the team.
• Data Pipeline Engineering: Design, develop, and maintain robust and scalable ETL/ELT pipelines using Python, SQL, and orchestration tools like Airflow, sqlmesh, or dbt.

---

Required Skills & Qualifications

• Snowflake: Understanding the engineering components and infrastructure of Snowflake, including knowledge of the Snowflake data platform.
• Terraform: Should be highly proficient in maintaining infrastructure through code.
• Version Control & CI/CD: Experience with version control (GitHub) and CI/CD for data pipelines, including working experience in GitHub Actions building complex pipelines.
• Programming: Working experience with Python and SQL.
• Apache Airflow: Understanding and experience of maintenance of Apache Airflow or Managed Airflow (MWAA).
• Problem-Solving: Strong problem-solving skills and ability to work independently in a fast-paced environment.
• AWS Services: Knowledge of AWS services including S3 and Lambda.