Job Description - Specialist Software Engineer - Python, Elastic
Search, Druid (25000M6Z)
Specialist Software Engineer - Python, Elastic
Search, Druid
-
(
25000M6Z
)
Missions
J
ob Summary:
We are seeking a highly skilled and motivated
Specialist Software Engineer
with deep expertise in
Big Data technologies
,
data pipeline orchestration
, and
observability tooling
. The ideal candidate will be responsible for
designing, developing, and maintaining
scalable data processing systems and
integrating observability solutions to ensure
system reliability and performance.
Key Responsibilities:
Big Data Engineering:
-
Design and implement robust data pipelines
using
Apache Kafka
,
Apache NiFi
,
Apache Spark
, and
Sqoop
.
-
Manage and optimize distributed data storage
systems including
Hadoop
,
HDFS
,
Druid
, and
ElasticSearch
.
-
Integrate and maintain data visualization
and monitoring tools like
Kibana
,
Grafana
, and
Logstash
.
-
Ensure efficient data ingestion,
transformation, and delivery across various
platforms.
Programming & Scripting:
-
Develop automation scripts and data
processing utilities using
Python 3
and
Shell scripting
.
-
Build reusable components and libraries for
data manipulation and system integration.
Observability & Monitoring:
-
Implement and configure observability agents
such as
Fluentd
,
Telegraf
, and
Logstash
.
-
Collaborate with platform teams to integrate
OpenTelemetry
for distributed tracing and metrics
collection (good to have).
-
Maintain dashboards and alerts for system
health and performance monitoring.
DevOps & CI/CD:
-
Contribute to CI/CD pipeline development
using
GitHub Actions
.
-
Collaborate with DevOps teams to ensure
seamless deployment and integration of data
services.
Collaboration & Documentation:
-
Work closely with cross-functional teams
including data scientists, platform
engineers, and product managers.
-
Document system architecture, data flows,
and operational procedures.
-
Participate in code reviews, knowledge
sharing sessions, and technical mentoring.
Required Skills & Qualifications:
-
Bachelor’s or Master’s degree in Computer
Science, Information Technology, or related
field.
-
5+ years of experience in Big Data
engineering and scripting.
-
Strong hands-on experience with:
-
Kafka, NiFi, Hadoop, HDFS, Spark,
Sqoop
-
ElasticSearch, Druid, Kibana, Grafana
-
Python3, Shell scripting
-
Logstash, Fluentd, Telegraf
-
Familiarity with
GitHub Actions
and basic DevOps practices.
-
Exposure to
OpenTelemetry
is a plus.
-
Excellent problem-solving, analytical, and
communication skills.
Preferred Qualifications:
-
Experience in building real-time data
streaming applications.
-
Knowledge of data governance, security, and
compliance in Big Data environments.
-
Certifications in Big Data technologies or
cloud platforms (AWS/GCP/Azure) are a plus.
|
|
