Databricks Architect

Databricks Architect

Job Summary:

The Databricks Architect will play a pivotal role in designing and implementing scalable data solutions leveraging Databricks, enhancing EXL's data capabilities. With a focus on optimizing processes and fostering data-driven decision-making, this position will drive our commitment to innovation and operational excellence.

Key Responsibilities:

• Design and architect robust data solutions on the Databricks platform to meet complex business needs.
• Lead end-to-end data pipeline development and optimization, ensuring high performance and reliability.
• Collaborate with data engineering and analytics teams to establish best practices for data management and processing.
• Mentor and train team members on Databricks technologies and data engineering methodologies.
• Conduct performance tuning and troubleshooting of existing data solutions to maximize efficiency.
• Stay abreast of industry trends and emerging technologies to recommend process improvements and new tools.
• Facilitate cross-functional collaboration to align data strategies with business objectives.

Requirements:

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• 15+ years of experience in data architecture, data engineering, or a similar role.
• Extensive expertise in Databricks and Apache Spark, including data ingestion, processing, and analysis.
• Proficient in cloud technologies, preferably AWS or Azure, with a focus on data services.
• Strong programming skills in languages such as Python, Scala, or Java.
• Demonstrated experience with big data technologies and frameworks, including Delta Lake and SQL.
• Excellent communication and leadership skills, capable of working in a collaborative environment.

Preferred Qualifications:

• Experience with machine learning frameworks and applications in a big data context.
• Knowledge of data governance practices and experience with data quality frameworks.
• Certifications in Databricks or related cloud platforms, highlighting commitment to ongoing professional development.
• Familiarity with Agile methodologies and project management tools.