Critical Requirement - Senior Cloud Engineer

ABOUT THE ROLE
We are seeking a Senior Cloud Engineer to design, implement, and administer the Azure infrastructure behind
our enterprise data and AI platform — centered on Azure Databricks, ADLS Gen2, Azure Data Factory, and Azure
Machine Learning. You will own the platform foundation (infrastructure as code, network and identity
architecture, compute governance, observability, and cost management) and partner closely with our
Databricks administration, data engineering, and data science teams.
This role suits an engineer who enjoys being the technical reference point for a platform — setting architectural
direction, partnering with adjacent teams on complex problems, and translating business requirements into
platform capabilities.
KEY RESPONSIBILITIES
• Infrastructure as Code (Terraform-first): Author and maintain reusable Terraform modules provisioning the
Azure data estate — Databricks workspaces, ADLS Gen2, Data Factory, Key Vault, networking, and Azure ML;
own remote state, module versioning, and drift detection.

• Network & Security Architecture: Design private networking and security posture — private endpoints, hub-
and-spoke topology, VNet injection, NSGs/firewall rules, managed identities, Key Vault-backed secrets,

RBAC, and data exfiltration controls.
• Data Lake Infrastructure: Own ADLS Gen2 at the infrastructure layer — storage architecture, hierarchical
namespace, ACL strategy, lifecycle/tiering, encryption, and access patterns.
• Observability & Reliability: Build and maintain the monitoring layer — Azure Monitor, Log Analytics,
diagnostics, alerting, and operational runbooks.
• FinOps & Capacity Planning: Own cost visibility and optimization across DBU and storage spend — tagging,
chargeback/showback, budget alerts, and capacity planning.
• CI/CD & Environment Management: Own deployment pipelines and the promotion path across dev, test,
and production environments.
• Databricks Account & Workspace Administration: Account console configuration, workspace provisioning
and topology, admin role delegation, and multi-workspace strategy (day-to-day Databricks operations sit
with a dedicated admin team; this role sets the standards they work from).
• Unity Catalog: Metastore design, catalog/schema/table permission models, storage credentials,
lineage/audit, and Delta Sharing configuration.
• Identity & Access: Entra ID integration, SCIM provisioning, identity federation, service principals, and token
policy.
• Compute Governance: Cluster policies, instance pools, autoscaling/autotermination standards, and SQL
warehouse sizing.
• Cost & Usage Analysis: System tables, usage attribution, and DBU forecasting, with the ability to
recommend remediation for cost spend.
• Technical Partnership: Publish standards, reference configurations, and self-service patterns; support data
teams on cross-cutting issues.

• Cross-Boundary Diagnostics: Investigate job failures, cluster/connectivity/permission issues, and isolate
root cause across infrastructure, Databricks, or workload.
ROLE SCOPE
• Production ETL/ELT pipeline development and Spark transformation work — owned by the Data Engineering
team.
• Model development, training, and tuning — owned by the Data Science team.
• Dimensional modeling, dbt development, and BI/semantic layer work — outside this role.
REQUIRED QUALIFICATIONS (MANDATORY)
• Terraform: Expert-level, hands-on. Authored and maintained production Terraform modules, managed
remote state, and run IaC through CI/CD — a core daily skill.
• Databricks Administration: Demonstrable hands-on experience — account/workspace administration, Unity
Catalog, cluster policies, identity federation, and cost governance.
• Technical Partnership: Proven track record as senior technical resource to adjacent teams, guiding
stakeholders from request to requirement and building consensus.
• Azure Networking & Security: Deep knowledge of private endpoints, VNets/hub-spoke design, NSGs, Entra
ID, managed identities, RBAC, and Key Vault.
• Data Lake Architecture: ADLS Gen2 design and access control at enterprise scale.
• Azure Data Factory: Platform-side experience with integration runtimes, managed VNet, credential
management, and deployment automation.
• Automation & Scripting: Strong Python, plus PowerShell and/or Bash, for platform tooling and automation.
• Spark & SQL Literacy: Sufficient working knowledge to diagnose infrastructure/configuration-level
performance issues; deep Spark development is not required.
PREFERRED QUALIFICATIONS
• Databricks Asset Bundles, the Terraform Databricks provider, and workspace-as-code patterns.
• Experience leading a Unity Catalog migration or multi-workspace consolidation.
• Azure Machine Learning, MLflow, or model-serving infrastructure experience.
• Kubernetes / AKS and containerized workloads.
• Policy as code — Azure Policy, OPA, Sentinel, or Checkov.
• Event-driven infrastructure — Event Hubs, Kafka, or Stream Analytics.
• Formal FinOps practice experience.
• Multi-region or multi-tenant Databricks deployments.
PREFERRED CERTIFICATIONS
• Databricks Certified Data Engineer Professional, or a Databricks platform administrator credential.
• Microsoft Certified: Azure Solutions Architect Expert (AZ-305).
• Microsoft Certified: Azure Administrator Associate (AZ-104).
• Microsoft Certified: DevOps Engineer Expert (AZ-400).
• HashiCorp Certified: Terraform Associate.