- Company Name
- CloudTech Innovations
- Job Title
- Lead Databricks Data Engineer
- Job Description
-
**Job title**
Lead Databricks Data Engineer
**Role Summary**
Lead architect and engineer for large‑scale data pipelines on Databricks, designing, implementing, and optimizing batch and streaming solutions that meet regulatory and business requirements in regulated sectors such as banking, finance, or insurance.
**Expectations**
- 8‑10 years of data engineering experience, with proven leadership in complex environments.
- Deep expertise in Databricks, Apache Spark, and Delta Lake across major cloud platforms.
- Ability to work independently, make pragmatic decisions, and mentor a small team of data engineers.
**Key Responsibilities**
- Design, develop, and own end‑to‑end ETL/ELT pipelines using Databricks and Spark for batch/streaming workloads.
- Build and maintain Delta Lake lakehouse solutions using Medallion architecture (Bronze, Silver, Gold) for analytics and ML use cases.
- Translate regulatory and business requirements into reliable data solutions with solution architects and stakeholders.
- Implement data governance, security, and access controls via Unity Catalog, IAM, encryption, and audit‑ready practices.
- Integrate Databricks with cloud services (AWS, Azure, GCP), including data storage, ingestion, and orchestration tools.
- Design and automate workflows with Airflow, dbt, or cloud‑native schedulers, ensuring reliability, observability, and cost‑efficiency.
- Tune Spark jobs and Databricks clusters for performance and SLAs.
- Apply DevOps/DataOps best practices: CI/CD pipelines, version control, automated testing.
- Support legacy Power BI reporting during modernization efforts.
- Provide technical guidance, mentorship, and set standards across engineering team.
**Required Skills**
- Databricks, Apache Spark, Delta Lake production expertise.
- Strong proficiency in Python or Scala; advanced SQL, performance tuning, data modeling.
- Experience with at least one major cloud platform (AWS, Azure, GCP).
- Hands‑on with orchestration tools: Airflow, dbt, Step Functions, or equivalent.
- Knowledge of data governance, security, and compliance in regulated environments.
- Problem‑solving mindset, ability to work independently.
**Required Education & Certifications**
- Bachelor’s degree in Computer Science, Data Engineering, or related field (or equivalent experience).
- Databricks Certified Data Engineer Associate preferred.
- Cloud data engineering certifications (AWS Data Analytics, Azure Data Engineer Associate, or GCP Data Engineer) desirable.
---