Databricks Data Engineer (Spark AND Unity)
Unison Group
- Location
- Onsite (Bengaluru, Karnataka)
- Employment
- Full-time
- Level
- Mid Level
Posted 2 weeks ago
About the role
Unison Group, a sustainability advisory firm, is seeking a Data Engineer to build scalable data pipelines using Azure Databricks. The role focuses on delivering reliable ETL processes and supporting data-driven insights for Central Asia projects.
Skills
Azure Databricks
PySpark
SQL
Python
Delta Lake
Unity Catalog
Azure DevOps
ETL
Data Pipelines
Medallion Architecture
Delta Live Tables
ADLS Gen2
Azure SQL Database
Bicep
CI/CD
Data Modeling
Full job details
Interview - Face to face
Position Overview:
Build and maintain scalable data pipelines using Azure Databricks to support DATA & AI ACDP projects. Apply 3-4 years of experience to deliver reliable ETL processes and collaborate on data-driven insights.
Key Responsibilities:
- Design, build, and optimize data pipelines in Azure Databricks for ingestion, ETL/ELT, and transformations, implementing Medallion Architecture (Bronze, Silver, Gold) with Delta Lake for data quality and versioning.
- Utilize Databricks Spark (PySpark, SQL), Delta Live Tables, and Unity Catalog for pipeline development, governance, and basic streaming.
- Integrate data from ADLS Gen2, Azure SQL Database, and SQL Pools; support orchestration via Databricks workflows.
- Implement CI/CD in Azure DevOps for Databricks deployments, including Bicep templates and ADF integration.
- Apply data quality measures (expectations, constraints), security (RBAC, encryption), and monitoring for performance/cost efficiency using auto-scaling and Photon.
- Work with analysts and stakeholders to refine requirements into functional workflows; handle data cleansing, modeling, and Python/SQL scripting for structured/unstructured data.
- Troubleshoot pipelines and contribute to documentation/best practices.
Required Qualifications:
- 3-4 years in data engineering, focused on Azure Databricks pipelines and ETL
- Proficiency in PySpark, SQL, Python; experience with cloud storage and basic orchestration
- Familiarity with Azure fundamentals and collaborative tools