Original job description
Experience: 5+ Years
Primary Skills: Azure Databricks, PySpark, SQL, ETL/Data Engineering, Azure Data Factory, Streaming
Cloud Platform: Microsoft Azure
Role Summary
We are looking for an experienced Azure Data Engineer to design, develop, and support scalable data ingestion and processing solutions on the Microsoft Azure platform. The candidate will be responsible for building reusable data engineering frameworks, integrating end-to-end data pipelines, supporting batch and streaming workloads, and ensuring data quality, reliability, and consistency across the data platform.
The role also requires strong troubleshooting capabilities and experience supporting production applications across L1, L2, and L3 support activities.
Key Responsibilities
Design, develop, and implement data ingestion pipelines from multiple structured and unstructured data sources using Azure Databricks.
Develop scalable, configurable, and reusable data ingestion frameworks that can support multiple datasets and source systems.
Build and maintain end-to-end data pipelines, moving data from source systems through transformation layers into target data repositories.
Develop data transformation and processing solutions using PySpark, Spark SQL, and SQL.
Implement data validation, reconciliation, error handling, audit logging, and data-quality controls to maintain accuracy and consistency throughout the pipeline.
Design and develop both batch and streaming data pipelines.
Work with event-based and streaming technologies such as Apache Kafka, Azure Event Hubs, or Spark Structured Streaming, depending on project architecture.
Develop and orchestrate data pipelines using Azure Data Factory (ADF) and integrate them with Azure Databricks.
Work with Azure data services such as Azure Data Lake Storage Gen2 (ADLS Gen2) and related target repositories.
Optimize Databricks/Spark workloads for performance, scalability, and reliability.
Collaborate with cross-functional teams responsible for components such as APIs, search capabilities, source applications, and downstream consuming systems.
Evaluate appropriate data engineering tools and technologies against functional, technical, performance, and customer requirements.
Troubleshoot data pipeline failures and production issues, perform root-cause analysis, and implement permanent fixes.
Participate in L1, L2, and L3 production support, including operational monitoring, incident resolution, defect fixing, and problem management.
Maintain technical documentation, operational procedures, and data pipeline specifications.
Communicate effectively with technical teams, business stakeholders, and customer teams.
Mandatory Technical Skills
Strong hands-on experience with Azure Databricks
Strong knowledge of Apache Spark / PySpark
Advanced SQL skills
Experience with Azure Data Factory (ADF)
Experience building ETL/ELT and data ingestion pipelines
Experience with ADLS Gen2 / Azure Storage
Strong understanding of Delta Lake / Delta tables
Experience developing reusable/configuration-driven data engineering frameworks
Knowledge of batch and real-time/streaming data processing
Experience with streaming technologies such as Kafka, Azure Event Hubs, or Spark Structured Streaming
Good understanding of data warehousing and data lake concepts
Experience implementing data-quality, reconciliation, logging, auditing, and exception-handling mechanisms
Experience troubleshooting production data pipelines
Experience with L1/L2/L3 application or data-platform support
Good to Have
Databricks Workflows / Jobs
Delta Live Tables / Lakeflow Declarative Pipelines
Databricks Unity Catalog
CI/CD experience with Azure DevOps or Git
Python
Azure Key Vault and secrets management
REST API integration
Experience With JSON, CSV, Parquet, And Other Data Formats
Data migration or modernization experience
Understanding of Medallion Architecture: Bronze, Silver, Gold
Monitoring, alerting, and operational support experience
Soft Skills
Strong verbal and written communication skills
Strong analytical and problem-solving abilities
Ability to independently troubleshoot complex data issues
Comfortable working with distributed and cross-functional project teams
Ability to communicate technical issues clearly to both technical and non-technical stakeholders
Strong ownership mindset for production systems
More about this job
Responsibilities
Design, build, and optimize reusable Azure data ingestion and processing pipelines for batch and streaming workloads, using Databricks, PySpark, SQL, Azure Data Factory, and Azure storage services. Maintain data quality and reliability, troubleshoot production issues, provide L1/L2/L3 support, and collaborate with technical teams and stakeholders.
Requirements
Requires 5+ years of experience, strong hands-on Azure Databricks and Apache Spark/PySpark skills, advanced SQL, and experience with Azure Data Factory, ADLS Gen2, Delta Lake, and ETL/ELT pipelines. Candidates should also have experience with batch and streaming processing, data-quality controls, production troubleshooting and support, and reusable data engineering frameworks.
Skills
- Azure Databricks
- PySpark
- Apache Spark
- SQL
- Azure Data Factory
- ETL/ELT
- Data Ingestion
- ADLS Gen2
- Delta Lake
- Streaming Data Processing
- Apache Kafka
- Azure Event Hubs
- Data Quality
- Production Support
- Python
- Azure DevOps
Visa sponsorship
Not detected in the job text
Categories
- Data & Analytics
- Technology
- Software
Keywords
- Azure Databricks
- PySpark
- Apache Spark
- Spark SQL
- SQL
- Azure Data Factory
- ADLS Gen2
- Azure Storage
- Delta Lake
- ETL/ELT
- Data Ingestion
- Data Engineering
- Batch Processing
- Streaming Data Processing
- Apache Kafka
- Azure Event Hubs
- Spark Structured Streaming
- Data Warehousing
- Data Lakes
- Data Quality
- Data Reconciliation
- Audit Logging
- Exception Handling
- Databricks Workflows
- Delta Live Tables
- Unity Catalog
- Azure DevOps
- Python
- Azure Key Vault
- Medallion Architecture
