We are seeking a highly skilled Senior/Lead Azure Cloud Data Engineer to design, develop, and optimize large-scale cloud-based data solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure Data Factory (ADF), Azure Databricks, PySpark, and Autosys, with hands-on experience building scalable data pipelines, data transformation frameworks, and enterprise-grade data platforms.
The candidate will collaborate with business stakeholders, architects, and development teams to deliver high-performance data solutions that support analytics, reporting, regulatory, and operational requirements.
Key Responsibilities
- Design, develop, and maintain scalable data ingestion and transformation pipelines using Azure Data Factory (ADF).
- Build and optimize data processing solutions using Azure Databricks and PySpark.
- Develop and maintain batch and near real-time data integration workflows.
- Implement data quality, validation, reconciliation, and monitoring frameworks.
- Schedule, monitor, and troubleshoot jobs using Autosys and cloud-native orchestration tools.
- Design and implement robust ETL/ELT solutions leveraging Azure cloud services.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Collaborate with Data Architects and Solution Architects to define enterprise data models and standards.
- Support production deployments, incident management, and performance tuning activities.
- Ensure adherence to security, governance, and compliance standards.
- Mentor junior engineers and provide technical leadership for project delivery.
- Participate in code reviews, architecture discussions, and best practice implementation.
Required Qualifications
- Bachelor’s or master’s degree in computer science, Information Technology, Engineering, or related field.
- 10+ years of experience in Data Engineering and Data Warehousing.
- 5+ years of experience with Azure Data Platform technologies.
- Strong hands-on experience with:
- Azure Data Factory (ADF)
- Azure Databricks
- ADLS
- MYSQL
- PySpark
- SQL
- Autosys
- Experience designing and implementing ETL/ELT frameworks.
- Strong understanding of distributed computing and Spark architecture.
- Experience working with large-scale structured and semi-structured datasets.
- Proficiency in Python and SQL programming.
- Experience with source control tools such as Git/Azure DevOps.
