Key Responsibilities
- Design and develop scalable data pipelines using Azure services.
- Build and maintain ETL/ELT workflows for structured and unstructured data.
- Develop data integration solutions using Azure Data Factory (ADF).
- Store and process data using Azure Data Lake Storage (ADLS).
- Develop and optimize data transformations using Azure Databricks and PySpark.
- Design and maintain data warehouses using Azure Synapse Analytics.
- Write optimized SQL queries, stored procedures, and data transformations.
- Perform data cleansing, validation, and quality checks.
- Integrate data from multiple sources such as databases, APIs, files, and cloud applications.
- Monitor pipeline performance and troubleshoot data-processing failures.
- Implement data security, access control, and governance practices.
- Optimize data pipelines and cloud resources for performance and cost.
- Collaborate with data analysts, data scientists, software engineers, and business stakeholders.
- Maintain technical documentation for data pipelines, data models, and architecture.
Required Technical Skills
Azure
- Azure Data Factory (ADF)
- Azure Data Lake Storage Gen2
- Azure Databricks
- Azure Synapse Analytics
- Azure Key Vault
- Azure Monitor
- Azure Storage
- Azure SQL Database
Programming & Data
- Strong SQL
- Python
- PySpark
- ETL/ELT concepts
- Data transformation and data cleansing
- Data modeling
- Data warehousing concepts
Database Technologies
- SQL Server
- Azure SQL
- PostgreSQL/MySQL
- Relational and dimensional databases
Data Engineering Concepts
- ETL/ELT pipeline development
- Batch and incremental data processing
- Star and snowflake schemas
- Fact and dimension tables
- Slowly Changing Dimensions (SCD)
- Data partitioning
- Data quality and validation
- Performance optimization
- Data lake and data warehouse architecture
DevOps & Version Control
- Git/GitHub/Azure Repos
- CI/CD pipelines
- Azure DevOps
- Infrastructure and deployment automation
- Environment management across development, testing, and production
Preferred Skills
- Microsoft Fabric
- Azure Event Hubs
- Azure Stream Analytics
- Azure Functions
- Power BI integration
- Delta Lake
- Apache Spark
- Kafka
- Terraform
- Cloud security and governance
- Microsoft Purview
