Required Skills & Qualifications
- Degree or diploma in Computer Science, Software Engineering, Information Technology, or a related field.
- Hands-on experience developing data engineering solutions using Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), and Azure Databricks.
- Strong programming skills in Python, SQL, PySpark, and Apache Spark for data processing and transformation.
- Experience developing and maintaining ETL/ELT pipelines, data ingestion frameworks, and scalable batch or near real-time data processing solutions.
- Hands-on knowledge of Delta Lake, Lakehouse Architecture, and Medallion Architecture (Bronze, Silver, Gold).
- Experience developing Databricks notebooks, workflows, data processing jobs, and optimizing Delta Lake tables.
- Strong understanding of data validation, data quality, reconciliation, monitoring, and troubleshooting of data pipelines.
- Experience with Azure DevOps, Git, CI/CD pipelines, automated deployments, and version control.
- Knowledge of REST APIs, data integration patterns, automation frameworks, and reusable development utilities.
- Understanding of data governance, metadata management, data cataloguing, security, privacy, and access control.
- Experience with software development best practices, including code reviews, automated testing, documentation, and secure coding.
- Strong communication and collaboration skills, with experience working in Agile/Scrum teams alongside architects, data engineers, analysts, and business stakeholders.
