Position Name – Microsoft Fabric & Airflow Engineer
Type of hiring – Subcon
Location – Brampton, ON (Onsite)
Job Description:
Responsibilities:
- Lead the offshore engineering pod, providing mentorship and guidance to team members.
- Design, build, and optimize data pipelines in Fabric Data Factory.
- Orchestrate workloads using Fabric Airflow to ensure efficient data processing.
- Manage Lakehouse and Warehouse items in OneLake, including configuration of shortcuts to various cloud storage solutions.
- Implement and maintain Direct Lake semantic model configurations and Oracle mirroring setups.
- Utilize Spark, PySpark, SQL, T SQL, and Delta Lake for data processing and transformation.
- Conduct interoperability testing with Databricks and Snowflake to ensure seamless integration.
- Utilize Azure DevOps and Git for version control and CI/CD of Fabric artifacts.
- Monitor consumption using the Fabric Capacity Metrics app and perform disciplined benchmarking.
- Document technical processes and maintain clean technical documentation.
- Coordinate across time zones with an onsite architect to ensure project alignment.
Mandatory Skills:
- Strong experience with Data Fabric technologies.
- Proficient in building and tuning pipelines in Fabric Data Factory.
- Hands on experience with Fabric Airflow for workload orchestration.
- Expertise in managing Lakehouse and Warehouse items in OneLake.
- Solid understanding of cloud storage solutions: ADLS Gen2, AWS S3, and GCP Cloud Storage.
- Proficient in Spark, PySpark, SQL, T SQL, and Delta Lake.
- Experience with Direct Lake semantic model configuration and Oracle mirroring setup.
- Familiarity with Databricks and Snowflake for interoperability testing.
- Fluency in Azure DevOps and Git for version control and CI/CD.
- Experience using the Fabric Capacity Metrics app for monitoring.
Preferred Skills:
- Experience in mentoring and leading engineering teams.
- Strong analytical and problem-solving skills.
- Excellent communication skills, both verbal and written.
