Job Title : Data Engineer
Location: Halifax, Canada
Duration Contract
Skill Cluster : Data-Cloudera with AWS Native Services, Spark, Python/Scala/Java, Kafka, HDFS, Hive.
Primary Skill : SQL, PySpark, Data analytics
Experience : 10+ years
Job Description :
Job Requirements: Senior Data Engineer
Position Summary:
- The Senior Data Engineer leads complex data engineering projects, working on designing data architectures that align with business requirements.
- This role focuses on optimizing data workflows, managing data pipelines, and ensuring the smooth operation of data systems.
Minimum Qualifications:
- 8+ Years overall IT experience with minimum 5 years of work experience in below techskills
Tech Skills:
- Strong experience in Python Scripting and PySpark for data processing? Proficiency in SQL, dealing with big data processing
- Experience in AWS cloud platform and its data services (S3, Redshift, Lambda,EMR, Airflow, Glue)
- Expertise in BASH/Shell scripting
- Strong understanding of healthcare insurance domain and experience leading data engineering teams
- Experience in Agile environments.
- Excellent problem-solving skills and attention to detail.
- Effective communication and collaboration skills.
Responsibilities:
- Leads development of data pipelines and architectures that handle large-scale data sets.
- Designs, constructs, and tests data architecture aligned with business requirements.
- Provides technical leadership for data projects, ensuring best practices and high-quality data solutions.
- Collaborates with product, finance, and other business units to ensure data pipelines meet business requirements.
- Oversees development of data solutions that enable predictive and prescriptive analytics.
- Ensures the technical quality of solutions, managing data as it moves across environments.
- Aligns data architecture to Healthfirst solution architecture.