Closed listing
Python, PySpark_London
- London, ON, Canada
Contract
This opening has closed.
The original listing is kept below for reference.
Job details
- Job type
- Contract
- Closing date
- Posting language
- English
- Posted
Similar jobs hiring now
Search current openingsOriginal job description
Archived listing. The details below describe a past opening.
Design, develop and maintain scalable ETL pipelines using AWS glue and python, write PySpark scripts for extraction, transformation and loading, configure glue jobs, crawlers and workflows for automated data processing.
Cloud Data Engineering: Work with AWS S3 for data storage and management. Use AWS IAM for access control and security, Leverage AWS Lambda for serverless orchestration and event driven processing.
Monitor Glue jon metrics, using AWS cloud watch, Tune Spark configurations for Optimal performance.
Document ETL processes, data flows, and technical decisions
Experience with infrastructure as code (Terraform, CloudFormation)
More about this job
Responsibilities
Design, develop, and maintain scalable ETL pipelines using AWS Glue, Python, and PySpark, including configuring Glue jobs, crawlers, and workflows. Manage cloud data storage, access control, orchestration, monitoring, and Spark performance, and document data flows and technical decisions.
Requirements
The role calls for experience with Python, PySpark, AWS Glue, and cloud data engineering services including S3, IAM, Lambda, and CloudWatch. Experience with infrastructure-as-code tools such as Terraform or CloudFormation is also specified.
Skills
- Python
- PySpark
- ETL Pipeline Development
- AWS Glue
- AWS S3
- AWS IAM
- AWS Lambda
- AWS CloudWatch
- Apache Spark Performance Tuning
- Data Engineering
- Infrastructure as Code
- Terraform
- AWS CloudFormation
- Data Pipeline Documentation
Visa sponsorship
Not detected in the job text
Categories
- Technology
- Data & Analytics
- Software
- Consulting
Keywords
- Python
- PySpark
- ETL
- AWS Glue
- AWS S3
- AWS IAM
- AWS Lambda
- AWS CloudWatch
- Apache Spark
- Cloud Data Engineering
- Data Storage
- Access Control
- Security
- Serverless Orchestration
- Event-Driven Processing
- Glue Jobs
- Glue Crawlers
- Glue Workflows
- Data Extraction
- Data Transformation
- Data Loading
- Spark Configuration
- Performance Tuning
- Infrastructure as Code
- Terraform
- AWS CloudFormation
- Data Flows
- Automated Data Processing