Databricks Data Engineer
- Toronto, ON, Canada
Applications open in a new tab.
Job details
- Experience
- 5+ years of Databricks hands on experience
- Posting language
- English
Original job description
Who we are looking for
This is a hands-on Data Engineer position. We are looking for candidate with good knowledge on Bigdata technology and strong development experience with Databricks.
What you will be responsible for
As Data Engineer you will
- Design & Develop custom high throughput and configurable frameworks/libraries
- Architect and implement scalable data ingestion pipelines using Azure Data Factory and Databricks
- Transform raw datasets into structured data lakes (Medallion architecture: bronze, silver, gold)
- Convert Hive Metastore to Unity Catalog and configure associated permissions
- Design and secure Delta tables with appropriate schema and access controls
- Enable business intelligence through Power BI integration and dashboard-ready datasets
What we value
These skills will help you succeed in this role
- Experience performing data analysis and data exploration
- Experience working in an agile delivery environment
- Strong knowledge on Databricks SQL/Scala - Data engineering pipeline
- Strong Experience in Unix, Python and complex SQL
- Strong critical thinking, communication, and problem-solving skills
- Strong hands-on experience in troubleshooting DevOps pipelines and Azure services
- Azure cloud, Apache PySpark
Education & Preferred Qualifications
- Bachelors Degree level qualification in a computer or IT related subject
- 5+ years of Databricks hands on experience
- Data Bricks Data Engineer Professional Certification
More about this job
Responsibilities
The Databricks Data Engineer at Rubicon Path will design and build data ingestion pipelines and reusable data frameworks. They will organize raw data into structured data lakes, move permissions to Unity Catalog, and prepare datasets for business intelligence reporting.
Requirements
Candidates need at least 5 years of hands-on Databricks experience, strong SQL, Scala or Python skills, and working knowledge of Azure cloud services and Apache PySpark. A bachelor's degree in a computer or IT subject is required, and the Databricks Data Engineer Professional certification is preferred.
Skills
- Databricks
- Databricks SQL
- Scala
- PySpark
- Python
- Unix
- Complex SQL
- Azure Data Factory
- Azure
- Delta Lake
- Unity Catalog
- Hive Metastore
- Medallion architecture
- Power BI
- DevOps pipelines
- Data ingestion
- Data lake design
- Data modelling
- Schema design
- Access control
- Data analysis
- Data exploration
- Troubleshooting
- Big Data technologies
- Agile delivery
Categories
- Software
- Technology
- Data & Analytics
Keywords
- Databricks
- Data Engineer
- Azure Data Factory
- Apache PySpark
- PySpark
- Delta Lake
- Unity Catalog
- Hive Metastore
- Medallion architecture
- Power BI
- Azure
- Python
- Scala
- SQL
- Unix
- DevOps
- Big Data
- Data lake
- ETL pipelines
- Agile
Seniority
Senior
Tools
- Databricks
- Azure Data Factory
- Power BI
- Hive Metastore
- Unity Catalog
- Delta Lake
- Apache PySpark
Languages
- English
Fields of study
- Computer science
- Information technology