Databricks Architect
- Canada
Contract
Applications open in a new tab.
Job details
- Job type
- Contract
- Posting language
- English
- Posted
Original job description
We are seeking an experienced Databricks Architect to design, develop, and implement scalable, secure, and high-performance data solutions using the Databricks Lakehouse Platform. The ideal candidate will have strong expertise in data architecture, cloud data engineering, ETL/ELT pipelines, data modeling, and big data technologies. This role will involve collaborating with business stakeholders, data engineers, analytics teams, and cloud architects to deliver enterprise-grade data platforms.
Key Responsibilities
- Support data migration, platform modernization, production troubleshooting, and disaster recovery planning.
- Collaborate with data engineers, data scientists, business stakeholders, and cloud teams to establish architecture standards and best practices.
- Integrate Databricks with enterprise applications, cloud storage, data warehouses, APIs, and BI tools.
- Implement workflow orchestration, CI/CD pipelines, automated testing, and deployment strategies.
- Optimize Spark jobs, SQL queries, cluster configurations, and compute resources for performance and cost efficiency.
- Build batch and real-time data processing pipelines using Spark Structured Streaming and Apache Kafka.
- Design and implement data governance, security, access controls, and lineage using Unity Catalog.
- Implement Delta Lake and Medallion Architecture (Bronze, Silver, and Gold layers) for data processing and quality management.
- Develop and optimize ETL/ELT pipelines using Apache Spark, PySpark, Python, and SQL.
- Define enterprise data architecture, including data ingestion, transformation, storage, processing, and consumption layers.
- Design and implement scalable Databricks Lakehouse architectures on Azure, AWS, or Google Cloud Platform (GCP).
More about this job
Responsibilities
Design and implement scalable Databricks Lakehouse architectures and enterprise data platforms, including ingestion, transformation, storage, processing, and consumption layers. Build and optimize batch and streaming pipelines, integrate enterprise systems, and establish governance, security, deployment, performance, and disaster recovery practices.
Requirements
The role calls for strong expertise in data architecture, cloud data engineering, ETL/ELT pipelines, data modeling, and big data technologies. It also requires experience with Databricks, Spark, Python, SQL, cloud platforms, data governance, and collaboration across engineering, analytics, business, and cloud teams.
Skills
- Databricks
- Data Architecture
- Cloud Data Engineering
- ETL/ELT
- Data Modeling
- Apache Spark
- PySpark
- Python
- SQL
- Spark Structured Streaming
- Apache Kafka
- Delta Lake
- Medallion Architecture
- Unity Catalog
- Workflow Orchestration
- CI/CD
Visa sponsorship
Not detected in the job text
Categories
- Technology
- Data & Analytics
- Software
- Engineering
Keywords
- Databricks Lakehouse Platform
- Enterprise Data Architecture
- Cloud Data Engineering
- Azure
- AWS
- Google Cloud Platform
- Data Ingestion
- Data Transformation
- Data Storage
- Data Processing
- Data Consumption
- ETL/ELT Pipelines
- Apache Spark
- PySpark
- Python
- SQL
- Spark Structured Streaming
- Apache Kafka
- Delta Lake
- Medallion Architecture
- Unity Catalog
- Data Governance
- Data Security
- Access Controls
- Data Lineage
- Workflow Orchestration
- CI/CD Pipelines
- Automated Testing
- Cluster Optimization
- Disaster Recovery