1. Design and Development of Data Pipelines
- Design, build, and optimize robust ETL/ELT pipelines using AWS services (S3, Glue, Lambda) and the Databricks platform (Spark, Delta Lake, Delta Live Tables).
- Ingest and process large volumes of structured and semi-structured data from diverse sources (APIs, databases, and streaming platforms like Kafka or Kinesis) into a centralized data lake or lakehouse.
2. Data Transformation and Modeling
- Develop and maintain data models (such as star/snowflake schemas and medallion architecture) optimized for analytics and BI tools using dbt (Data Build Tool).
- Write complex, efficient SQL queries and Python/PySpark code for data manipulation, transformation, and validation within the Databricks environment.
- Implement data quality checks, automated tests, and documentation as part of the dbt workflow while enforcing data governance and security standards.
3. Orchestration and Automation
- Orchestrate and monitor data workflows using Databricks Jobs or external orchestration tools such as AWS MWAA (Managed Workflows for Apache Airflow).
- Implement CI/CD pipelines and version control (Git) across all data engineering artifacts (code, configurations, dbt models) to ensure reliable and consistent deployments.
4. Performance Optimization and Operations
- Monitor, troubleshoot, and resolve production data pipeline issues to maintain high performance, reliability, and cost-efficiency.
- Tune Spark jobs and optimize Delta Lake features (Z-Ordering, partitioning) to scale with growing data volumes and complexity.
5. Collaboration and Support
- Partner with data scientists, analysts, and business stakeholders to gather data requirements and deliver actionable insights.
- Provide technical guidance on data engineering best practices, fostering a strong culture of data quality and governance.
Must-Have Skills
- Cloud & Data Platforms: AWS (Redshift, S3, Glue, Lambda), Databricks on AWS
- Transformation & Databases: dbt Core and dbt Cloud, SQL, SQL Server DB
- Scheduling & DevOps: Stonebranch scheduling tool, CI/CD pipelines, Git
- Methodology: Hands-on experience working in an Agile environment using JIRA
Good-to-Have Skills
- Experience with Tableau
- Exposure to Harness DevOps
- Proficiency in Linux / Unix environments