We are looking for a Senior Software Engineer / Big Data Engineer with strong hands-on experience in Apache Spark, Spark SQL, Python, and Scala.
🔹 Key Responsibilities
- Develop scalable data processing applications using Apache Spark & Spark SQL
- Build and optimize ETL/data pipelines using Python and Scala
- Work with large datasets using Hive, Parquet, and HDFS
- Optimize Spark jobs, queries, partitioning, and execution plans
- Develop batch and near-real-time data processing solutions
- Troubleshoot performance and data processing issues
- Implement testing, monitoring, and deployment processes
- Collaborate with Data Engineers, Software Engineers, and Data Scientists
🔹 Required Skills
- 5+ years of Software/Data Engineering experience
- Strong experience with Apache Spark, Spark SQL, Python, and Scala
- Hands-on experience with Hive and Parquet
- Strong SQL and query optimization skills
- Experience with distributed data processing and ETL
- Linux/Shell scripting and Git
🔹 Nice to Have
- AWS
- Airflow
- Data Lake/Lakehouse
- Iceberg, ORC, Avro
- Docker & Kubernetes
