Data Engineer (Databricks & DevOps)

We are looking for a Data Engineer to design, build, and maintain scalable data pipelines on AWS and Databricks. You will combine data engineering with DevOps practices (CI/CD, infrastructure as code, monitoring) to deliver reliable, automated, and secure data platforms that support analytics, reporting, and machine learning. Key Responsibilities • Design and develop batch and streaming data pipelines using Databricks (PySpark, Spark SQL, Delta Lake) on AWS • Build and optimize ETL/ELT workflows to ingest data from databases, APIs, files, and event streams • Implement lakehouse architecture using Delta Lake, Unity Catalog, and the medallion model (bronze/silver/gold) • Manage and orchestrate workflows with Databricks Workflows, Airflow, or AWS Step Functions • Provision and manage cloud infrastructure using Terraform or CloudFormation • Build and maintain CI/CD pipelines for data code and infrastructure (GitHub Actions, GitLab CI, Jenkins, or Azure DevOps) • Implement monitoring, logging, and alerting for pipelines and platform health (CloudWatch, Datadog, etc.) • Optimize Spark jobs and cluster configurations for performance and cost • Enforce data quality, governance, security, and access control standards • Collaborate with data scientists, analysts, and business stakeholders to translate requirements into technical solutions • Document pipelines, architecture, and operational runbooks Required Skills and Qualifications • Bachelor’s degree in Computer Science, Engineering, or a related field • Strong hands-on experience with Databricks (PySpark, Delta Lake, Jobs/Workflows, cluster management) • Solid experience with AWS services: S3, IAM, Glue, Lambda, EC2, Kinesis/MSK, Redshift, VPC, CloudWatch • Proficiency in Python and SQL • Experience with DevOps tools and practices: Git, CI/CD pipelines, Docker, and infrastructure as code (Terraform preferred) • Understanding of data modeling, data warehousing, and lakehouse concepts • Experience with workflow orchestration tools (Airflow, Databricks Workflows) • Familiarity with Agile/Scrum delivery Preferred Qualifications • Databricks Certified Data Engineer (Associate/Professional) • AWS certifications (Solutions Architect, Data Engineer, or DevOps Engineer) • Experience with Unity Catalog, DLT (Delta Live Tables), and Auto Loader • Knowledge of Kubernetes, Kafka, or dbt • Experience with data governance, lineage, and compliance (GDPR, PDPA, etc.) • Exposure to MLOps or supporting ML workloads • Scripting skills in Bash or PowerShell