Senior Data Engineer – AWS / Databricks / PySpark

Senior Data Engineer – AWS / Databricks / PySpark Role Overview We are seeking an experienced Senior Data Engineer to design, develop and maintain scalable data pipelines and reusable data products on an AWS and Databricks-based data platform. The role involves technical leadership in data engineering, lakehouse architecture, data quality, governance and performance optimisation to support analytics, reporting and AI/ML workloads. Key Responsibilities • Design, develop and optimise scalable batch and streaming data pipelines using AWS, Databricks, Python, PySpark and SQL. • Build and maintain lakehouse architectures using Delta Lake, including Bronze, Silver and Gold data layers, curated datasets and reusable data products. • Implement data ingestion from databases, APIs, files and event streams, including incremental loading, Change Data Capture (CDC), backfills and recovery. • Establish automated data quality checks, schema validation, reconciliation, metadata management and data lineage. • Develop and automate data workflows using Databricks Workflows, Lakeflow and AWS services, incorporating CI/CD, automated testing and infrastructure as code. • Implement data governance, access controls and security measures using Unity Catalog, AWS Glue Data Catalog, Lake Formation and IAM. • Monitor pipeline performance, troubleshoot production issues and optimise Spark/SQL workloads for scalability, reliability and cost efficiency. • Lead technical design and code reviews, establish engineering standards and mentor junior engineers. • Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions. • Maintain technical documentation, data models, operational procedures and knowledge transfer materials. Requirements • Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related discipline. • 6–10 years of data/software engineering experience, including at least 2 years working with cloud-based data platforms. • Strong hands-on experience with Python, PySpark, SQL, Apache Spark, Databricks and Delta Lake. • Proficiency in AWS data services such as S3, Glue, Lake Formation, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis, IAM and KMS. • Experience designing data lake/lakehouse architectures, data models and production-grade batch or streaming pipelines. • Strong understanding of ETL/ELT, CDC, incremental ingestion, data quality, schema validation, metadata and lineage. • Experience with workflow orchestration, Git, CI/CD, automated testing and infrastructure as code, preferably Terraform. • Knowledge of data pipeline monitoring, observability, troubleshooting and performance optimisation. • Familiarity with cloud security, encryption, access management and data governance. • Proven ability to provide technical guidance, conduct code reviews and mentor engineers. Good to Have • Experience with Databricks Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles. • Familiarity with Apache Airflow, dbt, Kafka, AWS MSK/Kinesis and AWS DMS. • Knowledge of data contracts, data products, semantic layers, master/reference data and data mesh concepts. • Experience supporting ML/AI data pipelines, feature engineering, feature stores or MLOps. • Exposure to Singapore public-sector environments, including IM8, VAPT, data classification and cross-agency data governance.