Data Quality Engineer, MPID Team

Job Description: • Design, build, and maintain layered Hive/Spark SQL data models across ODS, DWD, DWS and ADS, including shared dimension tables. Define each table’s grain, keys, business-date and partition logic, metric definitions, dependencies, and backfill and release procedures. • Develop and operate production batch ETL pipelines on Spark and Hive, including partner and source onboarding, incremental loads, scheduled runs, and historical backfills. Use staging and checkpoint patterns where needed so retries do not corrupt published data or SCD2 history. • Plan and execute data-store or platform migrations, including Hive-to-ClickHouse migrations where required. Design partition strategies, write migration scripts, reconcile source and target data, and validate query behavior across regions. • Build reusable cross-region models and parameterized pipelines for onboarding markets, partners, and business lines. Standardize schemas and metric logic to reduce one-off implementations. • Build and maintain lightweight internal tools for data operations, such as task scaffolding, logging and monitoring, and query or configuration interfaces. Automate repetitive metadata discovery and data-quality checks, including through AI-agent or LLM-assisted workflows where appropriate. • Maintain and optimize data ingestion pipelines by designing solutions for real-time updates, batch processing, and initial full loads to meet different data-source and business requirements. • Develop and maintain a Metric Insight system that meets business and BI needs by standardizing metrics, building dashboards, and integrating AI-powered analytics tools to provide a one-stop data analysis platform. Requirements: • Master’s degree in Computer Science, Software Engineering, Artificial Intelligence from a reputable university. • Hands-on experience building and supporting production batch data pipelines with Spark/Spark SQL and Hive; familiarity with HDFS and Hive Metastore. • Practical data-warehouse experience, including dimensional or SCD2 modeling, ETL design, partitioning, and backfill or recovery. • Strong fundamentals in data structures, algorithms, operating systems, and relational databases. • Strong SQL and proficiency in Python or Java; able to write complex transformations involving nested JSON, window functions, and query tuning. • Able to diagnose Spark performance issues using the Spark UI and execution plans, including joins, shuffle, skew, and resource settings; working knowledge of Linux and Git. • Strong ownership and communication: can trace data issues end to end, validate a repair, and explain findings and data limitations to partner teams. Kindly note that you can only be considered for one role at a time with any of the companies within our Group. If you have applied for other jobs with the Group, you will be considered for roles in the order of your application