Senior Data Engineer

Asia/Bangkok — daily overlap with Warsaw 14:00–18:00 BKK

Architect and build QTPILOT's quantitative research data infrastructure from scratch — designing scalable systems that power research pipelines, model development, and live trading monitoring. This is not a database maintenance role.

What you'll do

• •Design and build a point-in-time research data lake that records the state of market data as it was known at each historical moment, eliminating look-ahead bias in research and backtesting.
• •Build version-controlled, reproducible feature engineering pipelines. Every model output must be traceable back to the data state and transformations that produced it.
• •Handle the realities of equity market data: outliers and anomalous prints, corporate actions (splits, dividends, mergers, spin-offs), and symbol mapping / ticker changes across vendors.
• •Tune the data retrieval layer for both research and production access patterns — columnar formats, partitioning, caching, and query performance tuning.
• •Monitor production data flow and quality end-to-end: real-time data flow from exchanges, data availability checks for model predictions, and continuous data quality monitoring in production.

What we're looking for

• •Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or a related quantitative field.
• •Minimum 5 years in Data Engineering, with at least 2 years on production AWS infrastructure.
• •Track record of designing data systems from greenfield through to production.
• •Production-grade Python (not notebook-only).
• •AWS production experience: S3, Glue, Athena or EMR, Lake Formation, Lambda.
• •Workflow orchestration: Airflow, Dagster, or Prefect.
• •Time-series & columnar storage: Parquet, Arrow, ClickHouse, or equivalent.
• •Data lakehouse: Delta Lake, Iceberg, or Hudi.
• •Advanced SQL — both analytical and transactional databases.
• •Self-driven systems thinker; clear technical communicator who writes readable documentation.

Nice to have

• •Real-time streaming: Kafka, AWS Kinesis, or MSK.
• •Equity market data experience — tick data, OHLC, corporate actions, vendor reconciliation.
• •kdb+ or other specialized time-series database experience.
• •Experience inside a systematic trading team or hedge fund.
• •ML lifecycle support — feature stores, model registry, experiment tracking.