Market Data Engineering

A tape replay that “looks fine” can still fail your strategy: one venue switch, one vendor relabeling, or one licensing clause can silently change timestamps, missing trades, or allowed redistribution. For tick data engineering, that means your pipeline must treat data sources (exchanges, vendors, APIs) and licensing (redistribution, retention, audit) as first-class system components - not afterthoughts.

After this chapter, you will be able to (1) choose acquisition paths that match your latency and completeness needs, (2) verify API and schema guarantees before you store anything, and (3) translate licensing terms into concrete pipeline rules for storage and downstream use.

Why This Matters: Choosing sources and licensing prevents silent data drift

Tick data reliability depends on three contracts you must align: the exchange’s feed semantics, the vendor’s delivery and normalization, and your license’s allowed usage. If you ingest trades from multiple venues, you also inherit differences in ordering, symbol mapping, and timestamp representation (exchange time vs. arrival time). These differences show up as mismatched book events and incorrect microstructure features.

This chapter solves a specific failure mode: “works in backtests, breaks in production.” You often see this when the API changes fields, the vendor remaps instruments, or the license restricts redistribution to specific systems or time windows. You can prevent it by selecting sources with explicit guarantees and by building ingestion checks tied to those guarantees.

Use the Source Fit Matrix to keep decisions concrete. It scores each option by (a) data semantics match, (b) completeness controls, (c) access method stability, and (d) licensing fit.

Source Fit Matrix (quick reference)

How It Works: Select exchanges, vendors, APIs, and enforce licensing

You choose inputs by mapping your strategy needs to data semantics and then validating both API behavior and license constraints. Use these steps:

1. Define required semantics before you pick a feed

• Specify: trade events, quote updates, depth levels, and timestamp basis (exchange time vs. ingestion time).
• Example: if your features depend on book update order, require a documented ordering rule and venue timestamps.

2. Score options with the Source Fit Matrix

• Rate each option for semantics match, completeness controls, API stability, and licensing fit.
• Output: a shortlist with a “must-have” and “acceptable” category per field.

3. Validate API/schema behavior in a staging environment

• Capture raw responses, confirm symbol mapping, and verify timestamp types and precision.
• Confirm you can detect gaps: sequence numbers, trade IDs, or heartbeat fields.

4. Translate licensing into pipeline rules

• Convert terms into engineering constraints: retention duration, redistribution boundaries, audit logging, and permitted environments.
• Store license metadata alongside data lineage so you can prove compliance during audits.

Minimum validation outputs (per source)
- Instrument mapping table (vendor symbol -> internal instrument_id)
- Timestamp field inventory (names, types, precision, basis)
- Gap detection method (sequence/trade_id/heartbeat)
- Schema versioning behavior (does it break clients or add fields?)
- License constraints (retention, redistribution, audit requirements)

Real-world scenario (Nadia, 34, quant analyst at a prop shop)

Nadia needs tick data for one strategy that uses quote update ordering and one strategy that only uses trade prints. She builds ingestion for two venues and uses one vendor API for both.

1. She writes a requirements sheet:

• Strategy A: quote update events, ordering guarantee, exchange timestamps.
• Strategy B: trade prints only, exchange timestamps acceptable or ingestion timestamps with documented basis.

2. She evaluates three options with the Source Fit Matrix:

• Direct exchange for Strategy A, vendor API for Strategy B.

3. She tests the vendor API in staging for 24 hours:

• She records schema fields and checks for instrument remaps.
• She verifies gap detection triggers when sequence numbers jump or heartbeat stops.

4....