Market Data Engineering

This chapter solves a specific failure mode: “works in backtests, breaks in production.” You often see this when the API changes fields, the vendor remaps instruments, or the license restricts redistribution to specific systems or time windows. You can prevent it by selecting sources with explicit guarantees and by building ingestion checks tied to those guarantees.

Use the Source Fit Matrix to keep decisions concrete. It scores each option by (a) data semantics match, (b) completeness controls, (c) access method stability, and (d) licensing fit.

Source Fit Matrix (quick reference)

| Option type | Typical strengths | Typical risks | Best fit when | |---|---|---|---| | Direct exchange feed | Strong semantics, explicit venue identifiers | Hard operations, multiple connections | You need highest fidelity and can manage ops | | Market data vendor API | Normalized schema, bundling | Field changes, venue coverage gaps | You need speed-to-integration and consistent format | | Broker gateway / API | Easy access for trading + some market data | Limited tick depth, broker-specific rules | You want one system with acceptable granularity | | Custom aggregator | Tailored coverage | Unclear lineage, harder audits | You require cross-venue fusion with strict controls |

How It Works: Select exchanges, vendors, APIs, and enforce licensing You choose inputs by mapping your strategy needs to data semantics and then validating both API behavior and license constraints. Use these steps:

1. Define required semantics before you pick a feed - Specify: trade events, quote updates, depth levels, and timestamp basis (exchange time vs. ingestion time). - Example: if your features depend on book update order, require a documented ordering rule and venue timestamps.

2. Score options with the Source Fit Matrix - Rate each option for semantics match, completeness controls, API stability, and licensing fit. - Output: a shortlist with a “must-have” and “acceptable” category per field.

3. Validate API/schema behavior in a staging environment - Capture raw responses, confirm symbol mapping, and verify timestamp types and precision. - Confirm you can detect gaps: sequence numbers, trade IDs, or heartbeat fields.

4. Translate licensing into pipeline rules - Convert terms into engineering constraints: retention duration, redistribution boundaries, audit logging, and permitted environments. - Store license metadata alongside data lineage so you can prove compliance during audits.

text Minimum validation outputs (per source) - Instrument mapping table (vendor symbol -> internal instrument_id) - Timestamp field inventory (names, types, precision, basis) - Gap detection method (sequence/trade_id/heartbeat) - Schema versioning behavior (does it break clients or add fields?) - License constraints (retention, redistribution, audit requirements)

Real-world scenario (Nadia, 34, quant analyst at a prop shop) Nadia needs tick data for one strategy that uses quote update ordering and one strategy that only uses trade prints. She builds ingestion for two venues and uses one vendor API for both.

1. She writes a requirements sheet: - Strategy A: quote update events, ordering guarantee, exchange timestamps. - Strategy B: trade prints only, exchange timestamps acceptable or ingestion timestamps with documented basis. 2. She evaluates three options with the Source Fit Matrix: - Direct exchange for Strategy A, vendor API for Strategy B. 3. She tests the vendor API in staging for 24 hours: - She records schema fields and checks for instrument remaps. - She verifies gap detection triggers when sequence numbers jump or heartbeat stops. 4. She enforces license rules: - She configures retention to the allowed window and prevents exporting raw ticks to external research systems if the license forbids redistribution. 5. Expected outcomes: - Her ingestion either produces consistent normalized events or fails fast with a “schema/lineage/license mismatch” error before data hits production storage.

Quick checklist - Confirm timestamp basis and ordering rules per event type. - Verify instrument mapping stability and document remap behavior. - Implement gap detection using available identifiers (sequence/trade_id/heartbeat). - Store license terms as machine-readable metadata and enforce at retention/export time. - Run schema-diff checks on every deployment.

What to Watch For: Common mistakes and edge cases Unstable symbol mapping Vendors and brokers sometimes remap instrument identifiers after corporate actions or corrections. Do this: Maintain an internal instrument_id and build a vendor-to-internal mapping table with version history; reject data when mapping changes without a recorded event. Not this: Assume vendor symbols remain constant and overwrite mappings silently.

Timestamp basis confusion APIs often provide both exchange time and receipt/arrival time, sometimes under similar field names. Do this: Require explicit field inventory and basis tags; route Strategy A to exchange-time feeds only. Not this: Use a single “timestamp” column everywhere and let it represent different bases across sources.

License drift across systems Teams often store raw ticks in one place but export derived datasets (or raw extracts) to other environments that the license does not cover. Do this: Enforce export/egress rules in the data platform (retention window, allowed consumers, audit logs) and attach license metadata to each dataset. Not this: R

Not this: Rely on a human review at export time; you will miss edge cases like backfills and replays.

Source Fit Matrix quick reference Use this table during vendor/exchange selection to avoid mismatches between what your strategy needs and what the feed can guarantee.

| Feed attribute | Exchange direct (typical) | Vendor API (typical) | Source Fit Matrix decision | |---|---|---|---| | Quote update ordering | Usually stronger; depends on venue feed | Varies; may reorder during normalization | Choose exchange direct for ordering-sensitive logic | | Trade print identity | Trade ID may exist; may differ by venue | Often present but may map to internal IDs | Choose vendor only if trade identity + ordering meet your checks | | Timestamp basis | Exchange time, sometimes with multiple fields | Exchange time + receipt/processing time | Split pipelines by event type and timestamp basis | | Schema stability | More controlled but still versioned | Normalization layer can add/remove fields | Require schema diff tests in CI before deploy | | License controls | Contract-bound; varies by contract | Contract-bound; often stricter around redistribution | Attach license metadata and enforce at storage/export |

`yaml

Example: license metadata you should store per dataset dataset: name: ticks.normalized license: retention_days: 30 allowed_use: "internal trading research" allowed_export_targets: ["prod-warehouse-only"] audit_required: true lineage: sources: ["venue_X_direct", "vendor_Y_api"] `

Final selection rule: pick the feed that matches your strategy’s invariants (ordering, identity, timestamp basis) and then lock licensing and schema behavior into the pipeline so production data cannot drift silently. That discipline pays off when you start cleaning ticks and need consistent, explainable raw inputs.

End of chapter one. 7 more chapters in the full book.