Feature stores and ML pipelines for real-time trading analytics

Machine learning in a brokerage covers a wide range: churn prediction, deposit propensity, fraud and abuse detection, risk scoring of client exposure, next-best-action for account managers and signals for trading assistants. Many of these models need fresh data. A client's behaviour over the last five minutes says more about fraud or margin risk than a nightly snapshot. A feature store is the piece of infrastructure that makes this manageable.

The hard part is rarely the model. It is producing the same features, computed the same way, for training on years of history and for scoring live events within milliseconds. This is the problem a feature store solves.

Why features break in production

Three failure modes appear again and again:

• Training–serving skew. Data scientists compute trades_last_24h in a notebook with pandas; engineers re-implement it in the live service in another language. The definitions drift, and the model sees different inputs in production than in training.
• Data leakage. Training rows include information that was not available at prediction time, such as a balance updated after the event being predicted or a label-correlated status. Offline accuracy looks excellent and production results are poor.
• Duplicated effort. Each team rebuilds the same aggregates with slightly different logic, and nobody knows which "active days" is correct.

What a feature store provides

A feature store is a combination of a registry, storage and serving:

• Feature definitions as code: entity (client, account, instrument), source, transformation, aggregation window, owner, and description, kept in version control and reviewed.
• Offline store: full history of feature values in the warehouse or lakehouse, used to build training sets.
• Online store: the latest value per entity in a low-latency key–value store (Redis, a wide-column store, or similar) for real-time inference.
• Point-in-time joins: building training data where each label row gets feature values as they were at that moment, never later.
• Materialisation: jobs (batch and streaming) that compute features and write them to both stores from one definition.

Open-source projects such as Feast and several commercial platforms implement this pattern. You can also build a lean version on top of your existing warehouse, stream processor and cache, provided the definitions are shared.

Point-in-time correctness

This is the concept that most often separates working models from misleading ones. Suppose you predict whether a client will withdraw funds within seven days of a losing week. For each training example with timestamp t, every feature must be computed only from events before t. In SQL terms this is an "as-of" join: for each label row, take the latest feature row with feature_timestamp <= label_timestamp. Many warehouses and dataframe libraries support it directly, and feature stores implement it for you.

Also record event time (when it happened) separately from processing time (when you saw it). Trading data arrives late and out of order, especially across platforms and payment providers. Late data must update the right windows without rewriting history that a model already used.

Real-time feature engineering with streaming

Real-time features usually come from an event log (Kafka, Redpanda, or a similar system) carrying orders, executions, deposits, logins and price ticks. A stream processor (Flink, Spark Structured Streaming, or a lighter framework) computes windowed aggregates:

• Number and notional volume of trades per client over 1, 5 and 60 minutes.
• Realised and unrealised P&L over the session; drawdown from intraday peak.
• Deposit and withdrawal velocity; count of distinct payment instruments in 24 hours.
• Login anomalies: new device, new country, rapid IP changes.
• Instrument-level features: volatility over recent windows, spread, order-book imbalance.

Design choices that matter:

• Window type. Sliding windows give smooth features but cost more; tumbling windows are cheap but jumpy; session windows suit user activity.
• Watermarks. Decide how long to wait for late events before closing a window, and how late events after that are handled.
• State size. Per-client state across millions of accounts must be bounded. Use approximate structures (HyperLogLog for distinct counts, t-digest for percentiles) where exactness is not needed.
• Backfill. The same transformation must run in batch over history to populate the offline store. Frameworks that run identical logic in batch and streaming mode reduce skew dramatically.

The serving path and latency budget

For a fraud check on a withdrawal or a risk alert on a position, the full path (receive event, fetch features, score, decide) might have a budget of 50–100 ms. Typical allocation: a few milliseconds to read features from the online store with a single batched lookup, 5–20 ms for model inference (gradient-boosted trees are fast and strong for tabular data), and the rest for network and business logic. Keep models and feature fetches in the same region as the trading systems, cache static attributes, and set timeouts with safe fallbacks. If features are missing, use a conservative default decision rather than blocking the client indefinitely.

The ML pipeline for trading around it

A feature store is one part of a complete lifecycle:

1. Training pipeline: build a point-in-time training set, train, and evaluate on time-based splits (train on the past, test on later periods; random splits leak future information in time series).
2. Model registry: version model artefacts with the exact feature list and versions they expect.
3. Validation gates: compare against the current model on recent data, check calibration and fairness across segments, and require sign-off for risk-relevant models.
4. Deployment: shadow scoring first, then canary, with automatic rollback.
5. Monitoring: feature drift (population stability index or similar), missing-value rates, prediction distribution, latency, and real outcome metrics once labels arrive.

Governance in a regulated setting

Trading and client-risk models influence decisions about real people and real money. Keep lineage from every prediction back to feature values and model version, so any alert can be explained. Restrict access to personal features, document intended use, and review models periodically. Where models support decisions about clients, human review and clear escalation should remain part of the process.

Practical starting point

Do not start by deploying a platform. Start with one model that needs fresh data, such as withdrawal-fraud scoring. Define ten to twenty features as code, implement them once with batch backfill and streaming updates, build point-in-time training data, and serve from a simple online store. Once a second model reuses those features, the value of the feature store becomes obvious, and the platform can grow from real needs.