S
Sunilkumar Reddy Eraganeni
Guest
A few months ago I watched a fraud model flag a transaction correctly. The only problem was that it flagged it four minutes after the money had already left the account. The model was fine. The math was fine. The data pipeline was the thing that failed, quietly, the way pipelines usually do.
That incident stuck with me because it captures something a lot of AI teams still get wrong. We spend enormous energy tuning models, and comparatively little energy asking whether the data even arrives in time for the model's answer to matter. An "intelligent" system fed on stale data isn't intelligent. It's just slow and confident, which might be worse.
I want to push back gently on how loosely the phrase "real-time" gets thrown around in vendor decks. Real-time isn't a fixed number. It's whatever window the decision needs. A recommendation engine might tolerate a few seconds of staleness without anyone noticing. A trading system or a fraud check has a window measured in milliseconds, sometimes under 100ms end to end. An autonomous vehicle's perception stack doesn't get to negotiate at all.
So the first design question isn't "how do we make this fast," it's "how fast does this specific decision need to be, and what breaks if we miss that budget." That framing changes everything downstream, including whether you need streaming infrastructure at all. Plenty of teams reach for Kafka when a five-minute micro-batch on Snowflake or Databricks would have done the job with a fraction of the operational overhead. I've built both, and the streaming version is not automatically the better one.
Once you do need genuine low latency, the architecture tends to converge on a familiar shape, even if the specific tools change:
Event producers generate raw signals: a card swipe, a sensor reading, a clickstream event, an API call. These land on a streaming backbone (Kafka, Kinesis, or Event Hubs, depending on your cloud) which decouples producers from consumers and gives you replay if something downstream falls over.
From there, a stream processing layer, Flink or Spark Structured Streaming on Databricks, does the actual work: windowing, joins, enrichment, aggregation. This is where a lot of subtle bugs live, because event time and processing time are not the same thing, and late-arriving events will quietly corrupt your aggregates if you don't handle watermarks properly. I learned that one the hard way, staring at a metric that looked "off" for two days before realizing the join window was too tight.
Processed features flow into a feature store, something like Feast, which solves a problem people underestimate: point-in-time correctness. Training data has to reflect exactly what the model would have seen at inference time, not a leaked future value. Get this wrong and your offline accuracy numbers become fiction.
Then comes model serving, usually a low-latency endpoint, and finally a decision or action layer that actually does something: blocks a transaction, reroutes a shipment, adjusts a price. A feedback loop feeds outcomes back into the system so the model keeps learning from what actually happened, not just what it predicted.
People assume the model is the bottleneck. In my experience it rarely is. A well-optimized XGBoost or lightweight neural net can score a request in single-digit milliseconds. The time disappears elsewhere: network hops, serialization, a feature lookup that hits a cold cache, an orchestration layer adding retries you didn't ask for.
If you're building anything with a genuinely tight window, profile the whole path, not just the model. Ninety milliseconds of your hundred-millisecond budget can vanish before inference even starts.
None of this is free, and I'd be lying if I said otherwise. Exactly-once processing semantics are genuinely hard and most "solutions" are exactly-once with an asterisk. Schema evolution across a streaming pipeline with dozens of downstream consumers is a coordination problem as much as a technical one. And the operational cost of running Kafka clusters, stream processors, and a feature store is real money and real on-call burden, not a one-time setup fee.
There's also a quieter risk: autonomous decision systems fail fast and at scale. A batch job that misbehaves gets caught in a nightly review. A streaming system that misbehaves can make ten thousand wrong decisions before anyone notices, because nobody is in the loop by design.
So build the event-driven architecture when the decision genuinely needs it. Just go in with eyes open about what you're trading for that speed, and keep a human somewhere in the monitoring path even if not in the decision path itself.
That incident stuck with me because it captures something a lot of AI teams still get wrong. We spend enormous energy tuning models, and comparatively little energy asking whether the data even arrives in time for the model's answer to matter. An "intelligent" system fed on stale data isn't intelligent. It's just slow and confident, which might be worse.
What "real-time" actually means here
I want to push back gently on how loosely the phrase "real-time" gets thrown around in vendor decks. Real-time isn't a fixed number. It's whatever window the decision needs. A recommendation engine might tolerate a few seconds of staleness without anyone noticing. A trading system or a fraud check has a window measured in milliseconds, sometimes under 100ms end to end. An autonomous vehicle's perception stack doesn't get to negotiate at all.
So the first design question isn't "how do we make this fast," it's "how fast does this specific decision need to be, and what breaks if we miss that budget." That framing changes everything downstream, including whether you need streaming infrastructure at all. Plenty of teams reach for Kafka when a five-minute micro-batch on Snowflake or Databricks would have done the job with a fraction of the operational overhead. I've built both, and the streaming version is not automatically the better one.
The shape of an event-driven decision system
Once you do need genuine low latency, the architecture tends to converge on a familiar shape, even if the specific tools change:
Event producers generate raw signals: a card swipe, a sensor reading, a clickstream event, an API call. These land on a streaming backbone (Kafka, Kinesis, or Event Hubs, depending on your cloud) which decouples producers from consumers and gives you replay if something downstream falls over.
From there, a stream processing layer, Flink or Spark Structured Streaming on Databricks, does the actual work: windowing, joins, enrichment, aggregation. This is where a lot of subtle bugs live, because event time and processing time are not the same thing, and late-arriving events will quietly corrupt your aggregates if you don't handle watermarks properly. I learned that one the hard way, staring at a metric that looked "off" for two days before realizing the join window was too tight.
Processed features flow into a feature store, something like Feast, which solves a problem people underestimate: point-in-time correctness. Training data has to reflect exactly what the model would have seen at inference time, not a leaked future value. Get this wrong and your offline accuracy numbers become fiction.
Then comes model serving, usually a low-latency endpoint, and finally a decision or action layer that actually does something: blocks a transaction, reroutes a shipment, adjusts a price. A feedback loop feeds outcomes back into the system so the model keeps learning from what actually happened, not just what it predicted.
Where the latency budget actually goes
People assume the model is the bottleneck. In my experience it rarely is. A well-optimized XGBoost or lightweight neural net can score a request in single-digit milliseconds. The time disappears elsewhere: network hops, serialization, a feature lookup that hits a cold cache, an orchestration layer adding retries you didn't ask for.
If you're building anything with a genuinely tight window, profile the whole path, not just the model. Ninety milliseconds of your hundred-millisecond budget can vanish before inference even starts.
The honest caveats
None of this is free, and I'd be lying if I said otherwise. Exactly-once processing semantics are genuinely hard and most "solutions" are exactly-once with an asterisk. Schema evolution across a streaming pipeline with dozens of downstream consumers is a coordination problem as much as a technical one. And the operational cost of running Kafka clusters, stream processors, and a feature store is real money and real on-call burden, not a one-time setup fee.
There's also a quieter risk: autonomous decision systems fail fast and at scale. A batch job that misbehaves gets caught in a nightly review. A streaming system that misbehaves can make ten thousand wrong decisions before anyone notices, because nobody is in the loop by design.
So build the event-driven architecture when the decision genuinely needs it. Just go in with eyes open about what you're trading for that speed, and keep a human somewhere in the monitoring path even if not in the decision path itself.