realtime-anomaly-detection: Streaming Detection and Alerting

The problem

Anomalies in streaming data need catching in near real time, and simulated benchmarks lie.

Approach

Redpanda ingestion with Faust stream processing over rolling 1-minute, 5-minute and 1-hour windows. Three detectors with hot-reload: Isolation Forest, LSTM autoencoder and a Z-score baseline, plus severity scoring, alert deduplication and historical replay for backtesting. Validated against a real Kaggle fraud dataset after simulated data returned implausible results.

How it works

• Redpanda ingests events and Faust processes them over rolling 1-minute, 5-minute and 1-hour windows.
• Detectors are routed per source type: Isolation Forest and Z-score for fraud, Z-score with EWMA for server metrics, an LSTM autoencoder for IoT sequences.
• Z-score tracks a running mean and standard deviation per feature, so it needs no training run and suits high-volume metrics.
• The LSTM autoencoder reconstructs a sequence and flags spikes in reconstruction error, catching temporal anomalies a point detector cannot see.
• Severity scoring and deduplication sit between detection and alerting, so one incident does not become fifty pages.

Key decisions

Route by source type instead of picking one detector

Fraud is a multi-feature interaction, server metrics are a deviation from a running baseline, and IoT anomalies are temporal. No single detector handles all three well, so the router chooses per source.

Distrust the simulator

Injected anomalies are cleanly separable — large amount and foreign and odd hour, all at once — so the detectors score near-perfectly on them. Those numbers measure the pipeline, not detection ability, and the README says so before quoting any of them.

Revalidate against real fraud data

Real base rates are often 0.1% or lower, anomalies look almost normal, and there are usually no labels. The simulated result was published as a warning rather than as a result.

What broke

• Simulated data returned an implausible ~1.00 precision and recall. The pipeline was revalidated against a real Kaggle fraud dataset, and the simulation result was published as a warning rather than as a result.

Running it

Full setup, configuration and API reference are in the repository README.

Stack

• Redpanda
• Faust
• PyTorch
• TimescaleDB
• Grafana
• Alertmanager

Written up in

• Real-Time Anomaly Detection With Kafka and FaustTowards AI