Detection-first telemetry reduction

Summary

Cutting telemetry volume is easy. Cutting it without weakening detection is the actual problem. This paper describes a method that makes detection coverage an explicit, measurable constraint on every reduction decision, so that "did we just break a rule?" is answered before the change rather than after an incident. CyberAIX Pipeline implements this method natively; the method itself is tool-independent.

1. Why volume-first fails

Most reduction programs start with the noisiest sources and cut. Savings appear quickly, and so do silent gaps. A drop rule on firewall permit events also drops the lateral-movement rule that depends on internal permits. A field-trimming rule that removes a verbose payload also removes the one field a phishing detection parses. The analytics platform stays green because a rule with nothing to report looks exactly like a rule that is broken.

The correction is to invert the starting point: begin from detections, derive the data they need, and protect that set.

2. Building the protected set

List every live detection. Rules, correlation searches, and scheduled analytics from every engine in use.

Extract field references. For each rule, every field it reads: filters, aggregations, joins, thresholds, and anything reached through a lookup or macro.

Map fields to sources. For each field, which sources can populate it. The result is a matrix of sources against detections.

Weight by consequence. Rules tied to a regulatory control, an active threat campaign, or a recent confirmed incident are protected absolutely. Others are protected by default and negotiable with evidence.

The protected set is the union of every field read by a protected rule, per source.

3. Reduction techniques, safest first

Deduplication. Identical events arriving by multiple paths. Removes nothing unique. Always safe.

Field trimming. Removing fields outside the protected set that no analyst queries. Check a query log before trimming fields used ad hoc.

Event filtering. Dropping whole event types no rule reads. Requires replay validation, because an event type may be referenced under a different name.

Sampling. Keeping a statistical fraction of homogeneous, high-volume events. Appropriate only for events consumed in aggregate, never for events that matter individually.

4. Route, do not delete

Every reduction rule declares where removed data goes. The default is compressed, queryable object storage. Outright discard requires an explicit policy statement and a retention justification. Investigative and compliance access is preserved while the premium-tier saving is realized.

5. Prove it by replay

Before a reduction rule is activated:

1. Capture a representative window of historical events for the affected source, at least a week including a weekend.
2. Run every protected detection against the unreduced capture and record hit counts.
3. Apply the proposed rule to the capture.
4. Run the same detections against the reduced capture.
5. Any change in hit counts blocks activation. Adjust and repeat.

This step should be automated, and no rule should be activatable without a passing replay report attached.

6. Keeping it true

Drift monitoring. Baseline field presence per source and alert when a protected field's presence rate drops.

Rule lifecycle hooks. When a detection is added or changed, recompute the protected set and re-validate any affected reduction rule.

Quarterly review. Retire reduction rules protecting fields no longer read. Add rules for fields that have become noise.

7. What to expect

Organizations applying this method typically find that most analytics volume falls outside the protected set, and that deduplication and field trimming alone deliver meaningful savings before any event filtering is considered. Exact figures depend on source mix and detection maturity and should be measured during profiling rather than assumed in advance.

8. Closing

Volume-first reduction trades cost for invisible risk. Detection-first reduction makes the risk visible and puts it under control. It is more work at the start and far less after the first incident that did not happen.