Gordon Werner

RIT

and

S. Jay Yang

RIT

CLEAR-ROAD: Extraction of Temporally Co-occurring yet Rare Critical Alerts (pdf)

Intrusion detection systems generate a large number of streaming alerts. It can be overwhelming for analysts to quickly and effectively understand behavior within a network. Critical alerts occur so infrequently that it can be difficult to determine what surrounding alerts are actually related to them, providing a deep challenge to analysts. What if an analyst could provide a collection of known critical alerts and quickly receive a summary detailing their temporal behaviors within a network as well consistently co-occurring signatures that pre-empt or succeed the critical action? What if this information could be provided in near real time, with no training data, and with the capability to adapt to changing temporal patterns and relationships across signatures? The Concept Learning for Intrusion Event Aggregation in Realtime with Rare co-Occurring Alert signature Discovery (CLEAR-ROAD) answers that question, revealing consistent co-occurrences derived from alerts with similar temporal arrival patterns. Alerts are aggregated, or sequenced, based on their unique and invariant arrival patterns, not external training data. The signature patterns expressed by such temporal activity are then discovered through pattern mining techniques. A constrained databasing approach is used to reduce the number of sequences processed by an average of 90\% for individual streams. Case studies are conducted to analyze the co-occurring signatures found across two real world datasets, one from a SOC operation and another from a penetration testing competition. CLEAR-ROAD is able to find consistently co-occurring signatures across streams and datasets quickly and effectively. Differences in temporal behavior are also found to lead to unique co-occurring signatures for some critical alerts. Case studies show the clear and near-immediate benefits provided to analysts by the system.