Kyla Guru,

Robert Moss

and

Mykel Kochenderfer

End-to-End Framework using LLMs for Technique Identification and Threat-Actor Attribution (pdf, video)

Despite the growing number of cyber-attacks per day, technical attribution, or the act of identifying the responsible group behind a cyber-attack, remains a complex but mission-critical task for defenders. Delays in attribution often stem from the manual process of picking apart dense, unstructured forensic documentation to identify the tactics, techniques, and procedures (TTPs) of the threat actor, and then piecing together various information for attribution. While previous approaches have looked at classical NLP methods to identify TTPs, an end-to-end ML framework that uses LLMs to identify TTPs and then makes attribution predictions based on these TTPs has not yet been presented or evaluated. This research looks at evaluating the use of Large Language Models (LLMs) and vector embedding search for conducting attribution of a cyber-attack based on behavioral techniques identified within CTI documentation. We analyze similarity to human-generated TTP sets, as well as strengths and limitations of each approach, evaluating on analyst interpretability, tendency to hallucinate, and contextual understanding. This research also introduces an end-to-end ML model that takes in unseen documentation, extracts TTPs, and uses these TTPs to perform attribution. This research finds that while both approaches generate TTP datasets that are different from the tested human-generated datasets, they still prove useful and can be used to train a model that performs above baseline on cyber-attack attribution. This study also finds that the performance of the model greatly improves when a human analyst is added into the loop, providing more information to the model such as the relevancy of various threat actors at the time of analysis.