Strategic Guide to the Latest MITRE Endpoint Evaluations

MITRE Engenuity ATT&CK Evaluations

It’s that time of year again – the MITRE Engenuity ATT&CK evaluations of endpoint detection and response (EDR) products are out. As one of the only comprehensive efficacy testing of EDR products available to the public, it generates a lot of buzz in cybersecurity circles. Since this is the third year of its release, you may already know the drill. MITRE Engenuity publishes the raw results of its evaluation but does not offer up any analysis on those results, leaving it up to the public to derive value from the data and cross-compare products’ efficacy. As a security researcher, this gives me the opportunity to get really creative, making it one of my favorite projects to work on at K logix.

Before beginning an analysis, it is important to contextualize the data. A critical question researchers and readers alike should ask is – who is doing the research and why? Back in 2018, MITRE (a federally funded non-profit corporation) started the first round of testing to help organizations make informed technology decisions and encourage innovation. After finishing the 2019 round of testing, MITRE passed the torch to MITRE Engenuity, a non-profit tech foundation MITRE created in 2019. MITRE Engenuity is distinct from MITRE, with a separate board of directors and private funding. Depicted as a “bridge” to the private sector, MITRE Engenuity works towards advancing our critical infrastructure’s security through private sector collaboration. The EDR evaluations, which requires participation among competing companies, nicely complements MITRE Engenuity’s mission.

Since the evaluations are centered within the ATT&CK framework, it is important to understand ATT&CK before making sense of the results. ATT&CK stands for Adversarial Tactics, Techniques and Common Knowledge. Created by MITRE in 2013, ATT&CK emerged as a way to archive adversary behavior by documenting and categorizing adversary tactics, techniques and procedures. Since then, ATT&CK has evolved into a rich repository of adversary activity, offering professionals a common framework to work from when investigating a cyberattack, and is widely accepted within security circles; in 2020, McAfee and the University of California, Berkeley’s Center for Long-Term Cybersecurity (CLTC), conducted a survey among security leaders across 325 organizations and found that 80% utilize the ATT&CK framework.

ATT&CK has two components central to MITRE Engenuity’s EDR evaluations: tactics and techniques. Tactics are high level adversarial attack objectives and are presented linearly across an attack life cycle, starting from the initial point of entry to exfiltration. Techniques are categorized under tactics and are how adversaries achieve their high-level objectives. Adversaries typically combine techniques from many different tactics when targeting a network. In the EDR evaluations, evaluators simulate the behavior of a threat group, documenting the product’s detection of malicious techniques across the ATT&CK tactics (high level hacker objectives).

Testing Environment – 2019 vs. 2020

In each iteration of testing, analysts simulate a different advanced persistent threat group – choosing ATT&CK techniques that align with the group’s behavior. This year, 65 ATT&CK techniques were in scope for the evaluation. Analysts chose to emulate the financially motivated threat group Carbanak/Fin7, known to target financial institutions. According to MITRE Engenuity, Carbanak/Fin7 “often rely heavily on scripting, obfuscation and fully exploiting the users behind the machine while pillaging an environment.” Yikes.

There are notable differences between the 2019 and 2020 round of testing. In 2019, MITRE only tested Windows endpoints. In 2020, MITRE Engenuity expanded the environment scope to include Linux platforms. MITRE Engenuity also expanded the testing purview; for the first time, analysts evaluated the vendor’s protection capabilities. It is important to note that this occurred in a separate environment from the detection evaluation. During the evaluation of vendor’s detection coverage, all proactive protection and blocking capabilities were turned off (as in years prior).

Evaluators document when and how detections occur with a series of detection tags and modifiers. For every malicious technique attempted, testers note the types of detections that occurred with detection tags. Evaluators may add modifiers to provide more context to a detection tag.

The detection tags and modifiers in the 2020 evaluation are different from those used in 2019. Notably, MITRE Engenuity dropped the MSSP detection tag, which indicated if the vendor used analysts to gather relevant information and make a detection. Instead, all detection instances in the 2020 evaluations are fully driven by the product. Another significant difference between the two evaluations is the reduction in modifiers from seven to two. With less modifiers, the main detection tags hold more significance. When cross-comparing vendor performances over the two years, these differences are impactful since they change how we interpret the data.

Analyzing The Results

Now to my favorite step – processing the data and extrapolating valuable results. Here at K logix, the research team designed a repeatable methodology to engage with the MITRE Engenuity EDR evaluations. When crafting a methodology, it is important to keep the ‘why’ top of mind. For us, the ‘why’ is helping our customers find an EDR vendor with capabilities that best match their needs. With this intention in mind, we developed a customizable python script to parse through the raw detection data, incorporate customers’ unique considerations and compare vendor’s performances.

Our script collects detection occurrences, assigns a numerical value to the detection tags and modifiers, aggregates the scores, and then displays vendors’ detection coverage. The numerical value for the detection tags and modifiers is customizable; customers can change the value depending on the priority the detection tag and modifier holds to the organization. The script also enables customers to assign a weight to the tactics (the high-level hacker objectives from the ATT&CK repository); this means that if a customer values specific tactics more, the detections occurrences categorized under those tactics will hold more sway over the results. Afterwards, that data is distilled into the K logix endpoint database which incorporates other relevant EDR characteristics such as threat intelligence, automation, and device management.

For the past two years, the K logix research team has incorporated the MITRE/MITRE Engenuity data into our own EDR evaluations. Doing so situates the MITRE/MITRE Engenuity data within a broader EDR evaluation framework that includes more data points on beneficial capabilities. It also positions K logix to better understand the EDR space and how it changes over time. For more information and to see a representation of the results, contact one of our experts.