Why Detection Rate is the Incorrect Metric

Why Detection Rate is the Incorrect Metric

  • By
  • Orca Security
  • |
  • August 28, 2019

Detection rate is the first metric that comes to mind when discussing cybersecurity effectiveness. Everyone wants a higher detection rate –  we’re naturally inclined to view a product that detects 99.7% of threats as superior to one that detects 99.5% of threats. Consequently, vendors place heavy emphasis on this metric with on-the-surface impressive results. For example, more than half of the tested solutions in the recent av-comparatives test had a detection rate equal to or greater than 99.5%.

Stop obsessing over fractions of a percent

However, these results are misleading and this focus is misguided. It’s time to stop obsessing over a fraction of the percent of detection rate gaps and focus on gaps that are higher orders of magnitude instead. When assessing your detection rates, you need to ask yourself the following: How many of my assets are actually protected by these solutions? Which attack vectors do they account for? Do they offer continuous protection or ‘point in time’ security?

The vast majority of breaches occur when assets aren’t fully protected by existing security solutions. We often find that assets are only partially covered (if at all), that not all vectors are accounted for, and that scans aren’t carried out continuously. 

Learning from Research on 100s of cloud environments 

Having conducted research on hundreds (thousands?) of cloud environments and the way their scanning is conducted today, it’s no wonder that so many company assets remain vulnerable to cyber attacks. 

We’re no longer surprised to see the major risks other tools fail to detect, despite high detection rates reported. This is because the issues usually go beyond conventional detection rates- often due to the fact that these tools weren’t deployed across all points. This is why we need to shift organizational focus on to effective detection rates.

Agent-based scanners, network scanners, and cloud security posture solutions (CSPMs) are commonly incapable of protecting cloud assets across all the four layers of the cloud. 

Here are a few examples of what we’ve seen so far:

  • Malware on production server – In this case, our first scan detected malware in the production environment. It sneaked in via an open-source tool that was in use. While the organization had tools in place to scan open-source packages in the CI/CD pipeline, the malware entered the environment before it became well-known. While the malware later became well-known shortly after, the organization’s lack of continuous scanning resulted in a very low effective detection rate.
  • Critical servers not patched for years – In another customer environment, we saw distinct differences between the assets that the IT department owns and manages. Some were maintained and patched regularly while others were left untouched for years. Some of them even ran OSs that had reached their end-of-support phase years before. The vulnerability management tools they used were never integrated into these servers and they were completely unmonitored by the IT department and their tools. In this case, the per-asset integration required by the vulnerability management solution resulted in a very low effective detection rate.
  • RCE on an internet-facing web server – Our customers in this instance used a vulnerable Apache struts server to host their website on an internal URL. The customer had a network-based vulnerability scanner that failed to detect this vulnerability as it never managed to reach the internal URL. The fact that it was dependent on crawling or manual configuration resulted in a very low effective detection rate,  as it never reached this server.

In all of these cases, which are only a small portion of many, the issue doesn’t lie in the detection rate of the organization’s existing tools. The problem lies in their severely limited reach.

Call to action

Customers and security vendors alike must move towards a more real-life assessment of their tools. On top of the detection rate figure, they should count the effective detection rate in terms of the following: (1) the number of assets that will be protected; (2) for which attack vectors; and (3) whether it is a one time or continuous assessment. In simpler words, don’t forget to count the ‘footnotes’ of when a  cybersecurity solution no longer applies to your environment. They might be the doorways to future breaches.

The Orca Security Cloud Visibility Platform was built with a goal to maximize effective detection rate. This is one of the main virtues of Side Scanning – ability to reach 100% of the environment, without per-asset integration and using a one-time, read-only infrastructure level integration. In addition to providing obvious operational value, it means that the security is effectively and continuously applied to each and every asset, without exception.

Recent blog posts