There has been some controversy recently around Cylance and the methodology it recommends to test the efficacy of Cylance Protect compared with other endpoint security solutions. There are certainly issues with how antimalware tools are compared and evaluated—especially by the accepted, industry-standard measurements—but those issues are neither unique to, nor a function of Cylance itself.
A recent story from Ars Technica points to a battle over whether or not the files provided by Cylance for prospective customers to test various endpoint products are legitimate or not. According to the story, “In testing, Protect identified all 48 of the samples as malicious, while competing products flagged most but not all of them. Curious, the engineer took a closer look at the files in question—and found that seven weren't malware at all.”
The suggested implication of this finding is that Cylance is trying to game the system. At face value, it appears that Cylance is misleading prospects by using these 48 malware samples, even though some are not even recognized as malware by other endpoint solutions.
The reality, though, is that the fact they’re not recognized as malware by other endpoint security solutions is exactly the point. Chad Skipper, VP of Industry Relations for Cylance, explained, “Cylance does not mislead customers or prospective customers. When we create malware samples to test with, we employ the same methods and tools that hackers do, including creating mutations and packing the samples, to better emulate what attackers do for more meaningful testing. We are not running or using any tool that isn’t already in an attacker’s arsenal.”
The fact is that any time you pack a real file, there is a chance that the original piece of software will break. Some installers use internal checksums that are broken by the packing process, resulting in a valid file that does nothing (but still has the earmarks of malware, even if it doesn’t run correctly). Skipper stressed, “This is how it works in the real world and can be seen frequently in real malware, where the resultant mutated sample doesn’t operate anymore.”
Skipper assured me—with 100 percent certainty—that the original files Cylance packs are, in fact, legitimate malware. “As we do not control the inputs, sometimes this process can result in valid but harmless files being output as attackers change their tactics in how they generate input files. When we become aware of situations where outputs are no longer valid, we make adjustments to our process to remove these to ensure the fairness of results.”
In other words, while the industry standard is to compare antimalware products by detecting and blocking a library of known threats, the files from Cylance are actually a more authentic reflection of what an organization will be faced with in the real world. The reality is that attackers are constantly adapting and mutating threats, so being able to detect and block a threat that is already known is almost useless. It’s like shutting the barn door after the horses have already escaped.
“We agree with Ars Technica that this is an important issue for our industry,” declared Skipper. “By testing only on existing (and known) samples, a test does little to nothing to help someone understand how a product may fare in the real world. We strongly maintain that security teams test for themselves because these tests will be most relevant. Each environment is unique and some of the attempted attacks will be customized for your environment – and attackers will use packers and other techniques to evade traditional signature-based security technologies.”
Cylance is not the first to encounter this issue, either. Years ago, Webroot ran into similar issues trying to get ranked by industry-standard antimalware tests. The problem was that Webroot’s approach to identifying and blocking threats did not conform to the methods of other antimalware vendors, so testing against a library of known threats was simply not an effective measurement.
Security testing needs to change to consider more real-world scenarios. To their credit, NSS Labs as well as AV-Test are leading the industry in evolving testing methodologies to be more real-world. While no independent test is perfect, the goal is for the industry to move forward in developing techniques that will help vendors and customers better understand how products will operate when put into live-fire hostile environments. Hopefully, testing organizations will continue to make positive changes that actually serve the security community and the world at large.
