NSS Labs has filed an anti-trust law suit against CrowdStrike, Symantec, ESET, and the Anti-Malware Testing Standards Organization (AMTSO) over an alleged conspiracy to prevent independent testing companies from performing unbiased reviews of security software.

In a law suit filed on September 18th 2018 in California, NSS Labs - a for-profit independent security software testing company, states that they are the target of a concerted effort by endpoint protection developers to prevent the performing of objective tests of the developer's security products.

"NSS Labs, the world’s leading provider of cybersecurity testing services, is the direct target of a conspiracy among the EPP Vendor Conspirators, orchestrated in whole or in part through AMTSO, to restrict competition in the testing of cybersecurity products that are critical to, but often fail at, the protection of computer systems operated by governments, businesses and consumers," stated the complaint filed by NSS Labs. "NSS Labs already has suffered substantial damages flowing from the antitrust injury it suffered as a result of the conspiracy and will suffer further injury, including irreparable injury such as permanent loss of market share, unless the acts in furtherance of the conspiracy are enjoined." 

They further state that the Anti-Malware Testing Standards Organization (AMTSO) is being used by these vendors, who allegedly have employees in high level positions at AMTSO, to orchestrate the blocking of companies from testing products if they do not adhere to the organization's standards.

When the AMTSO testing standards were being created, NSS Labs states that they voted against them because they felt that the standards do "little to ensure that vendors cannot block or prevent testers from procuring the product to conduct a test, nor does it prevent vendors from intentionally sabotaging a test."

Of particular interest is a mention that NSS Labs allegedly has information that other testing companies only voted for the AMTSO standards because they were threatened by endpoint protection developers.

"NSS Labs voted against adoption of the AMTSO Testing Standard when adoption was put to a vote of the AMTSO membership and is informed and believes and thereon alleges that AV-Comparatives, AV-Test and SKD LABS also voted against adoption of the AMTSO Testing Standard. NSS is also informed and believes and thereon alleges that other EPP product testers voiced opposition to the adoption of the AMTSO Testing Standard but nevertheless voted for its adoption because EPP Vendor Conspirators threatened not to use their testing services if they voted against adoption of the AMTSO Testing Standard."

While it too soon to know whether there is actually a conspiracy, this is not the first time we have heard about vendors trying to prevent the release of independent test reports. For example, in 2017 CrowdStrike sued NSS Labs to try and prevent them from releasing a testing report during the RSA conference. Ultimately, a court allowed the test to be released

Vendors think lawsuit is baseless

Due to the start of legal proceedings, ESET was unwilling to comment other than to state that they deny the allegations.

"We are aware of the allegations made by NSS Labs. However, as legal proceedings have just been initiated, we are unable to say more at this time beyond the statement that we categorically deny the allegations. Our customers should be reassured that ESET’s products have been rigorously tested by many independent third-party reviewers around the world, received numerous awards for their level of protection of end users over many years and are widely praised by industry-leading specialists."

CrowdStrike has told BleepingComputer in a statement that NSS Labs obtains their products through fraudulent means and that the lawsuit is baseless.

NSS is a for-profit, pay-to-play testing organization that obtains products through fraudulent means and is desperate to defend its business model from open and transparent testing. We believe their lawsuit is baseless. 

CrowdStrike supports independent and standards-based testing—including public testing—for our products and for the industry. We have undergone independent testing with AV-Comparatives, SE Labs, and MITRE and you can find information on that testing here. We applaud AMTSO’s efforts to promote clear, consistent, and transparent testing standards.

In response to CrowdStrike's statement, Vikram Phatak, Chief Executive Officer of NSS Labs, told BleepingComputer:

“We are where we are because we refused to be pay-to-play and CrowdStrike knows it. Their smear tactics are par for the course. They should be ashamed of themselves.”  

BleepingComputer has also contacted Symantec and AMTSO, but had not received a response by the time of this publication.

Update 9/23/18: Added AMTSO's statement below:

The Anti-Malware Testing Standards Organization (AMTSO) is disappointed by the antitrust lawsuit raised by a member organization (NSS), and we categorically deny all claims made against us.
 
We want to clarify who we are and what we stand for. AMTSO was founded in 2008 as an international non-profit association that focuses on addressing the global need for improvement in the objectivity, quality and relevance of security testing methodologies. Our membership is 50+ security vendors and testers. AMTSO provides a forum to discuss, engage, and communicate practices that will advance ethical, transparent and standards-based security testing.
 
AMTSO’s testing standard has requirements for transparency and ethical engagements for both security vendors and testers, and it was developed by testers and vendors for the benefit of the customers of tests. The testing standard is voluntary. It holds both testers and vendors accountable to ethical and fair practices, including ensuring that tests are fair to all participants. It does not tolerate backroom deals, “fitted” results, or offering private, pay-to-play, undisclosed advantages to some vendors but not others.
 
NSS is a member of AMTSO, and one of their employees was an important member of the working group that developed the standard. Rather than trying to use the legal system to tear down what we all built together, we encourage NSS to bring its concerns back to the table and engage with the rest of AMTSO membership to make our industry better. We also encourage the broader cybersecurity community, including other vendors, security testers, press, and customers to read the AMTSO standard as well as our most recent blog entry to see how this standard makes testing more ethical and effective so customers will benefit.  

Update 9/26/18: Added Symantec's statement below:

Symantec is committed to the highest levels of integrity and security on behalf of our customers, employees, and partners. We rely upon testing from third-party organizations for an unbiased view of the effectiveness of our products. We also believe that ethical, fair and transparent testing methodologies across all vendors is fundamental to provide consumers with accurate and unbiased evaluations. Organizations such as AMTSO help provide consistency and a standard of excellence for testing within the industry for the benefit of end users.

Much of the security community has expressed concern and frustration with both the methodology and lack of transparency associated with the testing performed by NSS Labs. In our own experience we have felt concern regarding both their technical capability, as well as the practice of the NSS Labs “pay to play” model in relation to public tests.

We believe that testing, when done with appropriate technical rigor and without bias, is vital to ensure the ongoing improvement of our products and those of our competitors, resulting in a more competitive marketplace that benefits both consumers and enterprises with more effective protection against constantly evolving cyber threats. We are highly supportive of the testing organizations that adhere to the highest ethical standards, including those recommended by the AMTSO.

We are aware of the lawsuit filed by NSS Labs and we believe that their claims against us are entirely baseless. While it’s understandable that NSS Labs’ desire for profits may be inherently at odds with a non-profit, standards-based organization such as AMTSO, the integrity of the testing process should be of utmost importance, starting with transparency and equity for all participants. We welcome the opportunity to bring the discussion of fair and open testing further into the public conversation, while also shining a light on certain business practices within the testing industry.