Throughout the year AV-Comparatives.org tests security products in a variety of ways. Each product that passes a test receives at least a STANDARD rating. Those that perform better than most but still have areas needing work get an ADVANCED rating, and the very best receive the rating ADVANCED+. At the end of the year, the company reports on all tests and names a product of the year. This year Kaspersky took that honor.
How They Test
In the on-demand test researchers scan hundreds of thousands of malware samples and note what percentage each product eliminates. Theretrospective test is similar, but by forcing each product to use old virus definitions it emulates proactive detection of zero-day threats. The whole product dynamic test challenges each product to resist infection by hundreds of active threats over a period of weeks. The performance testmeasures how much impact each product has on system performance.
AV-Comparatives runs those four tests twice each year. This year saw the introduction of a new removal test. Researchers gathered a collection of threats that had been detected by every tested product for at least six months and then challenged each product to fully remove those threats. The chart below shows all results from last year.
In this chart ADV+, ADV, and STD stand for ADVANCED+, ADVANCED, and STANDARD. An empty cell indicates that the product was tested but failed to reach the STANDARD level. A black cell with “n/a” in white letters simply means AV-Comparatives didn’t test that particular product. In a grey box, “n/a” means the vendor actively choose to abstain from the test.
Why would anyone abstain? A number of vendors believe that the retrospective test doesn’t accurately reflect their product’s capabilities. You can figure out which ones from the chart.
Kudos to Kaspersky
Although the raw test scores may differ, the test reports advise that all products with the same rating should be treated as equivalent. To this end, the researchers use a clustering technique rather than predefining hard cut-offs for each rating level.
For selecting the product of the year, researchers look for the greatest number of ADVANCED+ ratings. This year there was a single clear winner. Kaspersky aced every single test. The 2010 winner, F-Secure, missed by just one score of ADVANCED rather than ADVANCED+.
Any product that earned at least five ADVANCED+ ratings is designated a top rated product. Besides Kaspersky, Avira, Bitdefender, ESET, and F-Secure earned this honor.
I should point out that the Webroot product tested here is completely different from PCMag Editors’ Choice Webroot SecureAnywhere Antivirus. The report states, “It is very possible that Webroot’s results in future tests will improve as a result of the new engine.” Norton, our other Editors’ Choice, was named product of the year for 2009. It didn’t get top ratings in 2011, but did score a win specifically for effective malware removal and low performance impact.
The full report lists winners in a number of specific categories including malware detection, scanning speed, and low false positives. You’ll also find a useful and informative report on the process of installing and using each product, with screenshots.