Avc Report25
0 likes299 views
This document summarizes the results of an on-demand malware detection test conducted by AV-Comparatives in February 2010. 20 antivirus products were tested on their ability to detect malware samples from a set containing over 1.2 million files. The summary includes detection rates for each product, with G DATA, AVIRA, and Panda detecting over 99% of malware. It also includes false positive results, with eScan, F-Secure, and others having very few false alarms. Finally, it shows the award levels reached by each product based on detection and false alarms, with some products reaching the highest ADVANCED+ level.
Avc Report25
- 1. Anti-Virus Comparative No. 25, February 2010 On-demand Detection of Malicious Software includes false alarm and on-demand scanning speed test Language: English February 2010 Last Revision: 17th March 2010 www.av-comparatives.org
- 2. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org Table of Contents Tested Products 3 Conditions for participation and test methodology 4 Tested product versions 4 Comments 5 What’s new in this test 5 Test results 6 Graph of missed samples 8 Summary results 9 False positive/alarm test 10 Scanning speed test 11 Award levels reached in this test 12 Copyright and Disclaimer 13 –2-
- 3. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org Tested Products • avast! Free Antivirus 5.0 • Kingsoft AntiVirus 2010 • AVG Anti-Virus 9.0 • McAfee AntiVirus Plus 2010 • AVIRA AntiVir Premium 9.0 • Microsoft Security Essentials 1.0 • BitDefender Antivirus 2010 • Norman Antivirus & Anti-Spyware 7.30 • eScan Anti-Virus 10.0 • Panda Antivirus Pro 2010 • ESET NOD32 Antivirus 4.0 • PC Tools Spyware Doctor with AV 7.0 • F-Secure Anti-Virus 2010 • Sophos Anti-Virus 9.0 • G DATA AntiVirus 2010 • Symantec Norton Anti-Virus 2010 • K7 TotalSecurity 10.0 • Trend Micro AntiVirus plus AntiSpyware 2010 • Kaspersky Anti-Virus 2010 • Trustport Antivirus 2010 –3-
- 4. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org Conditions for participation and test methodology The conditions for participation in our tests are listed in the methodology document at http://www.av-comparatives.org/seiten/ergebnisse/methodology.pdf. Before proceeding with this report, readers are advised to first read the above-mentioned document. The participation is limited to not more than 20 well-known and worldwide used quality Anti-Virus products, which vendors agreed to get tested and included in the public test-series of 2010. Tested Product Versions The Malware sets have been frozen the 3rd February 2010. The system sets and the products were updated and frozen on the 10th February 2010. The following 20 up-to-date products were included in this public test: • avast! Free1 Antivirus 5.0.396 • Kingsoft AntiVirus 2010.2.10.1 • AVG Anti-Virus 9.0.733 • McAfee AntiVirus Plus 14.0.306 • AVIRA AntiVir Premium 9.0.0.457 • Microsoft Security Essentials 1.0.1611.0 • BitDefender Anti-Virus 13.0.19.347 • Norman Antivirus & Anti-Spyware 7.30 • eScan Anti-Virus 10.0.1058.644 • Panda Antivirus Pro 9.01.00 • ESET NOD32 Antivirus 4.0.474.0 • PC Tools Spyware Doctor with Antivirus 7.0.0.514 • F-Secure Anti-Virus 10.12.108 • Sophos Anti-Virus 9.0.3 • G DATA AntiVirus 20.2.4.1 • Symantec Norton Anti-Virus 17.5.0.127 • K7 TotalSecurity 10.0.0025 • Trend Micro AntiVirus plus AntiSpyware 17.50.1366.0 • Kaspersky Anti-Virus 9.0.0.736 (a.b) • Trustport2 Antivirus 5.0.0.4087 K7, Panda, PC Tools and Trend Micro are new participants of the 2010 test-series. Please try the products on your own system before making a purchase decision based on these tests. There are also some other program features and important factors (e.g. price, ease of use/management, compatibility, graphical user interface, language, HIPS / behaviour blocker functions, URL filter/reputation services, support, etc.) to consider. Some products may offer additional features e.g. to provide additional protection against malware during its execution (if not detected in advance on-access or on-demand). Although extremely important, the detection rate of a product is only one aspect of a complete Anti- Virus product. AV-Comparatives provides also a whole product / dynamic test, as well as other test reports which cover different aspects/features of the products. 1 Alwil Software decided to participate in the tests with their free product version. 2 Based on two engines (AVG and Bitdefender). –4-
- 5. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org Comments Almost all products run nowadays by default with highest protection settings (at least either at the entry points, during whole computer on-demand scans or scheduled scans) or switch automatically to highest settings in case of a detected infection. Due that, in order to get comparable results, we tested all products with highest settings, if not explicitly advised otherwise by the vendors (as we will use same settings over all tests, the reason is usually that their highest settings either cause too many false alarms, have a too high impact on system performance, or the settings are planned to be changed/removed by the vendor in near future). To avoid some frequent questions, below are some notes about the used settings (scan of all files etc. is always enabled) of some products: AVIRA, Kaspersky, Symantec, TrustPort: asked to get tested with heuristic set to high/advanced. Due to that, we recommend users to consider also setting the heuristics to high/advanced. F-Secure, Sophos: asked to get tested and awarded based on their default settings (i.e. without using their advanced heuristics / suspicious detections setting). AVG, AVIRA: asked to do not enable/consider the informational warnings of packers as detections. Due that, we did not count them as detections (neither on the malware set, nor on the clean set). AV-Comparatives prefers to test with default settings. As most products run with highest settings by default (or switch to highest automatically when malware is found, making it impossible to test against various malware with “default” settings), in order to get comparable results we set also the few remaining products to highest settings (or leave them to lower settings) in accordance with the respective vendors. We hope that all vendors will find the appropriate balance of detection/false alarms/system impact and will provide highest security already by default and remove paranoid settings inside the user interface which are too high to be ever of any benefit for normal users. What’s new in this test You will notice that this time the test-set is smaller than previous times. This is because we are now trying to include in the test-set mainly prevalent real-world malware being still around (within last eight months). To build the test-set we consulted (but as it was a first try, we did not exclusively rely on) metadata and telemetry data collected and shared within AV industry, as well as querying various clouds afterwards and requesting data of the most-common user-submitted malware. Malware we see on user PC’s are automatically considered as important. Nevertheless, as with every first attempt, we noticed that not all sources are yet able to provide reliable prevalence data, so we had to clean up some sets afterwards. This will improve in future, as the industry is currently working on optimizing their data sharing processes. Anyway, considering that the used test-set will probably get smaller and focus mainly on threats that should be detected and should be easier to spot, we expect products to score higher. This is the reason why we may increase next time the marks to get higher awards. Next time also the marks for FPs to get “ADVANCED+” may be set stricter. What’s also new and interesting is the prevalence information we try to give inside the detailed false alarm report (see link on page 10). –5-
- 6. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org Test Results Below are the test results tables containing the detection rate details of the various products. –6-
- 7. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org –7-
- 8. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org Graph of missed samples (lower is better) Percentages refer to the used test-set only. Even if it is just a subset of malware, due its size, it is important to look at the number of missed malware. For example, 0.3% means almost 3700 missed malware samples from the used test-set. The results of our on-demand tests are usually applicable also for the on-access scanner (if configured the same way), but not for on-execution protection technologies (like HIPS, behaviour blockers, etc.). A good detection rate is still one of the most important, deterministic and reliable features of an Anti- Virus product. Additionally, most products provide at least some kind of HIPS, behaviour-based or other functionalities to block (or at least warn about the possibility of) malicious actions e.g. during the execution of malware, when all other on-access and on-demand detection/protection mechanism failed. Please do not miss the second part of the report (it will be published in a few months) containing the retrospective test, which evaluates how well products are at detecting new/unknown malware. Even if we deliver various tests and show different aspects of Anti-Virus software, users are advised to evaluate the software by themselves and build their own opinion about them. Test data or reviews just provide guidance to some aspects that users cannot evaluate by themselves. We suggest and encourage readers to research also other independent test results provided by various well-known and established independent testing organizations, in order to get a better overview about the detection and protection capabilities of the various products over different test scenarios and various test-sets. –8-
- 9. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org Summary results Please consider also the false alarm rates when looking at the below detection rates3! Total detection rates4: 1. G DATA 99.6% 2. AVIRA 99.3% 3. Panda 99.2% 4. Trustport 99.1% 5. McAfee 98.9% 6. PC Tools 98.7% 7. Symantec 98.6% 8. F-Secure 97.8% 9. ESET 97.7% 10. Bitdefender, eScan 97.5% 11. Avast 97.3% 12. Kaspersky 97.1% 13. K7 96.4% 14. Microsoft 96.3% 15. AVG 94.2% 16. Sophos 93.7% 17. Norman 92.7% 18. Trend Micro 90.7% 19. Kingsoft 81.8% The used test-set contains about 1.2 million malware samples and consists of: 3 We estimate the remaining error margin to be around 0.2% 4 Additional results: The McAfee detection rate with “very high” in-the-cloud sensitivity would have scored 99.0% and have “very many” false alarms. Baseline minimum detection rates of some products when there is no Internet connection available (i.e. without their in-the-cloud technology): McAfee: 94.9% (19 FPs) , Panda: 73.3% (32 FPs), Trend Micro: 68.5% (22 FPs) –9-
- 10. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org False positive/alarm test In order to better evaluate the quality of the detection capabilities of anti-virus products, we provide also a false alarm test. False alarms can sometimes cause as much troubles as a real infection. Please consider the false alarm rate when looking at the detection rates, as a product which is prone to cause false alarms achieves higher scores easier. All discovered false alarms were reported and sent to the respective Anti-Virus vendors and have now been already fixed. False Positive Results Number of false alarms found in our set of clean files (lower is better): 1. eScan 1 2. F-Secure 2 very few FP’s 3. Bitdefender, Microsoft, ESET 3 4. Sophos 4 5. Kaspersky, G DATA 5 6. PC Tools 8 few FP’s 7. Trustport 9 8. AVG 10 9. Avast, Symantec, AVIRA 11 10. Trend Micro 38 11. Panda 47 12. McAfee 61 many FP’s 13. Norman 64 14. Kingsoft 67 15. K7 193 very many FP’s The details about the discovered false alarms (including their prevalence) can be seen in a separate report available at: http://www.av-comparatives.org/images/stories/test/fp/avc_report25_fp.pdf The graph below shows the number of false alarms found in our set of clean files by the tested Anti- Virus products. – 10 -
- 11. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org Scanning Speed Test Anti-Virus products have different scanning speeds due to various reasons. It has to be taken in account how reliable the detection rate of an Anti-Virus is; if the Anti-Virus product uses code emulation, if it is able to detect difficult polymorphic viruses, if it does a deep heuristic scan analysis and active rootkit scan, how deep and thorough the unpacking and unarchiving support is, additional security scans, if it really scans all file types (or uses e.g. white lists in the cloud), etc. Most products have technologies to decrease scan times on subsequent scans by skipping previously scanned files. As we want to know the scan speed (when files are really scanned for malware) and not the skipping files speed, those technologies are not taken into account here. In our opinion some products should inform the users more clearly about the performance-optimized scans and then let the users decide if they prefer a short performance-optimized scan (which does not re-check all files, with the potential risk of overlooking infected files!) or a full-security scan. The following graph shows the throughput rate in MB/sec (higher is faster) of the various Anti-Virus products when scanning (on-demand) with highest settings our whole set of clean files (used for the false alarm testing). The scanning throughput rate will vary based on the set of clean files5, the settings and the hardware used. The average scanning throughput rate (scanning speed) is calculated by the size of the clean-set in MB’s divided by the time needed to finish the scan in seconds. The scanning throughput rate of this test cannot be compared with future tests or with other tests, as it varies from the set of files, hardware used etc. The scanning speed tests were done under Windows XP SP3, on identical Intel Core 2 Duo E8300/2.83GHz, 2GB RAM and SATA II disks. 5 to know how fast various products would be on your PC at scanning your files, we advise you to try the products yourself – 11 -
- 12. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org Award levels reached in this test AV-Comparatives provides a 3-level-ranking-system (STANDARD, ADVANCED and ADVANCED+). As this report contains also the raw detection rates and not only the awards, users that e.g. do not care about false alarms can rely on that score alone if they want to. AWARDS PRODUCTS (based on detection rates and false alarms) G DATA AVIRA TrustPort PC Tools Symantec F-Secure ESET BitDefender eScan Avast Kaspersky McAfee* Panda* Microsoft Sophos AVG K7* Norman* Trend Micro* Kingsoft *: those products got lower awards due false alarms The Awards are not only based on detection rates - also False Positives found in our set of clean files are considered. A product that is successful at detecting a high percentage of malware but suffers from false alarms may not be necessarily better than a product which detects less malware but which generates less FP’s. The awards were given according to the table below (may change next time): Detection Rates <87% 87 - 93% 93 - 97% 97 - 100% Few (0-15 FP’s) TESTED STANDARD ADVANCED ADVANCED+ Many (over 15 FP’s) TESTED TESTED STANDARD ADVANCED – 12 -
- 13. Anti-Virus Comparative - No. 25 – February 2010 www.av-comparatives.org Copyright and Disclaimer This publication is Copyright © 2010 by AV-Comparatives e.V. ®. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV- Comparatives e.V., prior to any publication. AV-Comparatives e.V. and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV-Comparatives e.V. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data. AV-Comparatives e.V. is a registered Austrian Non-Profit-Organization. For more information about AV-Comparatives and the testing methodologies, please visit our website. AV-Comparatives e.V. (March 2010) – 13 -