06.14.07
When Marketing Spin Hurts You
As most people know, my Bloglines is FULL of blogs that I read on a regular basis. Generally these blogs are interesting, however they are occasionally filled with marketing spin. This happened with a recent post on the F-Secure blog. It seems they failed the VB100 test (which they normally pass) and wanted to head it off at the pass before anyone else could talk about it. That's fine... they didn't send the most up-to-date signatures that they could have sent, and VB100 doesn't perform an update after an install. A mistake on F-Secure's part but it's minor... they even went so far as to get somebody from Virus Bulletin to comment on the quality of their product. I was impressed by this, generally I wouldn't think that a neutral organization would do this, so it gives the product a big of a boost in my eyes. However one statement in the post did catch me offguard:
"we nowadays ship around six updates a day"
I realize that this was meant as a positive thing, but it got me thinking and I just can't see it in a positive light. When you are pushing out that many updates it tells me one of two things. i) You are "sweatin' the small stuff" or ii) You have a bad QA process and need to push out fixes. I was curious about this, so I went over to Symantec and checked out Threat Explorer. It listed 62 new threats dated back to May 16th. So if we take May 16th - June 13th, that's 29 days. Which means 2.1 threats per day. Now, that's Symantec, not F-Secure so I went over and took a look at their Virus Description Modifications page. Of course there's no guarantee that these are new virus descriptions, in fact I noticed some that I'm fairly sure aren't... but I figure description modifications would include creation. Over the same 29 day period, they have 21 viruses. That's 0.7 threats per day. I grabbed a few more stats from other sites; Panda listed 11 (0.37 threats per day) and McAfee listed 24 (0.8 threats per day).
So, even if we use Symantec's numbers for F-Secure, there's no reason to push out more than 2 updates per day, and even then, I would feel that one update daily would be more than sufficient. Even that may be overkill in some cases but I could see a "daily" update. I think the last thing a company should be doing is bragging about 6 updates per day, especially when the number of new virus threats if much way below that. This makes me think of the poor QA we've seen out of AV companies in the past; McAfee update exterminates Excel or Flawed Symantec update cripples Chinese PCs. It really makes me question the integrity and quality of the updates that I'm receiving.
So kudos F-Secure on the quick thinking to put a positive spin on your failure to pass VB100, but at "six updates a day" you've failed something even bigger in my eyes. Given that the F-Secure blog was one that I enjoyed reading and that I'd just picked up my new Vista PC, I was thinking about buying F-Secure Internet Security 2007 and installing it on my machine. Now I'm afraid to... the product has just gone to the bottom of my list and it's all because "six updates a day" leaves a bad taste in my mouth.
[UPDATE]
I was able to walk the Exhibitors floor at InfoSec Canada earlier today and one of the things I was able to do was chat with a Sales guy from F-Secure. He was actually really great to talk to... I wanted to know more about the "6 updates a day" and he was more than happy to discuss it and provide me with some information. He said that 6 is just an average, that some days they are much higher with a record of ~21 in a single day. I found this little shocking, and he named other AV vendors and pointed out that some of them will do 20-30 updates daily. (I would love to hear from any AV vendors that read this, just how many updates your software has daily).
He also said that it's because their software includes not only updates for malware but also for spyware. Maybe it's just me, but I've always considered malware to be blanket term that includes worms, trojans, adware and spyware. Anyways that was another bit of justification he gave for the number of updates he did. Either way I'm still not overly impressed, however the guy was friendly enough and provided me a 30-day demo of all their products (including their mobile solution) so I may check them out and see what I think... but if I really do see 6-updates daily... I don't think it'll be around longer than 30 days. As a side note they had nice swag at their booth... the key chains that you push to separate into two key chains, with the F-Secure logo etched into the side.
[UPDATE 2]
F-Secure was nice enough to blog about this issue. However they couldn't even spell my name correctly, they currently have my last name spelled 'Regulay'. They attempt to address the issue I've brought up... and I'm glad that they have clarified (as kurt already had) that they only do descriptions for, what I'm going to call, 'media worthy' viruses. They were also good enough to provide a link to F-Secure Forums, which contains a change log for each of their releases (well almost each, I noticed certain daily releases weren't included...I'm guessing those are the ones with big QA fixes). I say big QA fixes because, for example, June 14th - Package #4 saw the deletion of 29 pieces of malware, most of which were replaced in that update. My assumption if they are replacing them is that they i) had bad QA or ii) jumped the gun and shipped improper detection before they had all the facts. They also mention that one day they had 11 updates... I definitely feel this is too much... I know there are lots of people that agree with me, since they've voiced their belief privately... hopefully some of them will speak up.
kurt wismer said,
June 15, 2007 at 1:23 am
you have miscalculated the number of new malware threats per day by a couple orders of magnitude, in part because you’re counting the wrong things… i think there’s an interesting misunderstanding here that basically boils down to the fact that those documented threats are not representative of total set of malware being processed by anti-virus companies each day - they’re just the ones that stick out…
Tyler Reguly said,
June 15, 2007 at 1:32 am
That could very well be the case… Yet I still wouldn’t care if it was 100 new threats daily… that doesn’t justify more than a single daily update.
One of the things that the F-Secure sales guy said to me was “this way you don’t end up waiting a month like you would with Microsoft”… The business industry asked for that change.. They didn’t want constant updates, they wanted a predictable pattern, and Microsoft gave them exactly what they asked for.
Any other industry publishing updates provides you with a list of what’s updated and what’s changed… I have been through the F-Secure FTP and website and haven’t seen a changelog for any of their updates. Even if it was 1000 new threats daily… how many of those are insignificant… how many of those affect a single person… or are submitted by the actual virus author and never released.
Until I see a valid reason for 6 updates a day, I’m going to write it off as frivolous and completely useless.
BeltnBraces said,
June 15, 2007 at 6:50 am
I think you may have missed the point. Would you like your new Vista PC to be open to attack by some new virus/exploit for up to 24 hours pending the release of an update that you could have had as soon as it was developed?
If the answer to the above is ‘No’ then now you have a valid reason for 6 updates a day.
If your answer to the above is ‘yes’, then… I dont know what to say without appearing rude.
kurt wismer said,
June 15, 2007 at 7:45 am
“One of the things that the F-Secure sales guy said to me was “this way you don’t end up waiting a month like you would with Microsoft”… The business industry asked for that change.. They didn’t want constant updates, they wanted a predictable pattern, and Microsoft gave them exactly what they asked for.”
you think the increased update frequency of anti-virus products wasn’t also requested by their customers? pressure to update more frequently probably started as early as the melissa word macro virus if not earlier…
“Even if it was 1000 new threats daily… how many of those are insignificant… how many of those affect a single person… or are submitted by the actual virus author and never released.”
it’s impossible to know beforehand which ones are going to be significant… and as for viruses that are submitted but never released, that’s not really the norm anymore in a financially motivated malware world…
“Until I see a valid reason for 6 updates a day, I’m going to write it off as frivolous and completely useless.”
the obvious (and by now rather old) reason is mass mailing worms… they were able to go global in 24 hours so a once a day update schedule wouldn’t have been sufficient to get ahead of a well timed release… the same principle applies to the non-replicative malware that is being mass mailed today (even though it’s not mass mailing itself)…
Ross Barrett said,
June 15, 2007 at 10:10 am
HT, it could be that they have multiple groups working on different aspects of their detection, e.g. a virus group, a ad/spyware group, a trojan/rootkit group, and that each group has it’s own schedule for updates. The updates go out when they are ready and are not held up by the other groups. Also, if I was inclined to install AV software on my box (which I’m not), I personally would want partial detection of a threat (for instance covering one of several vectors) ASAP and not have to wait for a perhaps more fully matured detection. So long as they are not quarantining critical parts of my OS or productivity suite I’m not worried about their QA process.
-Ross
Tyler Reguly said,
June 15, 2007 at 10:12 am
@BeltnBraces: The easiest way to avoid a virus is to be smart. Don’t visit certain websites, don’t click links you don’t know, etc… As far as exploits, a simple firewall would do the trick there
Yet you seem to be the one missing the point. Microsoft does this… if there’s any exploits to be worried about, it’s quite often in one of their products yet they have a regular patch cycle. As for my PC being open to attack for 24 hours… As far as I’m concerned it should be open to attack for 24 hours anyways… I don’t see how they could possibly properly QA their signatures in various environments to ensure no problems if they are releasing them in less than 24 hours.
@kurt
Do you have evidence that customers want this many updates? I would love to see it.
As for it being impossible to tell which will be significant… I disagree… it’s not unlike vulnerabilities. Significance is definitely measurable and risk can be associated with a virus. Anyone who thinks you can’t determine (within a reasonable degree of accuracy) the significance of a viruses, probably shouldn’t be working in the industry. I’m not saying it has to be exact… but you can start with basic categories… here’s a simple example:
Type (I’m sure this could be improved upon, as I said… simple example)
Rating 5 - Worm
Rating 4 - Virus
Rating 3 - Trojan
Rating 2 - Spyware/Keylogger
Rating 1 - Adware
Method of Spreading (again it could be improved upon)
Rating 5 - Multiple Remote Exploits
Rating 4 - Single Remote Exploit / Email (Requiring Viewing Only) / Website (Requiring Viewing Only)
Rating 3 - Email (Require User Interaction) / Website (Requiring User Interaction)
Rating 2 - Installation of other software
Rating 1 - Direct Download
Damage
Rating 5 - Monitoring of Personal Information
Rating 4 - Botnet
Rating 3 - Remote Access
Rating 2 - File Deletion
Rating 1 - Self Replication (No malicious activities)
Sure I’ve left plenty out.. but there’s a very basic system… a Scoring system of 15. Those rated 10-15 may warrant multiple updates daily as they are ready…Those rated 5- 10 could go daily or maybe even weekly and those that are below 5 could go weekly or even monthly.
Tyler Reguly said,
June 15, 2007 at 10:17 am
@Ross
Unfortunately I was typing my last comment while you were leaving yours.
It is entirely possible that different teams go at different times… That’s an update cycle I could see… However if that was the case, then I want the software to allow me to select when I receive updates from certain teams..
As for wanting immediate detection of threats covering multiple vectors… I agree with that… You’ll note in my rating system that those would still receive daily updates. I would want a signature for a very real threat, however 95% of the information listed on F-Secure’s annoucement forum (Which I’m about to add to the parent post) is fairly useless / minor stuff.
[for malware that,new virus descriptions]-0.7 New Threats Per Day? « ewarblog test 1 said,
June 15, 2007 at 1:31 pm
[...] Tyler Reguly over at ComputerDefense.org saw our weblog post on the missed VB100 test. We mentioned there that we release about six updates per day. He felt that it was overkill to do that many updates based on our number of new virus descriptions. The fact is that we normally only create descriptions for malware that are widespread, that are unique, that we get questions about, or that get mentioned in the media. It has little to do with the amount of new malware our products detect. [...]
kurt wismer said,
June 15, 2007 at 7:22 pm
“Do you have evidence that customers want this many updates? I would love to see it.”
what, you mean like users saying ‘i prefer X because it has weekly/daily/hourly updates’ when most didn’t update that frequently? because that i’ve certainly seen, however it’s been a while (the update frequency hasn’t really increased that much in recent memory) and sifting through google groups for something like that is no fun…
alternatively, the very fact that the updates are that frequent could be considered as evidence for the existence of market pressure for frequent updates… distributing updates costs money, the more you distribute the more it costs… unless all anti-virus companies (all of them increased their update frequency) are run by buffoons who throw good money after bad, it’s logical to assume they increased their update frequency because they were losing market share to competitors who’d already increased their own update frequency…
“As for it being impossible to tell which will be significant… I disagree… it’s not unlike vulnerabilities. Significance is definitely measurable and risk can be associated with a virus. Anyone who thinks you can’t determine (within a reasonable degree of accuracy) the significance of a viruses, probably shouldn’t be working in the industry. I’m not saying it has to be exact… but you can start with basic categories…”
i’m afraid you’ve oversimplified things far too much… external and largely unpredictable factors (ie. the systems and people that a piece of malware encounters in the wild, as well as the nature in which it’s deployed and the effort put into doing so) are the key elements that determine which will be significant and which won’t… enumerating properties of the malware itself tells you about some of it’s potential but not enough to predict what will happen in practice to any usable degree of accuracy…
or at least it never has in the past… if you’re certain you can do what others have failed to do then you have a bright future in the industry, assuming you can prove it…
David Harley said,
June 24, 2007 at 8:25 am
I’m a little late into this discussion, so I’ll pick on one or two points, not quite at random.
1) The fact that you’re focusing on viruses suggests to me that you’re still hung up on a 1990s worldview. Viruses haven’t disappeared, but their market share has shrunk to the point where it’s rarely worth talking about them in isolation from other forms of malwre. The trouble is, the current threatscape is infinitely more complex (not to say messy) and dynamic. AV is a much tougher job than it was then. Most relevantly to this discussion, sheer glut makes it much harder to count what’s out there, let alone categorize it so that we know that my sample X is the same sample as your sample Y. If you can’t do that, you have little hope of realistically assessing real-world risk. I’m with Kurt on that (hi, Kurt! Long time no mail.)
2) I’d accept that spyware -is- malware. (Usually.) Though exactly what either is remains contentious. Your contact at F-Secure may have been distinguishing on the grounds that AV is generally expected to catch at least a proportion of all malware, whereas there are products that specialize in spyware, but I’m not convinced that it’s germane to your point either.
3) I think your mention of QA fixes is a wee bit off-base. Granted, an update may sometimes include other stuff (partly depending on the vendor) like slipstream bug fixes, engine updates, application patches, whatever, but most of the time it’s just definitions. Signatures, if you must. And yes, there are high profile, PR-disastrous problems with false positives from time, but they’re actually astonishingly rare. Yes, an update sometimes improves on a previous definition. What’s wrong with that? You modify a definition as you learn more about a dynamic threat. Often, your modification is to genericize it so that catches more variants and subvariants. I’m reassured when I notice that a vendor is responsive to better information: I don’t assume they were wrong in the first place. Sometimes they were, of course, but I’m not perfect either.
4) You’re mistaken if you think that a vendor’s current threatlist tells you much about the samples they’re looking at currently, let alone the global threatscape. It’s not a metric: at most, it’s a list of malware they’re guessing to be most “interesting” at the moment. But it’s just a guess, and will include fairly generic names, not the dozens of subvariants and repacks that may be on the workbench at the moment.
5) Your rating system would be practically valueless to me, wearing my administrator hat. Actually, the rating systems some AV companies still have don’t work very well for me either. They’re usually weighted towards multiple reports from multiple sites, but as a useful metric, that only works for old-fashioned mass mailers and the like, and only then if very carefully done. In real life, I used to find them a liability, because I wasted a lot of time explaining to local administrators that either they or the vendor site were misinterpreting.
There was a time when I updated my users’ systems according to my own risk assessments, but that was several years ago and the threatscape was very different. If I were still that hands-on, I’d accept as many updates as the vendor chose to supply, so that neither I nor they wasted time guessing which upcoming variants would have the most impact. Subject to the major proviso that an update mechanism that hurts business processes is a major problem. I’d expect my user population to have a similar perspective in one sense: “I don’t mind how often you update as long the updates don’t get in my way.” In another sense, there is a difference in that they’d generally expect 100% detection in return for all those updates, and that isn’t going to happen.
In fact, you’re not altogether wrong: it’s not necessarily the vendor with the most updates you want to use, except in so far as it shows that they’re responsive to processing incoming samples. What I want are vendors who:
* update as often as they can -but-
* aren’t totally dependent on their updating processes, ie have advanced heuristics
* have as little impact as possible on business processes: that is, they don’t slow my systems to a crawl when they update, or when they run a background scan, or insist on rebooting my system because they’ve just downloaded an engine update despite the fact that I’m in the middle of something critical.
I’m afraid your complaint is naive. It assumes that AV vendors should and can be better than they are. My view is that they’re better than most people outside the industry think they are. And if you want to focus on the things they don’t do optimally, which is fair enough, update frequency isn’t the place to start -unless- you understand the limitations on the industry and the technology. Sorry, but I’m not sure you do.