AardvarkBusiness.net - Business Search Engine AardvarkBusiness.net - Business Search Engine



 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister   ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

     

AVG8 skews all web stats badly...!

 
 
Post new topic   Reply to topic    AardvarkBusiness.net Forum Index -> Domains & Hosting Forum
View previous topic :: View next topic  
Author Message
trev
President
President


Joined: 15 Oct 2002
Posts: 3011
Location: NW England, UK
4037 ants

PostPosted: Sat Jun 14, 2008 11:47 pm    Post subject: AVG8 skews all web stats badly...! Reply with quote

I've been trying to track down why some of our sites are getting stupidly busy recently and thought maybe it was just either good rankings or lots of people wanted to see stuff. However it seems there might be another reason - AVG Anti Virus.

Essentially say a user searches for Buckingham Palace on Google. AVG will load up each of those usually 10 results as if they came from the users browser, and scan them for problems. This results in every link getting a page view and very possibly a "unique visitor" counted.

That's fine if the user actually visits the page, but most likely they'll click on say result 3, go off to that and never look at any others. The other sites have wasted bandwidth, server power on a non-human user, and they think they've got another unique to count. Multiply this up across larger popular sites with lots of pages and you start to really eat resources.

Considering there are around 70 million AVG users in the world and only 20 million of them have upgraded so far, this isn't looking good. The only way currently to spot this is to look for the user agent:

Quote:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813


It's the 1813 bit that gives the game away currently, but that could change. Assuming this isn't being used by other malware yet, we have a site getting around 80,000 uniques a month. I checked the logs and since the 8th June, we've been hit by 95322 pre-fetch checks from that user agent. Over 12000 just today...!

The register on AVG8


For anyone with root access to the server, you can check the logs by:
(replace example.com with the web site domain, no www.)

Code:
cd /usr/local/apache/domlogs/
grep -c "Windows NT 5.1;1813" example.com


Code:
cd /usr/local/apache/domlogs/
grep -Pc "14/Jun/2008.+Windows NT 5.1;1813" example.com


First one checks all the entries in the log, the second checks the 14th's entries. This probably only works on Apache, but should be tweakable for other log formats.

Trev
_________________
Mad Theories - Ambleside - Coniston - Grange
Back to top
View user's profile Send private message Visit poster's website
pinyo
Executive PA
Executive PA


Joined: 05 Aug 2004
Posts: 77

94 ants

PostPosted: Mon Jun 16, 2008 8:09 pm    Post subject: Reply with quote

Couldn't you identify AVG IP address and block it?
_________________
Moolanomy
Personal Finance Buzz
Back to top
View user's profile Send private message Visit poster's website
trev
President
President


Joined: 15 Oct 2002
Posts: 3011
Location: NW England, UK
4037 ants

PostPosted: Mon Jun 16, 2008 8:24 pm    Post subject: Reply with quote

Nope as the IP address is that of the user running the AVG software and downloading the search engine results. It's all run from each users copy of AVG and since it doesn't report back to a central place there's no whitelisting which would at least improve things a little.

Even worse it appears it doesn't do any caching, so for example a user clicks a link then goes back to Google, the software rescans all the links. At least those don't appear as uniques, but they still take up bandwidth and processor time.

Trev
_________________
Mad Theories - Ambleside - Coniston - Grange
Back to top
View user's profile Send private message Visit poster's website
trev
President
President


Joined: 15 Oct 2002
Posts: 3011
Location: NW England, UK
4037 ants

PostPosted: Wed Jul 09, 2008 3:56 pm    Post subject: Reply with quote

From what I've heard, AVG have now amended this system so it'll stop scanning everything and only scan the page you're looking at - like everyone told them to. Still don't know why they can't run it as a proxy as has been suggested a few times.

Trev
_________________
Mad Theories - Ambleside - Coniston - Grange
Back to top
View user's profile Send private message Visit poster's website
SeoAddict
Secretary
Secretary


Joined: 10 Aug 2008
Posts: 7

10 ants

PostPosted: Sun Aug 10, 2008 3:06 pm    Post subject: avg 8 Reply with quote

I think AVG 8 checks up if a site is safe to look at or not.
it definitely uses up some system memory so it's up to you if you want
to use AVG 8.0 or not...
_________________
Make Money Online
Free Small Businesses Network Tools || Squidoo Lens
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    AardvarkBusiness.net Forum Index -> Domains & Hosting Forum All times are GMT + 1 Hour
Page 1 of 1

 
Google
 
Business Forum Sport Forum Travel Forum


Powered by php B B © 2001, 2002 php B B Group

AardvarkBusiness.net Business Search Engine & Directory