Forum Moderators: open

Message Too Old, No Replies

Multiple sites showing similar crawl spikes

         

DiscoStu

8:06 pm on Feb 12, 2010 (gmt 0)

10+ Year Member



This is more of an observation than a question, but I wanted to see if someone had a comment on it (or finds it interesting at all). I have a bunch of sites that are part of a thinly disguised network (= they're not officially part of a network, but we don't go to any lenghts to hide it i.e. they link randomly to each other etc). When checking the crawl stats I realized that those sites almost identical patterns of crawl spikes. When I checked a site that had nothing to do with that network the spike patter looked completely different.

Anyone care to venture a guess or comment on to how this works exactly?

dstiles

12:17 am on Feb 13, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Scrapers following links between sites?

For "scrapers" read "anything that's not a real bot", including home-based accelerators, distributed bots, exploiters trying to dump virus code on you... Pretty much anything, really. :(

One of the worst I find are super-fast broadband connections with some idiot who's just found out about the latest firefox add-in for downloading sites. Several hundred pages from a site, on to next site.

Only solution: kill them!

DiscoStu

10:09 pm on Mar 10, 2010 (gmt 0)

10+ Year Member



The crawl rates were checked from googl webmaster tools, so the stats are all from googlebot...

keyplyr

1:52 am on Mar 11, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



@DiscoStu

Consider using a more specific log analysis program to identify exactly what user agents requested what and when from your server. Then always do a manual verification in your raw server logs.

The reports freely offered at Google Webmaster Tools have their uses, but IMO if you're serious about online business management, you need better tools than that.