So I got my Spider filter working, an what did I see

1 spiders for every 5 humans is what ! , page views are even worse

[vite_rts]

Hi All

So, My site are very low volume, total off 50 uniques per day on average over 2 active sites

According to my spider filter , which is unhappily confirmed by webtrends from my ISP an google analytics

up to 10 of my uniques would be spiders an up to 40% of page views are triggered by these spiders

Is this what others are seening?

Does google bot need 67 page views in 1 day to index just 15 pages or might I be doing something wrong that makes spiders unable to index my site efficiently,

Don't get me wrong, google bot is welcome, I am just concerned that I am not making life difficult for the cute little spiders :-)

[DamonHD]

Hi,

I reckon that 90%+ of my traffic is bots, not necessarily good or useful ones.

I try to eliminate them from my user count and try to make sure that it is cheap to serve them pages.

Rgds

Damon

[vite_rts]

did you mean 90% or 9%,

you surely must be a chap with uniques/day figures in the 100s or 1,000's

90% of that traffic level is is,,

[DamonHD]

Hi,

90 not 9.

More than 9 hits in 10 on every mirror or my main site are spiders/bots in my estimate.

So I try to make sure that they end up consuming much less than 90% of bandwidth, else I could not possibly justify rolling out small mirrors closer to my users: they'd simply be swamped with useless bot traffic.

I'm getting uniques/day in the thousands, yes.

Rgds

Damon

[waziwazo]

I have a small site too and i get about 150 visitors/day
About 65 real visitors and the 85 others are bots or spiders of all sort.

Bots and spyders usualy use less bandwidth, 20% in my case when they are nearly 60% of my visitors.

You have to look carefully at your log to find all the spiders and bot. Analysis program would alway miss a few. If a visitor ask for your robots.txt files or if you see the command HEAD in your log file("HEAD /index.html HTTP/1.1") this visitor is very likely a bot.

Sorting bots an real visitors is very important for marketting purpose. Before removing bots ~50% of my visitors are from USA and after removing this fall to about 4%. Most bots are from USA :)

[trillianjedi]

I'm running at about 70%-80% bots for most sites, as a guestimate. It could be higher.

The majority of them are, as Damon says, quite useless. In fact there are only 3 bots that are technically "allowed in". The rest are ignoring my robots.txt directive.

It is quite high on my agenda to fix this issue. Next thing on my list, in fact.

TJ

[DamonHD]

Hi,

Indeed, the spider traffic is so problematical that I have had to really cut down the pages returned to bots. It isn't intended as "cloaking" per se, and I'm happy for a user or a bot to see the "bot" ("lite") version of a page (indeed, the "lite" version is what will appear in G's cache for example), but I do worry that I will get slapped by the SEs at some point.

Rgds

Damon