Spider referer info

[rpking]

Do 100% of spiders have no referer info? My script currently ignores any request that has any referer info, but over the weekend it missed a Scooter request.

Thing is, that IP is on my list and the script seems to function fine, so I was wondering whether this cheeky Scooter may have had a none-empty referer field.

[littleman]

Not 100%, but I would say that it is 100% for the ones that matter. There are a few 'harvesting' type of bots that use a referrer - but none of the bots that you want to see your cloaked pages do.

[Brett_Tabke]

rpking, that is actually a very good trick. I can't recall that ever occuring to me to test for a referrer - thank you.

[cirelle]

I've been (for the most part) basing my criteria on the ip to determine who's who in the zoo with regard to se's. Is that wrong?

c

[Air]

Welcome to WmW cirelle.

Nothing wrong with using IP's to identify spiders. Most people that use auxillary methods still use them in conjunction with IP addresses.

[cirelle]

Thanx Air. Glad to be here

c

[Brett_Tabke]

There are so many different spiders running around now, that it is very handy to look for universal methods of identification. Althought the newer spiders work with http 1.1 and can look just like a browser, there are other methods of identification beyond ip/agent.