Forum Moderators: open
161.65-104.adsl.ij.net - - [08/Apr/2003:04:54:46 -0700] "GET /Internet_Stuff.html HTTP/1.0" 200 11864 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
Does this look like a spoofed googlebot?
I looked for that Googlebot IP Number post which was floating around here yesterday , but can't find it this morning.
Pendanticist.
Pendanticist.
I think it was moved to the spider identification forum.
<Didn't realize Google had soooo many of them.>
Thanks.
Pendanticist.
# Block all user-agents identifying as Googlebot/ from non-Google IP address ranges
RewriteCond %{REMOTE_ADDR} !^64\.68\.82\.
RewriteCond %{REMOTE_ADDR} !^216\.239\.46\.
RewriteCond %{HTTP_USER_AGENT} ^Googlebot/
RewriteRule .* - [F]
The above can also be done with SetEnvIf and deny from, but I'll need more coffee first, since mod_rewrite is my native language and I only took mod_access as a second language...
Warning: When Google adds a new IP address range, you must add it to the code above!
Jim
now if everyone changed their mozilla useragent tags to GoogleBot then that sure would confused the **** out of you paranoid bunch!
Paranoia has nothing to do with it vincevincevince.
See, there are those webmasters (such as myself) who garner no income from their work, rather they do it for the 'altruistic glow' when they sees individuals (as well as major institutions) perusing & enjoying your work. Getting some value out of it...so to speak.
Then, you have those timid folks who, for whatever reason, feel they need to lie to folks like me by falsifying their UA. As in life, when anyone falsly represents themselves to me, I am concerned.
I am concerned because they've already proven themselves unworthy of using the material I so freely provide them by 'hiding'.
Hence, when those spoofed UAs come up, it is up to us (as responsible webmasters) to ensure those visitors don't harvest or otherwise steal our work.
Hell, I had a student (from an .edu) rape my content overnight taking as many as 13 files per second! Eating up the bandwidth which I pay.
Paraniod? Nah. Protective is a much better word.
Pendanticist.
There are some webservices, though I have forgotten the urls, that spoof the useragent. They are called like "see exactly what search engines see". I've tried these myself, so some of you can have googlebot in your logs that are me :)
But these services are useless anyway. No pro cloaker, cloaks just by the user-agent.
Each webmaster has the right to determine who can and cannot access his/her site, and for what purposes. Once you've been site-jacked or had your server brought to its knees by a DOS attack, maybe you'll understand. At any rate, name-calling and ad-hominem attacks serve no purpose here on WebmasterWorld.
Jim
"there are those webmasters (such as myself) who garner no income from their work, rather they do it for the 'altruistic glow' when they sees individuals (as well as major institutions) perusing & enjoying your work. Getting some value out of it"
Nice choice of words:
As for being paranoid, try tailing your log file for a couple of hours watching the same IP number taking 1,000's of pages and after you have stopped head butting the screen trying to stop the text shooting up the screen faster than you can read it, then you would add the UA or IP number to your ban list.
I don't think the word paranoid is the correct word to use, you are just protecting what you have spent months or years building.
And as for being paranoid this guy [webmasterworld.com] pulled down more than 50,000 pages from my site yesterday (only have about 1,500) and sucked up more than a gig of bandwidth. It's my dime he's playing with!
Jim
