Case sensitive URLs...

Typed, .htaccess and access log files and 302s.

[pendanticist]

Recently, I've had some mighty peculiar log entries, most of which deal with case sensitivity. Let me explain.

RootURL/multicultural.html (note 'm')

RootURL/Multicultural.html (note 'M')

If I type in 'multicultural' (lower case) after my root URL I get a 404 (my homemade missing.html) error code, just about what I'd expect myself and everyone else to get.

However, in my access log files a visitor:

xx.xxx.x.xx - - [14/Nov/2002:09:36:41 -0800] "GET /multicultural.html HTTP/1.1" 302 242 "-" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT)" (note the lower case 'm')

Now, if I'm reading this correctly, 302 codes indicate 'found' which tells me the user in question actually 'visited' the page he/she requested.

The user (noted above) systematically visited each of 160 other indices/files within my domain and judging by the time stamps, my access log files indicate each request was honored.

Last year I changed all my internal URLs and modified my .htaccess file accordingly. All old URL/index/file requests (except for those requested by good ole Slurp/cat) have been successfully redirected to the new URL/index/file.

Will I now have to modify my .htaccess file to reflect this case problem? If so, how hefty can that file be before the good bots choke on it?

Also, and I don't know if this is related, but I've had EmailSiphon banned for sometime now and basically, within this same time frame I've seen EmailSiphon's requests also being honored (200 code).

xxx.xxx.x.xx - - [14/Nov/2002:11:38:42 -0800] "GET /Calendars.html HTTP/1.1" 200 3062 "-" "EmailSiphon"
xxx.xxx.x.xx - - [14/Nov/2002:11:38:45 -0800] "GET /Classics_A-H.html HTTP/1.1" 200 13828 "-" "EmailSiphon"

What could the problem be?

I should point out that I (just today) modified EmailSiphon's ban from:

RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]

to:

RewriteCond %{HTTP_USER_AGENT} EmailSiphon [NC,OR]

in the hope of correcting whatever the malady might have been that allowed EmailSiphon the run of my site.

Thanks.

[jdMorgan]

pedanticist,

What is the code you are using to do the 404 redirects? An error there can cause a 302-Moved Temporarily to be returned rather than a 404-Not Found. I assume you're on Apache, so it should be:

ErrorDocument 404 /404.html

It might also be the case that one of your RewriteRule sets is redirecting that access.

The EmailSiphon block looks fine as you had it originally. Are you sure there is not another problem, such as a missing [OR] on another RewriteCond in that set? Also, your RewriteRule might have a problem.

Jim

[pendanticist]

Jim,

Thanks for responding.

Yes - Apache.

My missing.html file is a hand crafted 404 page and I have only the very basic understanding of the specifics of this coding.

Perhaps the best way to answer simply, albiet a tad lengthy is to show you...
____________________________________________________

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
AuthName (Deleted)
AuthUserFile (Deleted)
AuthGroupFile (Deleted)
ErrorDocument 404 www.blah-blah.com/missing.html
# Send a permanent redirect from our old file to our new file
Redirect permanent /1ABSITEMAP.html www.blah-blah.com/About_Site_Map.html
Redirect permanent /1Translate.html www.blah-blah.com/About_Translate.html
Yada
Yada
Yada
Redirect permanent /1AB_Publish.html www.blah-blah/.com/About_Published.html
RewriteEngine On
# RewriteCond %{HTTP_USER_AGENT} ^Mozilla* [OR]
RewriteCond %{HTTP_USER_AGENT} ^almaden [OR]
RewriteCond %{HTTP_USER_AGENT} ^Anarchie [OR]
RewriteCond %{HTTP_USER_AGENT} ^ASPSeek [OR]
RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^BatchFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:[e-mail] [OR]
RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR]
RewriteCond %{HTTP_USER_AGENT} ^bumblebee [OR]
RewriteCond %{HTTP_USER_AGENT} copier [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^CICC [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^DA [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Wonder [OR]
RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^Drip [OR]
RewriteCond %{HTTP_USER_AGENT} ^DSurf15a [OR]
RewriteCond %{HTTP_USER_AGENT} EasyDL [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EasyDL/2.99 [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} EmailSiphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} FrontPage [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetSmart [OR]
RewriteCond %{HTTP_USER_AGENT} ^gigabaz [OR]
RewriteCond %{HTTP_USER_AGENT} ^gotit [OR]
RewriteCond %{HTTP_USER_AGENT} grab [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} grub-client [NC,OR]
RewriteCond %{HTTP_USER_AGENT} .*httrack.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} ^httpdown [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Indy.Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Explore\ 5\.x [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetLinkagent [OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetSeer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Iria [OR]
RewriteCond %{HTTP_USER_AGENT} ^JBH*agent [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^JustView [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^lftp [OR]
RewriteCond %{HTTP_USER_AGENT} ^likse [OR]
RewriteCond %{HTTP_USER_AGENT} ^Link*Sleuth [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Memo [OR]
RewriteCond %{HTTP_USER_AGENT} ^microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla*MSIECrawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSIECrawler [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSProxy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetMechanic [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} offline [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
RewriteCond %{HTTP_USER_AGENT} ^Openfind [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ping [OR]
RewriteCond %{HTTP_USER_AGENT} ^PingALink [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^QRVA [OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} spider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Scooter [OR]
RewriteCond %{HTTP_USER_AGENT} ^Seeker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SlySearch [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snake [OR]
RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR]
RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Szukacz [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vacuum [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website [OR]
RewriteCond %{HTTP_USER_AGENT} ^Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebHook [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebMiner [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebMirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Whacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^x-Tractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xenu [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus [OR]
____________________________________________________

Other than having reduced redundancy in the 'Redirect permanent' area, this coding/text (including spaces, line breaks, slashes, etc.) is just as it appears within my .htaccess file.

I did not modify anything in the 'RewriteCond' area.

Don't know how long I'll be able to stick around...been a loooonnnnnnng day.

Thanks again :-)

Pendanticist.

[jdMorgan]

Pedanticist,

OK, your 302-instead-of-expected-404 problem is caused by your ErrorDocument statement. Do not use a URL, use only the local path, i.e.

ErrorDocument 404 /missing.html 

as shown in my first post. See the 5th paragraph "Note that when..." in the ErrorDocument directive [httpd.apache.org] section of the Apache Core Features documentation for clarification.

Now I'll study your RewriteConds to try to find the other problem... :)

-later-

Hey! Where's the RewriteRule that goes at the end? Can't tell much without that...

Jim

[pendanticist]

Morning Jim,

ErrorDocument 404 www.blah-blah.com/missing.html has been changed to - ErrorDocument 404 /missing.html

Regarding "Hey! Where's the RewriteRule that goes at the end? Can't tell much without that..."

I gleaned all that you see from posts within WebmasterWorld and at no time did I actually 'see' an ending RewriteRule. Like I said, everything you see (other than private information deleted on my part) is exactly as it appeared in my .htaccess file.

I'm guessing that RewriteEngine Off would be that closing line?

Ignorance really is bliss you know. <chuckle>

Thanks again.

Pendanticist.

[jdMorgan]

Pedanticist,

No RewriteRule! :o

Adding:

.
.
.
RewriteCond %{HTTP_USER_AGENT} ^Xenu [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus [OR]
RewriteRule .* - [F]

Will work nicely.

Or, if you also have a custom 403 page, you could use:

.
.
.
RewriteCond %{HTTP_USER_AGENT} ^Xenu [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus [OR]
RewriteRule !^custom403\.html$ - [F]

This is all you should need to fix the probem. A RewriteEngine off directive is not needed.

See the handy Apache mod_rewrite bliss-reduction guide [httpd.apache.org]. ;)

Before modifying your RewriteConds, take a look in there at the short-form regular expressions guide table. It will prevent problems to know that "^.*xyz.*$" is the same as "xyz" and that "file*" does not mean "'file' followed by anything" the way it does in DOS, for example. I see a few non-optimal RewriteConds in your list, e.g.

RewriteCond %{HTTP_USER_AGENT} .*httrack.* [NC,OR]

Which could just as well be expressed as

RewriteCond %{HTTP_USER_AGENT} httrack [NC,OR]

The quick-guide will help to prevent major disasters caused by minor errors in regular-expressions-based rewrite rules.

I wax pedantic, so I'll close - That should take care of EmailSiphon and the others for you!

Jim

[pendanticist]

Hi Jim,

>No RewriteRule!
Well, now that my memory is sufficiently shaken I recall I did have a line in there but took it out some weeks back. I inadvertantly was directing all my traffic to the url contained in that last line. Freaked me right out.

:-¦

>Adding:
>.
>.
>.
>RewriteCond %{HTTP_USER_AGENT} ^Xenu [OR]
>RewriteCond %{HTTP_USER_AGENT} ^Zeus [OR]
>RewriteRule .* - [F]

>Will work nicely.

Great! :-)

>Or, if you also have a custom 403 page, you could use:
Now that has some appeal. Sometime back, I customized a 404.html file to something more pleasant and apt to retain the visitor longer. Much more suitable than those drab browser specific 404 pages. So, I assume the fundamentals are the same, create some kinda "Nah!" page only call it 403.html.

>.
>.
>.
>RewriteCond %{HTTP_USER_AGENT} ^Xenu [OR]
>RewriteCond %{HTTP_USER_AGENT} ^Zeus [OR]
>RewriteRule!^custom403\.html$ - [F]

And at this very point here ^ is what redirects to that custom page. Cool.

>This is all you should need to fix the probem. A >RewriteEngine off directive is not needed.

Got it.

>See the handy Apache mod_rewrite bliss-reduction guide.

I briefly checked that out. Definately gotta spend some time in there.

>Before modifying your RewriteConds, take a look in there >at the short-form regular expressions guide table. It will >prevent problems to know that "^.*xyz.*$" is the same >as "xyz" and that "file*" does not mean "'file' followed >by anything" the way it does in DOS, for example. I see a >few non-optimal RewriteConds in your list, e.g.

My knowledge of DOS is nill, so I can't draw the comparison, but I do know what you are referring to. Precise spacing and placements of specific characters critical to effective, uh, operation of the coding.

If it's not so far in depth, are those few you mentioned easily repaired?

>RewriteCond %{HTTP_USER_AGENT} .*httrack.* [NC,OR]

>Which could just as well be expressed as

>RewriteCond %{HTTP_USER_AGENT} httrack [NC,OR]

I see. Less is better.

>The quick-guide will help to prevent major disasters >caused by minor errors in regular-expressions-based >rewrite rules.

I think I can relate to that. All my SE bots travel thru my files well, except for Inktomi. That thing has yet to properly grab the new location of previously re-directed files. Nooooooobody else has a problem but them. <shrug> If it weren't for my plate being so full at the moment I would immerse myself in these subtleties.

>I wax pedantic, so I'll close - That should take care of >EmailSiphon and the others for you!
Great :-) EmailSipon has been hitting me hard this last two days. This is the first they've ever bothered me.

Anyway, I can't Thank You enough for the great solutions.

>Jim

PeNdanticist ;-) (I misregistered the added n and never changed it.)

[jdMorgan]

Pendanticist,

Well, now that my memory is sufficiently shaken I recall I did have a line in there but took it out some weeks back. I inadvertantly was directing all my traffic to the url contained in that last line. Freaked me right out.

This problem was probably caused by that commented-out line at the top:

# RewriteCond %{HTTP_USER_AGENT} ^Mozilla* [OR]

That would have blocked almost all browsers, because the UA usually starts with "Mozilla/x.xx "

You might want to add a few:
^Mozzilla [NC,OR] (note that it's misspelled)
e.?mail [NC,OR] ...(intentionally unanchored, will catch anything with "e-mail, email, e_mail, etc. in it)
Offline [NC,OR] ....(also unanchored, and allows you to remove several redundant entries)

Check the Search Engine Spider forum [webmasterworld.com] occasionally for the latest rogue User-agent discussion.

You might want to sticky-mail a forum moderator and ask him/her to correct your screen name.

Glad to assist!
Jim

[pendanticist]

Thanks again Jim,

Sorry for the delay in getting back. Working on my Internship for school and time is not all my own.

I'll tell you, I had a heckuva time with this whole thing until I re-read the "Perfect .htaccess ban list" thread again. Wasn't until I got to some of your older posts that I saw what was fouling me up w/403 and 500 Internal Server errors.

.
.
.
RewriteCond %{HTTP_USER_AGENT} ^x-Tractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xenu [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus [OR]

I was leaving the last (Zues) [OR] on.

Then, your closing RewriteRule .* - [F] worked. I altered in accordance with those previous posts too by adding ,L within the brackets.

All seems more copasetic now.

Thanks again. I certainly appreciate it :-)

Pendanticist.

[jdMorgan]

pendanticist,

closing RewriteRule .* - [F] worked. I altered in accordance with those previous posts too by adding ,L within the brackets.

Ooops! My bad... An [L] flag is redundant with either an [F] or a [G] flag. You shouldn't need to add it.

Jim

[pendanticist]

That's ok.

In an effort to ban those IPs that have been requesting formmails, it seems that I read somewhere that all one has to do in order to ban any visitor is to put their IP number:
____________________________________________________

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
deny from all
...IN HERE...

allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
____________________________________________________

Yet, when I did enter the first one, got an Intenal Server Error.

What am I doing wrong? Is it my recollection?

Regards,
Pendanticist.

[pendanticist]

This is getting old now. Checking my access log files a bit ago I discovered this:

64.68.82.66 - - [19/Nov/2002:04:14:23 -0800] "GET /robots.txt HTTP/1.0" 403 208 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

If I'm reading this correctly, Googlebot just got slapped with a 403 error code?

What did I do to deserve this?

I've removed the bot [NC,OR] to see what happens, but I did not intentionally seek to ban Goolebot.

Any suggestions?

Pendanticist.