boitho.com bot violating robots.txt

Specifically requested only forbidden files

[jazzguy]

"boitho.com-dc/0.75 ( http*//www.boitho.com/dcbot.html )" came from 129.241.104.168. It specifically targetted disallowed files from robots.txt, ignoring all other pages.

The info page says it's a distributed crawler, so just like my policy for the cronic robots.txt violater Grub, I banned the user agent and the entire IP block associated with the offending IP.

[jazzguy]

Ohhhhhh, my bad, I just found robots.txt he was referring to all th etime in the beginning of the thread, reposting it here

<snip ridiculous bogus robots file>

A perfect example of the sarcasm that resulted in my rescinding the offer to help you.

[Lord Majestic]

A perfect example of the sarcasm

Thanks dude, you sure can make other people feel appreciated!

that resulted in my rescinding the offer to help you.

I was trying to get you to help me help you, however it is now clear that you need the kind of help that is beyond my field of expertise.

I suggest you refer to post #2 in this thread where I asked for robots.txt rather politely:

Just out of interest, can you post your robots.txt?

100 posts and counting -- and no robots.txt yet.

[edited by: Lord_Majestic at 8:25 pm (utc) on June 14, 2005]

[rj87uk]

I will say sorry for him(is it even needed?) and also say that he didnt mean to reject your multiple offers.

[bcolflesh]

A perfect example of the sarcasm...

It's not sarcasm to hold out your hand to a friend in need - robots.txt mistakes can cause spiders to improperly interact with your site - there are tons of good validators that can clear up these issues - check 'em out!

[jazzguy]

if this is indeed the contents of your robots.txt file, it is not valid

Either you're a complete idiot or you're still trolling. If the latter, I guess you can carry on as you see fit. If the former, that's something you'll have to work out for yourself.

Oh and in case it was the former, of course it's not the contents of my robots.txt file.

[jazzguy]

I think he may be on to something... worth looking into I would say.

See above

[bcolflesh]

...of course it's not the contents of my robots.txt file

Doh! Sorry about the mixup friend - can you post the contents of your actual robots.txt file here?

[jazzguy]

Doh! Sorry about the mixup friend - can you post the contents of your actual robots.txt file here?

Troll,

Already covered multiple times in this thread.

[Lord Majestic]

Oh and in case it was the former, of course it's not the contents of my robots.txt file.

Hey, it was a fair guess given the circumstances! :)

At least now when you send people away to check thread for that mystical robots.txt, something remotely relevant to it can be found.

If I made a few people smile today then posting here was not a complete waste of time.

[rj87uk]

I thought Trolls only make one post then leave again? Hes been here for a while now asking for the 'robots.txt' file so we can all help you!

Ps. Do you really have one?