Just got a logfile analyser....

Yeah, I know "about time TJ" but confused...

[trillianjedi]

Got a couple of questions I'd be really grateful if you guys could help me with:-

1. I can see that we're giving out a lot of "forbidden" messages for requests for our robots.txt. I'm not sure why? Is there something simple I've missed?

2. I see that google has been freshbotting us quite a bit, last go about 6 or 7 days ago, but those results don't seem to appear in any of the datacentres. Any ideas?

3. Who are seventwentyfour dot com?

Many thanks,

TJ

[pixel_juice]

>>lot of "forbidden" messages for requests for our robots.txt

That would worry me. Over 20/30 sites i've never had forbidden for robots.txt

>>seventwentyfour dot com

Send a bot out looking for broken links so they can send you spam regarding their commercial services (sorry 7-24 but that's sure the way it looks to me...)

[trillianjedi]

Can anyone help? I'm the most concerned about this robots.txt issue - I really would like at least the good bots to be able to read this. At the moment, there seems to be a priviledge issue?

Thanks,

TJ

[cornwall]

>>Who are seventwentyfour dot com?

That is the easiest one to answer. As pixel_juice says they look for broken links.

If you do a Google search for seventwentyfour dot com you can get the full story

[sugarkane]

> robots.txt

I assume you can access the file okay yourself via a browser?

If so, have you got any User Agent banning or similar going on (eg spambot blocking) which might be causing the problem?

[trillianjedi]

Thanks sugarkane.

I feel completely stupid, but didn't check it with a browser whilst logged in to the site *as a guest*!

D'oh!

Your comment made me think about it - and then got this error.

Problem is now resolved - thanks for your help.

TJ