Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Problem with robots.txt



10:50 pm on Nov 24, 2003 (gmt 0)

10+ Year Member

I have problems with my robots.txt. When I uploaded it, the server trafic died after ten minutes and I could not access any sites. Then I removed the file and the traffic was on again after 20min.

This is the content of my robots.txt:

User-agent: *
Disallow: redirect.php
Disallow: /shared

What's wrong? I don't was any spinder to access the script redirect.php or any file in the folder /shared...

Thanks for you help!


11:17 pm on Nov 24, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Have you checked your robot file here [searchengineworld.com] on this site

That should give you the answer.


11:55 pm on Nov 24, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I am not sure that I understand the possible mechanism.

Nobody but a robot should request a robots.txt file, so I do not see how such a file, however badly formed, could have any effect other than banning robots. Such an effect would not show on server traffic within ten minutes. There must be something else going on.

Did you modify the .htaccess file at the same time by any chance?


8:08 am on Nov 25, 2003 (gmt 0)

10+ Year Member

Yes I checked the syntax with a validator script and No I didn't change .htaccess. It works perferctly on my friends server.

Featured Threads

Hot Threads This Week

Hot Threads This Month