Welcome to WebmasterWorld Guest from 54.226.110.143

Forum Moderators: goodroi

Message Too Old, No Replies

Problem with robots.txt

     
10:50 pm on Nov 24, 2003 (gmt 0)

New User

10+ Year Member

joined:July 2, 2003
posts:13
votes: 0


I have problems with my robots.txt. When I uploaded it, the server trafic died after ten minutes and I could not access any sites. Then I removed the file and the traffic was on again after 20min.

This is the content of my robots.txt:

User-agent: *
Disallow: redirect.php
Disallow: /shared

What's wrong? I don't was any spinder to access the script redirect.php or any file in the folder /shared...

Thanks for you help!
/Niels

11:17 pm on Nov 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 24, 2002
posts:894
votes: 0


Have you checked your robot file here [searchengineworld.com] on this site

That should give you the answer.

11:55 pm on Nov 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 14, 2002
posts:1192
votes: 0


I am not sure that I understand the possible mechanism.

Nobody but a robot should request a robots.txt file, so I do not see how such a file, however badly formed, could have any effect other than banning robots. Such an effect would not show on server traffic within ten minutes. There must be something else going on.

Did you modify the .htaccess file at the same time by any chance?

8:08 am on Nov 25, 2003 (gmt 0)

New User

10+ Year Member

joined:July 2, 2003
posts:13
votes: 0


Yes I checked the syntax with a validator script and No I didn't change .htaccess. It works perferctly on my friends server.