Welcome to WebmasterWorld Guest from 188.8.131.52 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe and Support WebmasterWorld
Problem with robots.txt Nisse msg:1527107 10:50 pm on Nov 24, 2003 (gmt 0) I have problems with my robots.txt. When I uploaded it, the server trafic died after ten minutes and I could not access any sites. Then I removed the file and the traffic was on again after 20min.
This is the content of my robots.txt:
Disallow: redirect.php Disallow: /shared
What's wrong? I don't was any spinder to access the script redirect.php or any file in the folder /shared...
Thanks for you help!
Staffa msg:1527108 11:17 pm on Nov 24, 2003 (gmt 0)
Have you checked your robot file here [ searchengineworld.com] on this site
That should give you the answer.
Mohamed_E msg:1527109 11:55 pm on Nov 24, 2003 (gmt 0)
I am not sure that I understand the possible mechanism.
Nobody but a robot should request a robots.txt file, so I do not see how such a file, however badly formed, could have any effect other than banning robots. Such an effect would not show on server traffic within ten minutes. There must be something else going on.
Did you modify the .htaccess file at the same time by any chance?
Nisse msg:1527110 8:08 am on Nov 25, 2003 (gmt 0)
Yes I checked the syntax with a validator script and No I didn't change .htaccess. It works perferctly on my friends server.