homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Problem with robots.txt

 10:50 pm on Nov 24, 2003 (gmt 0)

I have problems with my robots.txt. When I uploaded it, the server trafic died after ten minutes and I could not access any sites. Then I removed the file and the traffic was on again after 20min.

This is the content of my robots.txt:

User-agent: *
Disallow: redirect.php
Disallow: /shared

What's wrong? I don't was any spinder to access the script redirect.php or any file in the folder /shared...

Thanks for you help!



 11:17 pm on Nov 24, 2003 (gmt 0)

Have you checked your robot file here [searchengineworld.com] on this site

That should give you the answer.


 11:55 pm on Nov 24, 2003 (gmt 0)

I am not sure that I understand the possible mechanism.

Nobody but a robot should request a robots.txt file, so I do not see how such a file, however badly formed, could have any effect other than banning robots. Such an effect would not show on server traffic within ten minutes. There must be something else going on.

Did you modify the .htaccess file at the same time by any chance?


 8:08 am on Nov 25, 2003 (gmt 0)

Yes I checked the syntax with a validator script and No I didn't change .htaccess. It works perferctly on my friends server.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved