homepage Welcome to WebmasterWorld Guest from 107.22.37.143
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Problem with robots.txt
Nisse




msg:1527107
 10:50 pm on Nov 24, 2003 (gmt 0)

I have problems with my robots.txt. When I uploaded it, the server trafic died after ten minutes and I could not access any sites. Then I removed the file and the traffic was on again after 20min.

This is the content of my robots.txt:

User-agent: *
Disallow: redirect.php
Disallow: /shared

What's wrong? I don't was any spinder to access the script redirect.php or any file in the folder /shared...

Thanks for you help!
/Niels

 

Staffa




msg:1527108
 11:17 pm on Nov 24, 2003 (gmt 0)

Have you checked your robot file here [searchengineworld.com] on this site

That should give you the answer.

Mohamed_E




msg:1527109
 11:55 pm on Nov 24, 2003 (gmt 0)

I am not sure that I understand the possible mechanism.

Nobody but a robot should request a robots.txt file, so I do not see how such a file, however badly formed, could have any effect other than banning robots. Such an effect would not show on server traffic within ten minutes. There must be something else going on.

Did you modify the .htaccess file at the same time by any chance?

Nisse




msg:1527110
 8:08 am on Nov 25, 2003 (gmt 0)

Yes I checked the syntax with a validator script and No I didn't change .htaccess. It works perferctly on my friends server.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved