homepage Welcome to WebmasterWorld Guest from 54.237.125.89
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Problem with robots.txt
Nisse




msg:1527107
 10:50 pm on Nov 24, 2003 (gmt 0)

I have problems with my robots.txt. When I uploaded it, the server trafic died after ten minutes and I could not access any sites. Then I removed the file and the traffic was on again after 20min.

This is the content of my robots.txt:

User-agent: *
Disallow: redirect.php
Disallow: /shared

What's wrong? I don't was any spinder to access the script redirect.php or any file in the folder /shared...

Thanks for you help!
/Niels

 

Staffa




msg:1527108
 11:17 pm on Nov 24, 2003 (gmt 0)

Have you checked your robot file here [searchengineworld.com] on this site

That should give you the answer.

Mohamed_E




msg:1527109
 11:55 pm on Nov 24, 2003 (gmt 0)

I am not sure that I understand the possible mechanism.

Nobody but a robot should request a robots.txt file, so I do not see how such a file, however badly formed, could have any effect other than banning robots. Such an effect would not show on server traffic within ten minutes. There must be something else going on.

Did you modify the .htaccess file at the same time by any chance?

Nisse




msg:1527110
 8:08 am on Nov 25, 2003 (gmt 0)

Yes I checked the syntax with a validator script and No I didn't change .htaccess. It works perferctly on my friends server.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved