homepage Welcome to WebmasterWorld Guest from 54.204.73.126
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Problem with robots.txt
Nisse

10+ Year Member



 
Msg#: 205 posted 10:50 pm on Nov 24, 2003 (gmt 0)

I have problems with my robots.txt. When I uploaded it, the server trafic died after ten minutes and I could not access any sites. Then I removed the file and the traffic was on again after 20min.

This is the content of my robots.txt:

User-agent: *
Disallow: redirect.php
Disallow: /shared

What's wrong? I don't was any spinder to access the script redirect.php or any file in the folder /shared...

Thanks for you help!
/Niels

 

Staffa

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 205 posted 11:17 pm on Nov 24, 2003 (gmt 0)

Have you checked your robot file here [searchengineworld.com] on this site

That should give you the answer.

Mohamed_E

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 205 posted 11:55 pm on Nov 24, 2003 (gmt 0)

I am not sure that I understand the possible mechanism.

Nobody but a robot should request a robots.txt file, so I do not see how such a file, however badly formed, could have any effect other than banning robots. Such an effect would not show on server traffic within ten minutes. There must be something else going on.

Did you modify the .htaccess file at the same time by any chance?

Nisse

10+ Year Member



 
Msg#: 205 posted 8:08 am on Nov 25, 2003 (gmt 0)

Yes I checked the syntax with a validator script and No I didn't change .htaccess. It works perferctly on my friends server.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved