homepage Welcome to WebmasterWorld Guest from 23.23.12.202
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt file question?
What's this?
printlist

10+ Year Member



 
Msg#: 82 posted 11:42 am on Oct 9, 2002 (gmt 0)

I am reworking an old site and found a robots.txt file in the root directory. Contents are:
User-agent: *
Disallow: /admin/

I don't know what this means. I definitely want this site and all its filed crawled by google and other search engines.

Is this file preventing the crawl? Would deleting it altogether be better?

Thanks,
Steve

 

chiyo

WebmasterWorld Senior Member chiyo us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 82 posted 11:45 am on Oct 9, 2002 (gmt 0)

It is telling all spiders not to spider any files in the /admin/ directory only. From the sound of the name its probably a directory with files in it you dont want spidered! eg drafts, programs, caches etc.

korkus2000

WebmasterWorld Senior Member korkus2000 us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 82 posted 11:46 am on Oct 9, 2002 (gmt 0)

I think you want to keep crawlers out of your admin area. I would keep it. Here is a tutorial to help you.

[searchengineworld.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved