homepage Welcome to WebmasterWorld Guest from 54.196.199.46
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt
webboy1




msg:1527530
 10:40 am on Jul 22, 2002 (gmt 0)

Hi,

I think this will work, but i thought i would get a few more opinions to be sure.......

Robots.txt are used to stop SE's from searching through pages you do not wish them to see.
Does this mean that if a site has been updated, and some old pages no longer exist.........in theory i could put these old links in my robots.txt and they would eventually cancel themselves from view on the SE's?

Am i thinking correct? or do robots not work like this?

regards
webboy

 

engine




msg:1527531
 10:43 am on Jul 22, 2002 (gmt 0)

You could do that, however, the SE should record the 404 and update their DB. Probably not worth the effort to update the robots.txt for a few files, IMHO. Worth it for a whole directory on your site.

The problem is not the robots.txt file, but the fact that many SEs don't update their database often enough, resulting in stale SERPs.

[edited by: engine at 10:45 am (utc) on July 22, 2002]

incywincy




msg:1527532
 10:45 am on Jul 22, 2002 (gmt 0)

hi webboy,

personally i wouldn't do that because over time your robots.txt file would become rather large and unwieldly. just let the spiders generate 404s and they'll get the message that the pages have gone.

as far as i know there are no detrimental effects when doing this.

webboy1




msg:1527533
 10:48 am on Jul 22, 2002 (gmt 0)

cheers guys.

webboy

Brett_Tabke




msg:1527534
 12:04 pm on Jul 22, 2002 (gmt 0)

Agreed, most bots have a tougher time going back and purging pages that were once available than they do purging a 404 page.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved