Welcome to WebmasterWorld Guest from 54.144.107.83

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt

     
10:40 am on Jul 22, 2002 (gmt 0)

Full Member

10+ Year Member

joined:Mar 15, 2002
posts:346
votes: 0


Hi,

I think this will work, but i thought i would get a few more opinions to be sure.......

Robots.txt are used to stop SE's from searching through pages you do not wish them to see.
Does this mean that if a site has been updated, and some old pages no longer exist.........in theory i could put these old links in my robots.txt and they would eventually cancel themselves from view on the SE's?

Am i thinking correct? or do robots not work like this?

regards
webboy

10:43 am on July 22, 2002 (gmt 0)

Administrator from GB 

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:May 9, 2000
posts:22282
votes: 236


You could do that, however, the SE should record the 404 and update their DB. Probably not worth the effort to update the robots.txt for a few files, IMHO. Worth it for a whole directory on your site.

The problem is not the robots.txt file, but the fact that many SEs don't update their database often enough, resulting in stale SERPs.

[edited by: engine at 10:45 am (utc) on July 22, 2002]

10:45 am on July 22, 2002 (gmt 0)

Full Member

10+ Year Member

joined:Feb 5, 2002
posts:333
votes: 0


hi webboy,

personally i wouldn't do that because over time your robots.txt file would become rather large and unwieldly. just let the spiders generate 404s and they'll get the message that the pages have gone.

as far as i know there are no detrimental effects when doing this.

10:48 am on July 22, 2002 (gmt 0)

Full Member

10+ Year Member

joined:Mar 15, 2002
posts:346
votes: 0


cheers guys.

webboy

12:04 pm on July 22, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


Agreed, most bots have a tougher time going back and purging pages that were once available than they do purging a 404 page.