- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Why should I have a robots.txt file?
Glitzer - 4:41 pm on Jan 4, 2007 (gmt 0)
|Well-behaved robots, including those from major search engines, will not fetch a page if it is Disallowed in a properly-formatted robots.txt file. |
If they already have pages indexed or cached that have recently been disallowed in a revised robots.txt, then they won't remove those pages the SE's already indexed.
Quite a dilemma for webmasters.
Thread source: http://www.webmasterworld.com/robots_txt/3203372.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com