Tim - 7:24 am on Jan 31, 2007 (gmt 0) If I use Delete URL, do I also need to use Robots.txt?
Info on how this interacts with robots.txt is included in our help documentation: [help.yahoo.com...]
Yes. Once you've used Delete URL to remove a URL from our index, we recommend using robots.txt to exclude the content from being added to out indexed again. Robots.txt excludes Yahoo!'s crawler (Slurp) from re-indexing your content. A robots.txt exclusion rule reduces the number of pages Slurp will read from your server. It also helps you save bandwidth on your website.
Note: Site Explorer Delete URL service facilitates the quick removal of your URL from our index and is helpful when a URL that you have excluded, continues to show up as a thin document, but it may not prevent our crawler from continuing to crawl the URL again. It is not an alternative to using robots.txt.
If I use Delete URL, do I also need to use Robots.txt?