Msg#: 3046318 posted 7:15 pm on Aug 16, 2006 (gmt 0)
Suppose eBay or NY Times completely forbade search engines from crawling. For a query such as [ebay] or [ny times], the best result would still be the url reference (i.e. the uncrawled url without title and snippet) because those are navigational queries. That was the situation with both those sites 4-5 years ago, by the way. :)
When we can't crawl a site, we still try to find e.g. descriptions from the ODP to make the snippet prettier instead of completely empty. And if we think that's the best result, then we can show that url reference even if we haven't crawled the site.
Msg#: 3046318 posted 9:14 pm on Aug 16, 2006 (gmt 0)
3 months ago I blocked MSN in robots.txt couse he eats my bandwidth. I ranked first for a keyword contained in my domain name in the same manner.It took 3 weeks then my site was delisted. Then I unblocked him and after 1 month my site apeared in SERP again.