Page is a not externally linkable
- Google
-- Google News Archive
---- Is Google crawling IRC?


punta - 4:03 pm on Nov 11, 2003 (gmt 0)


My best guess: Google is scraping Usenet for URLs. Sites may be mentioned on Usenet long before Googlebot can find them. If Google is trying for the biggest, most complete index, it makes sense to look for URLs in as many places as possible.

I don't believe they are trying to obtain the biggest index. Google don't index 'island' pages. If a page hasn't got any links to it and it's submitted to google via add URL, it doesn't stay in the index for long (if it gets there at all). Surely indexing these pages would increase the size of the index.


Thread source:: http://www.webmasterworld.com/google_archive/18226.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com