Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Googlebot possibly ignoring robots.txt


phranque - 8:20 pm on May 3, 2012 (gmt 0)


And how is the URL discovered?

it doesn't have to be "explicitly published".
examples of how are given in the linked google support thread.
"unintended" urls can "legitimately" be harvested from referrers in published log files, browser toolbars, gmail, ...


Thread source:: http://www.webmasterworld.com/robots_txt/4446154.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com