Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- files requested by robots.txt


wilderness - 12:46 am on Jun 6, 2003 (gmt 0)


Andrea
Welcome to Webmaster World.

If you have pages on your website(s) which do not have links pointing to them anyplace on the internet?

That I would suggest NOT pointing to disallow these pages in robots.txt.
Why point a devious bot or person to something which you desire not to exist in a public forum, which for the most part is your robots.txt

I'll give you a couple of examples.

At one point I was courageous enough to use my standard email and a two-line signature in my Usenet participation mail submissions.
I was looking for a way to develop a 6-generational family database structure using MS-Acess. In my inquiry I provided an example to file which was displayed on a page in a folder which is basically private and used sparingly for a few friends.
Some four years later, I get an occassioanl referral from google archive of groups looking for that no-longer existent example. :(

On another occassion somebody in Usenet was inquiring about the free web pages which are provided with a discount registrar.
To show that person that a pop-was added I provided that domain name in a usenet mail.
Again four years later I get referrals from that mail in usenet.

Initailly when I created the folder my sole intent was to keep it private from the main-stream internet. These two slips taught me valuable lessons.

Don


Thread source:: http://www.webmasterworld.com/robots_txt/28.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com