Unused folders and pages

[Jon_King]

I often find myself leaving "extra" versions of web pages and even entire folders used for experimental purposes that get uploaded to a particular site.

Do these extra pages and stuff have an effect on crawlers or create other issues with respect to SEP?

[keyplyr]

"Experimental" type pages are dangerous if they are uploaded to your domain only if they contain duplicate content as other pages, or other undisclosed reasons as appropriate.

One way to cover yourself is to use: <meta name="robots" content="noindex"> in the head section of these type pages so they will be ignored by the bots. Of course, paying better attention to what you upload wouldn't hurt either. :)

[martinibuster]

I seriously doubt that they will be dangerous because as long as they're not linked from anywhere else on the site, then the robots won't find it.

Something similar came up last week.

If you don't believe me (AND YOU SHOULD NOT) you can always download xenu link sleuth and set it loose on your web site and see what it will find. A spider follows a similar path, jumping from link to link - unless it's looking for a specific file/directory name with a specific document in it.

I've got client mockups, staged sites, secret sauce recipes, all kinds of stuff sitting on my site and nothing has ever touched it.

You would have to sucessfully guess the directory names to do it.

[Jon_King]

Then are you suggesting that a site is crawled starting from the index page. And if a page is not in the link pyramid of the index page it does not exist as far as crawlers are concerned?

[martinibuster]

Yes.
If you're paranoid you can always put the no index no follow meta on your hidden pages. I myself don't.

[Jon_King]

I'm not quite that paranoid,
but thanks Martinibuster.

[Visit Thailand]

I would say engines can find them as they may well end up in someone elses referrer log.

What I do is make the directory password protected and also use robots.txt to try and stop robots from having a peek.