Can robots.txt Help Protect From the Competition?

Want indexed pages but hidden from link: allinurl, etc.?

[tama]

I have a general interest site that has a lot of completely unrelated pages doing strong business, and I've noticed a couple of competitors with the same ideas, promoting the same products and it seems to be more than a coincidence. I'm suspecting they used the the "site:" and "allinurl" command and systematically duplicate my ideas/pages.

Is there any way I can protect my site from having this happen again? Is there a command I can insert into the robots.txt that will still allow my pages to be indexed, and retain the benefit of having a large site, yet keep these same pages from being listed using all the Google search commands i.e. link: allinurl, etc.?

[DerekH]

allow my pages to be indexed, <snip> yet keep these same pages from being listed using all the Google search commands

If you take time to look at what you wrote there, you'll realise that you've asked the impossible...

"I want my pages to be indexed and listed, but not accessible to any queries..."

And if you take the time, you'll therefore realise the answer!

Sorry...
DerekH

[tama]

More like:

"I want my pages to be indexed and listed, but not accessible to ALL unique Google command queries..."

I want Average Joe Google user to be able to find my site through the standard search query. But (if it's possible) I don't want my entire site listed using all the other Google search commands (i.e. allinurl, link:, etc.) that Average Joe Google users rarely use.

Maybe this is impossible - I don't know. Hence the question.

[inbound]

If someone wants to steal your information then it's a lot easier than using google to find pages.

If your pages are unrelated but are able to be found through links from the homepage then you can run a simple bot that will take a copy of the entire site and dump it conveniently on the desktop of the thief!

This happens alot but it isn't the only way people can use your information, see page hijacking.

It's scary what can happen when people see you are making money, being the web it could be anyone, anywhere so legal recourse is often difficult.

A major problem with this is that many highly qualified yet unemployed computer scientists exist around the world (the old USSR states are a prime example). Take the fact that wages are low by US/UK standards and you get a scenario where even relatively low earnings from good sites can be targetted. These are the amounts people won't chase too much.

I know people will think I'm having a go, I'm not and I think there are just as many dodgy people in the UK as anywhere.

I do have first hand experience of this so I wouldn't underestimate the motivation of people to earn a buck. A buck goes a lot further in certain countries.

Having said all that, I'd knock on your neighbours door. It's just as likely to be him as anyone else :)