I run a message board it is the one from cgi-factory
I want search engines to index each message thread but unfortunatly there are 'edit' and 'delete' links on each message and Google has followed each of these and indexed each of the page it throws up - this is using huge bandwith for no reason.
So i thought i could use the robots.txt to stop it but i am new to robots.txt so dont really know how.
The message board pages i want indexed have the URL:
when you click an edit function you get the URL:
Is there a way i can stop indexing of all the edit pages easily?
Another way i have thought was to add something in the <head> tag to stop search engines following links off each of these pages. Is there such a command?
All messages have a master template so it would be simple to add. However this would only cover new messages not all the old ones.