Page is a not externally linkable
DoppyNL - 9:53 am on Apr 7, 2004 (gmt 0)
A crawler has no way of knowing what a form is used for and what the result may be of posting the form. Problems could be huge. They expect to find more content when submitting forms?
The minute they are starting to submit forms automaticly with a crawler they will be in my robots.txt with a complete disallow + a complete ban (I know, they won't be able to read the robots.txt then....)
I don't want them filling in my forms, as this would result in entry's in the database and who knows what.
I good constructed website will allow crawlers to reach all content via normal links. So this would not be neccesary.