Page is a not externally linkable
- Search Engines
-- Alternative Search Engines
---- A Dipsie tidbit


DoppyNL - 9:53 am on Apr 7, 2004 (gmt 0)


The minute they are starting to submit forms automaticly with a crawler they will be in my robots.txt with a complete disallow + a complete ban (I know, they won't be able to read the robots.txt then....)

A crawler has no way of knowing what a form is used for and what the result may be of posting the form.
I don't want them filling in my forms, as this would result in entry's in the database and who knows what.

Problems could be huge.

They expect to find more content when submitting forms?
I good constructed website will allow crawlers to reach all content via normal links. So this would not be neccesary.


Thread source:: http://www.webmasterworld.com/alternative_search_engines/1375.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com