Googlebot crawling non existing links from site search box

[darshanaasodekar]

My site has a search box (search feature) where customers can find the products they want to search on my site.

But a weird thing that google bots seem to do is putting any keyword in the search box. Because of which some non existing urls (404 not found) are found in google webmaster tools.


Is it possible for google bots to put such random keywords in the search box?

Also, what is the solution to get rid of this?

[deadsea]

My guess is that Google is observing what users type in the search box when the user has the Google toolbar installed. Googlebot comes back and tries the same thing that a user tried.

On all the sites that I work on, we provide both HTML and XML sitemaps for Googlebot. Every page on the site is discoverable by crawlers through those mechanisms. Therefore we put the site search urls in robots.txt. So for example if a site search call the following url:
http://example.com/search?q=keyword
we would put
Disallow: /search
into robots.txt

[darshanaasodekar]

I have added search to robots.txt....... waiting for results now.. Thanks for the help...