Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- How to Stop BecomeBot?


Clint - 9:51 am on Jun 23, 2005 (gmt 0)


Interesting. The Become bot only visited ONE of my pages. It first hit my robots.txt file, then only ONE webpage. I searched their SE for the product on that page, and I was 1st on the 1st page. I searched for other specific products I carry, and I'm also on the first page for them. Strange that it chose one of my FORWARDED domains that point to my MAIN domain as 1st spot in one search! FAIK, coincidentally that could be the very first time their bot has ever visited. I don't now why it chose this one page though and no others to visit! It was not even an entry page.

Their SE doesn't seem to do very well with the generic searches I first tried like "[blue widget] sales" or dealers, all I saw were non relevant results. But when searching for a specific product, it does better. I like the fact they have a rating system where you can rate each search, and I like their "auto complete" of searches. I've never seen that before with any SE.


Thread source:: http://www.webmasterworld.com/robots_txt/670.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com