Page is a not externally linkable
shopvince - 11:40 pm on Jun 19, 2009 (gmt 0)
It was crawling at 25 pages/second so we bumped the crawl rate down to 1 but it took them about 8 hours to change their rate. What's really odd is that the adidxbot never actually read our robots.txt file! So it must get the info from another bot (MSNBot?). All the IPs were unique (i.e. they weren't the same as any other MSNBot). We still aren't sure what Microsoft is doing with this Bot. Anyone know?
The adidxbot/1.1 (+http://search.msn.com/msnbot.htm) bot drove us crazy for the last week or so. It was crawling every SEM keyword URL we had configured in our MSN account. Sort of a "Welcome to Bing" I guess.