MSIE Crawler, visits daily, follows overture listing

A nasty little 'spider' that is costing me money

[werty]

So I am purchasing some stuff on overture and there is a weird referrer that I keep getting.

I have a feeling it is someone with a bot that clicks the advertising to waste my money.

They come in from yahoo, clicking on the sponsored link and only get the robots.txt. The IP is different each time.

XXX.XXX.XXX.XXX - - [17/Sep/2003:10:05:15 -0700] "GET /robots.txt HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.3.2.0; .NET CLR 1.1.4322; MSIECrawler)"

It seems to come around once a day. The phrase costs about 6 dollars per click so it would be costing me 42 dollars a week.

Any suggestions? Write my Account manager at Overture?

[WebJoe]

Normally, this ua is used if someone adds the page to the iE favorites and chooses the "available offline"-option. (see [webmasterworld.com...] easy to verify qith a page that displays your user agent string)

Depending on the update schedule and the time the users computer is online (or auto-dial enabled), it is possible that you see that ua-string regularly with varying IP-adresses.

The IE built-in "crawler" just follows the links from the bookmarke page (I don't know the pattern), so I think what seems to be a following of the overtour listing is just a coincidence.

[jdMorgan]

werty,

You could disallow MSIECrawler in robots.txt, and then back up the disallow by blocking it using .htaccess or similar (this "block" must still allow MSIECrawler to fetch robots.txt).

If this is really MSIECrawler, it should "give up" on your site. If it isn't, then it won't.

And yes, you should report this if it's costing you that much! There is no reason for someone to bookmark a sponsored link (the tracking link, rather than your actual site) and to "make it available offline." It must have been done manually, and intentionally.

Jim

[werty]

I originally had the MSIE Crawler blocked in my robots.txt, but allowed it yesterday since I thought they may be trying to get other pages and obeyed the robots.txt and crawled no further.

I guess I will block it again, and contact overture.

Thank you for the help.