Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: mack
I have links on my site in the form of:
First of all MSNBot tries to index "/www.externalsite.com/index.asp" on my server.
Second, which is worse, only seconds later it tries to index pages linked to from www.externalsite.com on my server. For example, on the external site this link exists:
and MSNBot tries to index this on my server:
Oh - MSN advised me to change the URL's of my links, because it confused their bot...
Has anyone seen this before, and am I right in concluding that these are errors in MSNBot?
Ok I realy am not 100% about this but ... if the method you are using is innocent, it may just be the fact that many other use this method to grab pages on thier own server from others sites.
I have found a number of .asp?myurl pages in search engines that are basically my content with my url in its url after a question mark, but ... the content has been cached and thus indexed as part of thier site. Enter duplicate content problems.
To be honest I dont fully understand this issue thus will leave it there. If you or anyone knows how this?myurl thing works it may explain what I am trying to describe. What is the asp page doing with the?urlhere is it cacheing it locally?
BTW - the bot in question has no reverse DNS specified, and since I emailed MSN about it it has been scanning my site around once every hour - more specifically it's been scanning those asp?URL= links only. I'm starting to suspect it's a new bot they're testing.
The IP is 18.104.22.168.