Real source of bot visitor...

[sarahk]

I have a bot that visits my site - it's coming from 2 different sources (at least) and both appear to be dial up connections. One is on the Bellsouth (ARIN) network and the other is in the APNIC domain - so physically quite different.

They ask for "/" and "/v2/" rather than explicit file names but don't go through all the folders. At first I thought it might be a link I'd asked to have listed elsewhere but now I don't think so.

I've modified my .htaccess so they go to my index page but I'd rather feed them the data they want. Now if they have a hosted site and are checking their links through a dial up connection it will be tricky to find out what sites these bots are linked to. ANY IDEAS?

This is the only info I gather. Are there more questions I could be "asking"? I use PHP.

["REMOTE_ADDR"]=> string(11) "219.76.64.3"
["REMOTE_PORT"]=> string(5) "29837"

["REMOTE_ADDR"]=> string(12) "68.19.196.21"
["REMOTE_PORT"]=> string(5) "50084"

[Romeo]

Hi sarahk and welcome.

From my own experience, private bots coming from private subscriber lines are in almost all cases bad, nasty, or at least dubious. The question in this case should be: what is that bot to poke around in my site I originally set up for human users and not bots? what is their real purpose? do they cause more traffic (that I have to pay for)? trying to rip content? trying to harvest mail addresses to send spam later?
If they just looked for a /v2 and initially got a "404 - not found" I would just take back your change to your .htaccess and ignore them.
You don't owe them any service.

Regards,
R.

[Netizen]

A request for "/" is the index page of your site. As Romeo said, private bots aren't worth the trouble. Just look after the main search engines.

[sarahk]

thanks for the feedback guys.

I know about / but it was the v2/ that goes nowhere.

I have no problems making my site attractive to the main engines (just visit NOW please freshbot) but also want to ensure thay my external links work and that when the webmasters of those sites validate their links my site looks good.

[Netizen]

But if people link to the wrong places on your site presenting a custom 404 page is probably the best thing to do.

[sarahk]

aha, what I think I might give them is a quick picks page with all the links on it, that way the "bot" will find the recipricol link (because that's what I think it's trying to do). I can mark the page noindex, nofollow so it doesn't end up in the search engines.