Coding a spider

How to give yourself an agent name

[chris_f]

I am thinking of building a small time search engine for one of my sites. I wish people to see that I am crawling their site. For instance, I want them to see BotA instead of my IP Address. How do I declare an agent name for peoples logs?

Any know?

[PsychoTekk]

it's impossible to replace the IP, but you can set the HTTP_USER_AGENT variable to "BotA", so in their logs the bot's name will show up.
the client side name of the variable is 'User-Agent' (HTTP_USER_AGENT is the serverside variable that contains the name string)
for example:
GET ... HTTP/1.1
Host: ...
User-Agent: BotA
Connection: close

[Brett_Tabke]

It all depends upon the language and interface chris. It is doubtful, that you will be managing the http connection yourself, but there will probably be some setting somewhere to adjust the user agent header.

[chris_f]

That's exactly what I was looking for thanks. My question is, however, how would I make the user agent say BotA when I grab the pages for the search engine. I am coding in ASP.net if anyone can help.

P.S. I didn't make my question clear. I wasn't trying to alter the IP Address field.

[PsychoTekk]

i myself have no idea od ASP but this might help you:
[aspfusion.net...]

[jdMorgan]

chris_f,

And, after having read around in this forum, I'd recommend adding a valid e-mail address, or even better, a web page URL in the UA for your fellow wmw webmasters to check on your robot when they spot it in their logs.

My policy is to block any unknown robot I can't check out, and it looks like several others here do the same...

Cheers,

Jim