spider simulater

how do we define user agents?

[iggy99]

not sure if i am in the correct forum topic for this but here goes!

we are trying to use [searchengineworld.com...]

and "see what the spiders see"

how do we define user agents?

all i see is mozilla and no way to define user agent...

what am i missing???

are there any other spider / agent checking apps available?

the url i am trying to check is:
[webshop101.com...]

[Mardi_Gras]

It is a generic spider - you can't simulate a specific user agent.

[iggy99]

i am confused

on the page for the spider it talks about showning information specific to "Search Engine"

if infact we are using Mozilla/4.0 (compatible; MSIE 5.0; Windows NT 5.0) as a user agent we will never see any agent specific content ---

i have written a php agent sniffer to use for my title keywords etc and would like to check it out to see if the display actually changes from une agetn to another ---

anybody know of any spider simulators that might be of help?

many thnaks

[Brett_Tabke]

Nah, I won't do a custom user agent again - people were bombing it with requests trying to snoop on other folks. So, I put in the custom user agent started passing the users ip with the request. All the goofy stuff went down.

It's certainly not meant to be any kind of cloaking detector. It's a code viewer.

[iggy99]

i understand ---

are you familiar with any other similer apps?