homepage Welcome to WebmasterWorld Guest from 54.197.110.151
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
How do I use the Sim Spider tool
to see what spiders see?
grnidone



 
Msg#: 90 posted 3:21 pm on Jul 19, 2000 (gmt 0)

I feel so stupid asking this question.

How do I use that Sim Tool Brett has on his site? I don't understand the "referer" part.

-G

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 90 posted 5:31 pm on Jul 19, 2000 (gmt 0)

You understand the concept of Refers? Everytime you connect to a site off a link, your browser sends that link to the next site right? follow that.

The referrer and agent part of Sim Spider just allows you to set that referrer and/or agent name that will be reported to the site you connect too.

You won't fool any cloakers into thinking you are a search engine spider by changing the agent name to a se spider. It is mainly a tool to strip the html and look at the 'naked' page with tags to see what a search engine will see.

The agent name is handy if a site is 'bigoted' and only allows certain (read: ie) browsers in the front door.

Referrer isn't all that important and probably should just be left blank.

grnidone



 
Msg#: 90 posted 5:36 pm on Jul 19, 2000 (gmt 0)

I understand the concept of "refers", but I wasn't understanding how it was going to help me see what search engines see.

Thanks for the help.
-Heather

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 90 posted 9:32 pm on Jul 19, 2000 (gmt 0)

Ok, some pages require a 'referrer' from an on-site link to view a page. Other still, you just may want to, umm, fake out thinking you are coming from some other link.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved