homepage Welcome to WebmasterWorld Guest from 54.242.126.9
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Where Can A Newbie Find More Info?
I need to learn more...
jingleheimer

10+ Year Member



 
Msg#: 303 posted 8:28 pm on Jan 23, 2001 (gmt 0)

I've noticed that Searchengineworld.com and webmasterworld.com are great places to go to learn about Search Engine Optimization, but where does a newbie go to learn about Search Engine Spider Identification besides this forum. I've built a great website (in my opinion) and now I've submitted to enough search engines that I'm finally getting hits on my website. However, I've got many questions. For example, my statistics software (using LiveStats 5.0 software from MediaHouse) shows that I am getting about 5000 sessions a week, but only about 20% of those are referred sessions. Is the other 80% coming from Spiders? I've also noticed that a lot of my stats show stuff like "cache-ra04.proxy.aol.com". These are filling up all over the place and I don't know why they're so popular. Can anyone help or point me to the right direction so I can find out more information? Thank-you so much.

Jon

 

Froggyman



 
Msg#: 303 posted 6:56 am on Jan 24, 2001 (gmt 0)

Try the Web Robots Database at:

[info.webcrawler.com...]

It's a little outdated but still has a lot of good info and some good links.

WebRookie

10+ Year Member



 
Msg#: 303 posted 6:20 pm on Jan 24, 2001 (gmt 0)

Here are more spider links to check out:

[edit: url snipped: bt]

Botspot
[botspot.com...]

Spiderkb.com
[spiderkb.com...]

List of resources:
Intelligent Agents WWW sites
[comp.lancs.ac.uk...]

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 303 posted 6:37 pm on Jan 24, 2001 (gmt 0)

snipped the urls since jk has dropped urls many times this week. nuff.

Eathan

10+ Year Member



 
Msg#: 303 posted 11:07 pm on Jan 24, 2001 (gmt 0)

Hey cool! WebRookie, you are officially the first person to actually recommend my site, and not as an example of just how "Under Construction" a site can be. :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved