Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Where Can A Newbie Find More Info?

I need to learn more...



8:28 pm on Jan 23, 2001 (gmt 0)

I've noticed that Searchengineworld.com and webmasterworld.com are great places to go to learn about Search Engine Optimization, but where does a newbie go to learn about Search Engine Spider Identification besides this forum. I've built a great website (in my opinion) and now I've submitted to enough search engines that I'm finally getting hits on my website. However, I've got many questions. For example, my statistics software (using LiveStats 5.0 software from MediaHouse) shows that I am getting about 5000 sessions a week, but only about 20% of those are referred sessions. Is the other 80% coming from Spiders? I've also noticed that a lot of my stats show stuff like "cache-ra04.proxy.aol.com". These are filling up all over the place and I don't know why they're so popular. Can anyone help or point me to the right direction so I can find out more information? Thank-you so much.



6:56 am on Jan 24, 2001 (gmt 0)

Try the Web Robots Database at:


It's a little outdated but still has a lot of good info and some good links.


6:20 pm on Jan 24, 2001 (gmt 0)

10+ Year Member

Here are more spider links to check out:

[edit: url snipped: bt]



List of resources:
Intelligent Agents WWW sites


6:37 pm on Jan 24, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

snipped the urls since jk has dropped urls many times this week. nuff.


11:07 pm on Jan 24, 2001 (gmt 0)

10+ Year Member

Hey cool! WebRookie, you are officially the first person to actually recommend my site, and not as an example of just how "Under Construction" a site can be. :)

Featured Threads

Hot Threads This Week

Hot Threads This Month