homepage Welcome to WebmasterWorld Guest from 107.22.37.143
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Visit PubCon.com
Home / Forums Index / Library
Forum: Search Engine Spider and User Agent Identification
Displaying Topics 39 - 78 (48 total) Sorted by: Date-Last-Post, Direction: reverse
#Subjectsort orderDate
39: new post   The IRS Bots - Xenon Spiders Web for the Tax Man
"A five-nation tax enforcement cartel has been quietly cracking down on suspected internet tax cheats, using a sophisticated web crawling program to monitor transactions on auction sites, and track operators of online shops, poker and porn sites."
Jan 25, 2007
40: new post   MSN Robot Crawlers Renamed
WebmasterWorld Exclusive: MSNDude "These are all Microsoft bots that have been around for a while...."
July 31, 2006
41: new post   Updated Bot List
Oct 4, 2003
42: new post   New Fast/Overture Robot UA
Sept 7, 2003
43: new post   YahooSeeker/1.0
What is it?
Aug 23, 2003
44: new post   Blocking WebPage Monitoring Services
Most of these services are unwelcome bandwidth sponges. Lets call them out, identify them, and block them together.
Apr 9, 2003
45: new post   Bad Boy Bot of the Week
Pick a bot, any bot: This weeks fun one is an affiliate bot out walking and not respecting robots.txt.
Dec 6, 2002
46: new post   Mysterious User Agent and 404 Random Strings
"Altavista and DIIbot use suspicious request methods to test 404 errors."
June 11, 2002
47: new post   Search Engine and Spider IP Lists
Nice resource on the major operating spiders.
June 10, 2002
48: new post   How important is the Robots.txt file now?
"Am I missing out on a lot of traffic by not having a robots.txt file? Or is that a thing of the past?"
Jan 25, 2002
#Subjectsort orderDate
open in new window=open in new window

   
<< Back 1 2 Next >>
Home / Forums Index / Library
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved