Welcome to WebmasterWorld Guest from 18.104.22.168 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
sv-fw.looksmart.com addvisors msg:397113 5:31 pm on Oct 31, 2003 (gmt 0) What is this.sv-fw.looksmart.com
I appeared today, but does looksmart use a spider?
addvisors msg:397114 6:53 pm on Oct 31, 2003 (gmt 0)
Btw. Hello everybody, this is my first post in here.
Look forward to spending more time in here.
pendanticist msg:397115 7:23 pm on Oct 31, 2003 (gmt 0)
Welcome to WebmasterWorld addvisors. :)
Actually, Looksmart has two of them:
(http://www.WISEnutbot.com is a search engine.)
Their newest Dead Link Checker.
22.214.171.124 - - [29/Oct/2003:15:58:55 -0800] "GET /robots.txt HTTP/1.1" 200 1524 "-" "Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01 (firstname.lastname@example.org; http*//www.WISEnutbot.com)"
126.96.36.199 - - [17/Oct/2003:23:20:59 -0700] "GET /robots.txt HTTP/1.1" 200 1524 "-" "Mozilla/4.0 compatible ZyBorg/1.0 (email@example.com; http*//www.WISEnutbot.com)"
Sorry you had to bump your own 'first post' :(
Normally, answers are lots faster in coming.
addvisors msg:397116 7:32 pm on Oct 31, 2003 (gmt 0)
Thanks a lot for the answer,
Does that mean that the wisenutbot will eventually crawl my site when it has visited my index.htm now?
pendanticist msg:397117 7:54 pm on Oct 31, 2003 (gmt 0)
It has been my experience that once a (legitimate) bot visits my robots.txt file (even if only one visit has been made as you mentioned), eventually that bot will return and begin the indexing process.
You might find
this thread [ webmasterworld.com] interesting reading as it has links to specific posts where various bots (good/bad and evil) are pretty much all mentioned. The most current is at the bottom of that page.