Welcome to WebmasterWorld Guest from 184.108.40.206 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor 2014
What is this.sv-fw.looksmart.com
I appeared today, but does looksmart use a spider?
Btw. Hello everybody, this is my first post in here.
Look forward to spending more time in here.
Welcome to WebmasterWorld addvisors. :)
Actually, Looksmart has two of them:
(http://www.WISEnutbot.com is a search engine.)
Their newest Dead Link Checker.
220.127.116.11 - - [29/Oct/2003:15:58:55 -0800] "GET /robots.txt HTTP/1.1" 200 1524 "-" "Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01 (email@example.com; http*//www.WISEnutbot.com)"
18.104.22.168 - - [17/Oct/2003:23:20:59 -0700] "GET /robots.txt HTTP/1.1" 200 1524 "-" "Mozilla/4.0 compatible ZyBorg/1.0 (firstname.lastname@example.org; http*//www.WISEnutbot.com)"
Sorry you had to bump your own 'first post' :(
Normally, answers are lots faster in coming.
Thanks a lot for the answer,
Does that mean that the wisenutbot will eventually crawl my site when it has visited my index.htm now?
It has been my experience that once a (legitimate) bot visits my robots.txt file (even if only one visit has been made as you mentioned), eventually that bot will return and begin the indexing process.
You might find
this thread [ webmasterworld.com] interesting reading as it has links to specific posts where various bots (good/bad and evil) are pretty much all mentioned. The most current is at the bottom of that page.