Forum Moderators: DixonJones

Message Too Old, No Replies

Question about a weblog entry and a spider

robots.txt was visited

         

tilt

6:25 am on Nov 4, 2001 (gmt 0)

10+ Year Member



I just noticed the following entry in my weblog:

sjc.lycos.com (206.79.171.xxx) - Spider/Robot
02 Nov -- 02:13:39 -- 00:00 -- /robots.txt
02 Nov -- 02:13:39 -- -- /

To me, this looks like a lycos spider came to my site but left without visiting any of the pages. Why didn't it visit my pages?

My site is new, and I've just submitted it to search engines in the past few weeks.

My robots.txt file has only one entry:

User-agent: *

My intent is to allow all robots access to all pages.

I'm new at this, so any and all help is welcome. Thanks!

SmallTime

11:36 am on Nov 4, 2001 (gmt 0)

10+ Year Member



I think you need the disallow
line, even if blank -
User-agent:*
Disallow:

take a look at [searchengineworld.com...]

Sometimes they seem to drop by just to say hi.

tilt

1:40 am on Nov 5, 2001 (gmt 0)

10+ Year Member



Smalltime, thanks for that link. I found some good helpful info there. Looks like I got my work cut out for me. I like it. ;)