Welcome to WebmasterWorld Guest from 54.224.172.145

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt appearing in my log files

But I have n't got one there!

     
8:29 am on Jul 21, 2004 (gmt 0)

Junior Member

joined:May 1, 2004
posts:144
votes: 0


Hi, there are entries in my log file, see below..

GET /robots.txt - 80 VHOSTS\domain621531

but I have n't got that file. I am beginning to think it might be lurking somewhere and putting of bots! For instance Google refuses to index my site other than the front page despite good links, navigation etc..

9:47 am on July 21, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Feb 5, 2002
posts:333
votes: 0


What appears in your logs may just be an attempt by a spider to retrieve this file, it doesn't necessarily mean that the file exists. What was the HTTP response code returned by your webserver?
2:38 pm on July 21, 2004 (gmt 0)

Junior Member

joined:May 1, 2004
posts:144
votes: 0


404! Of course. Thanks for pointing that out. I can rest now!

Any ideas why google will only index my first page?

1:48 pm on Aug 2, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 30, 2002
posts:303
votes: 0


You should really add a robots file otherwise you will find your error logs get bunged down with 404 for it.

As far as google not spidering your site, have you got any external links to your content pages (as oppposed to just the home page)? I've found this can certainly help.

2:59 pm on Aug 5, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:June 19, 2003
posts:198
votes: 0


Is there any risk to turning off a web crawler [webcrawler.com] (wow, it's still online!) with adding a robots.txt? (like, might they not get a 404 and just assume they're not allowed?)

What do you put in a robots.txt that allows full site indexing?