homepage Welcome to WebmasterWorld Guest from 54.211.97.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt appearing in my log files
But I have n't got one there!
uk_webber



 
Msg#: 426 posted 8:29 am on Jul 21, 2004 (gmt 0)

Hi, there are entries in my log file, see below..

GET /robots.txt - 80 VHOSTS\domain621531

but I have n't got that file. I am beginning to think it might be lurking somewhere and putting of bots! For instance Google refuses to index my site other than the front page despite good links, navigation etc..

 

incywincy

10+ Year Member



 
Msg#: 426 posted 9:47 am on Jul 21, 2004 (gmt 0)

What appears in your logs may just be an attempt by a spider to retrieve this file, it doesn't necessarily mean that the file exists. What was the HTTP response code returned by your webserver?

uk_webber



 
Msg#: 426 posted 2:38 pm on Jul 21, 2004 (gmt 0)

404! Of course. Thanks for pointing that out. I can rest now!

Any ideas why google will only index my first page?

elgumbo

10+ Year Member



 
Msg#: 426 posted 1:48 pm on Aug 2, 2004 (gmt 0)

You should really add a robots file otherwise you will find your error logs get bunged down with 404 for it.

As far as google not spidering your site, have you got any external links to your content pages (as oppposed to just the home page)? I've found this can certainly help.

broniusm

10+ Year Member



 
Msg#: 426 posted 2:59 pm on Aug 5, 2004 (gmt 0)

Is there any risk to turning off a web crawler [webcrawler.com] (wow, it's still online!) with adding a robots.txt? (like, might they not get a 404 and just assume they're not allowed?)

What do you put in a robots.txt that allows full site indexing?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved