Welcome to WebmasterWorld Guest from 23.22.220.37

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt

Generates a 404 error in my logs

     
4:39 pm on May 2, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 28, 2002
posts:3443
votes: 1


there is on my statistic page about how my page is doing a error it says it gets a 404 and this robots.txt, what can that mean.

thanks

zeus

P.s I dont use robots.txt in meta

5:45 pm on May 2, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:May 9, 2001
posts:416
votes: 0


Sounds like spider(s) look for your robots.txt file, don't find it, and are passed a 404 error.
5:50 pm on May 2, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 30, 2000
posts:3300
votes: 0


Hi zeus,

Welcome to WebmasterWorld! You will like it.

Here is good info about this file that should reside at root level of your site.

[searchengineworld.com...]

Enjoy!

8:15 pm on May 2, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 25, 2002
posts:470
votes: 0


If you have no reason to disallow a spider from entering any directory on your site, is there any reason to have Robots.txt at all?
8:28 pm on May 2, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 30, 2000
posts:3300
votes: 0


Yes.

Stat files
cgi-bin
image folder
work in progress
duplicate pages made for different platforms and browsers
personal stuff
clients folders
cloaked pages

Anything that needs to be stored on site you dont want to publish for some reason.

This file can let you disalow specific files or folders to specific well behaving spiders.

If you just want to free your error log file from robots.txt 404, just paste this in a .txt file and leave it at root level.

User-agent: *
Disallow:

9:36 am on May 3, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 28, 2002
posts:3443
votes: 1


Thanks everybody and macguru I have just put my site through that validator and got many errors, but my site works perfect.

If a spider cant find my page that is bad, all my visits come from search engines.

Must I have robot.txt now, because I have never used it before and im listed good under another domain name which site is down now.

zeus

12:33 pm on May 3, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 28, 2002
posts:3443
votes: 1


I found a new error thats cald favicon.ico I dont know what that is?
12:46 pm on May 3, 2002 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
posts:2530
votes: 1


Favicon is the little image you get when you bookmark a site in IE.
Try the Site Search on top of this page - very good for finding information.
Favicon [webmasterworld.com]

Welcome btw :)

1:52 pm on May 3, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 28, 2002
posts:3443
votes: 1


Thanks for the link, I have forgot everything about ICO because I dont use and I think that is ok for the S.E, but I still wonder about the error about robots.txt, that must mean that they where looking for and diddent find it and thats it and it is not essential for a listing at search engines wright.

thanks all

zeus