Welcome to WebmasterWorld Guest from 50.16.126.107

Forum Moderators: goodroi

Message Too Old, No Replies

[404] HTTP Page robots.txt not found 404 error

     
12:11 pm on Dec 5, 2003 (gmt 0)

New User

10+ Year Member

joined:Dec 5, 2003
posts:5
votes: 0


Could anyone tell me why this error generated on log files?

Cheers!

12:34 pm on Dec 5, 2003 (gmt 0)

Full Member

10+ Year Member

joined:May 27, 2003
posts:247
votes: 2


robots.txt is a standard way that search engines determine areas of your site to search.

for more see..
[searchengineworld.com ]

12:37 pm on Dec 5, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 12, 2003
posts:50
votes: 0


Hello Strike,

and wellcome. Search engine spiders are trying to find this file first. It tells them what they can and cannot spider. If it does not exist they think that everything on the website is spider food.

With robots.txt file the webmaster can ask spiders not to go to given directories or given files. Most robots follow the instructions but not all.

12:44 pm on Dec 10, 2003 (gmt 0)

New User

10+ Year Member

joined:Dec 5, 2003
posts:5
votes: 0


thanks all for your help!
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members