Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Status: 404(not found) in Google Webmaster Tools

9:14 pm on Apr 23, 2008 (gmt 0)

Junior Member

5+ Year Member

joined:Mar 10, 2008
votes: 0

My website does not appear to have a robots.txt file even though when I do a check, Webmaster Tools says the Google bot is allowed.

On another one of my sites, it says:

status: 200(success)

even though I can't find a robots.txt file in the directory of the domain there.

Strange. Should I just go ahead and create a robots.txt file for the site that claims it can't be found?

thx for any comments.

12:48 pm on Apr 27, 2008 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
votes: 67

If there is no robots.txt, googlebot will assume all pages are allowed to be crawled.

If you uploaded a robots.txt file but do not see it, check your ftp settings. Some ftp programs do not automatically display certain files like .htaccess files and robots.txt. They may keep these files hidden.

6:39 pm on May 6, 2008 (gmt 0)

Full Member

5+ Year Member

joined:Sept 11, 2007
votes: 0

please create your rotbos.txt file and place that in a root directory or where you site contents are placed.

your robots.txt file may appear as www.widgetfoo.com/robots.txt