homepage Welcome to WebmasterWorld Guest from 23.20.61.85
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Status: 404(not found) in Google Webmaster Tools
Boulder90

5+ Year Member



 
Msg#: 3633559 posted 9:14 pm on Apr 23, 2008 (gmt 0)

My website does not appear to have a robots.txt file even though when I do a check, Webmaster Tools says the Google bot is allowed.

On another one of my sites, it says:

status: 200(success)

even though I can't find a robots.txt file in the directory of the domain there.

Strange. Should I just go ahead and create a robots.txt file for the site that claims it can't be found?

thx for any comments.

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3633559 posted 12:48 pm on Apr 27, 2008 (gmt 0)

If there is no robots.txt, googlebot will assume all pages are allowed to be crawled.

If you uploaded a robots.txt file but do not see it, check your ftp settings. Some ftp programs do not automatically display certain files like .htaccess files and robots.txt. They may keep these files hidden.

bilalseo

5+ Year Member



 
Msg#: 3633559 posted 6:39 pm on May 6, 2008 (gmt 0)

please create your rotbos.txt file and place that in a root directory or where you site contents are placed.

your robots.txt file may appear as www.widgetfoo.com/robots.txt

thanks,

bilal

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved