homepage Welcome to WebmasterWorld Guest from 54.167.173.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt Was Inaccessible
Crawl Postponed Because Robots.txt Was Inaccessible
poojabhatt




msg:4495523
 4:55 pm on Sep 15, 2012 (gmt 0)

Hey guys, I am getting error in my webmaster tool that- CRAWL POSTPONED BECAUSE ROBOTS.TXT WAS INACCESSIBLE. What does it means? How to remove this. I am seeing that my website is not crawling by google. Please find the attachments and let me have the thoughts

 

jimbeetle




msg:4495527
 5:03 pm on Sep 15, 2012 (gmt 0)

Welcome to WebmasterWorld, poojabhatt.

Try accessing your robots.txt with fetch as googlebot from within webmaster tools and note the results.

phranque




msg:4495571
 8:58 pm on Sep 15, 2012 (gmt 0)

welcome to WebmasterWorld, poojabhatt!

is that a recent message?
google will typically show this when the request for robots.txt returns a non-404/410 4XX or a 5XX status code.
for example if your server was temporarily down and supplied a 503 Unavailable response.

g1smd




msg:4495577
 9:14 pm on Sep 15, 2012 (gmt 0)

It's a nightmare when that happens. That message sticks around for weeks and months, long after it ceases to be true.

If you're very unlucky the crawl errors numbers box and crawl errors graph below that completely disappears, and you get no crawl error data for the next few months either.

poojabhatt




msg:4495614
 2:39 am on Sep 16, 2012 (gmt 0)

Hi jimbeetle, How to Try accessing robots.txt with fetch as googlebot? I don't know this. Can you tell me the process?

lucy24




msg:4495621
 3:07 am on Sep 16, 2012 (gmt 0)

If you're in Google Webmaster Tools you're most of the way there. It's

:: shuffling papers ::

"Fetch as Google" under the "Health" tab. You can also try "Blocked URLs" on the same tab. This shows you the robots.txt that g is currently using. (They took away the "make a robots.txt" part, but the rest of the page is unchanged.) If there's a robots.txt displayed at all, then you know they are lying in their teeth and they had no trouble finding the file.

poojabhatt




msg:4495789
 4:21 pm on Sep 16, 2012 (gmt 0)

Hi please check out my webmaster status. Google fetch I have already done that before. and I don't know about blocked urls. I have a print preview how can I show up the attachments.
Also let me know- what to do with :: shuffling papers ::, where I have to put that.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved