Welcome to WebmasterWorld Guest from 23.22.220.37

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt Was Inaccessible

Crawl Postponed Because Robots.txt Was Inaccessible

     
4:55 pm on Sep 15, 2012 (gmt 0)

New User

joined:Sept 15, 2012
posts: 3
votes: 0


Hey guys, I am getting error in my webmaster tool that- CRAWL POSTPONED BECAUSE ROBOTS.TXT WAS INACCESSIBLE. What does it means? How to remove this. I am seeing that my website is not crawling by google. Please find the attachments and let me have the thoughts
5:03 pm on Sept 15, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Oct 26, 2002
posts:3292
votes: 6


Welcome to WebmasterWorld, poojabhatt.

Try accessing your robots.txt with fetch as googlebot from within webmaster tools and note the results.
8:58 pm on Sept 15, 2012 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10542
votes: 8


welcome to WebmasterWorld, poojabhatt!

is that a recent message?
google will typically show this when the request for robots.txt returns a non-404/410 4XX or a 5XX status code.
for example if your server was temporarily down and supplied a 503 Unavailable response.
9:14 pm on Sept 15, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


It's a nightmare when that happens. That message sticks around for weeks and months, long after it ceases to be true.

If you're very unlucky the crawl errors numbers box and crawl errors graph below that completely disappears, and you get no crawl error data for the next few months either.
2:39 am on Sept 16, 2012 (gmt 0)

New User

joined:Sept 15, 2012
posts: 3
votes: 0


Hi jimbeetle, How to Try accessing robots.txt with fetch as googlebot? I don't know this. Can you tell me the process?
3:07 am on Sept 16, 2012 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12696
votes: 244


If you're in Google Webmaster Tools you're most of the way there. It's

:: shuffling papers ::

"Fetch as Google" under the "Health" tab. You can also try "Blocked URLs" on the same tab. This shows you the robots.txt that g is currently using. (They took away the "make a robots.txt" part, but the rest of the page is unchanged.) If there's a robots.txt displayed at all, then you know they are lying in their teeth and they had no trouble finding the file.
4:21 pm on Sept 16, 2012 (gmt 0)

New User

joined:Sept 15, 2012
posts: 3
votes: 0


Hi please check out my webmaster status. Google fetch I have already done that before. and I don't know about blocked urls. I have a print preview how can I show up the attachments.
Also let me know- what to do with :: shuffling papers ::, where I have to put that.