Robots.txt added to Google Sitemaps

New feature for sitemaps...

[BillyS]

Gives you the ability to test out changes to your robots.txt file and the response given for each of your webpages as you make changes to the robots.txt.

You can also test different user agents. The key here is to test changes before going live with a new robots.txt

Pretty cool feature!

[BillyS]

Geeze, I thought this would generate some dialog...

[abates]

Interesting!

Also on the "crawl stats" page at the bottom I now see:
"Your page with the highest PageRank"

With a three month history.

And "Page analysis" now has lists of word commonly used on your site and in links to your site.

[BillyS]

>>And "Page analysis" now has lists of word commonly used on your site and in links to your site.

There are a couple of words in that list that I know I used (intentionally) but I'm surprised they made the list. I don't think they are the most frequently used words.

[Tearabite]

The feature hasnt been working for me for the last several hours.. anyone having trouble with it?

[abates]

Is it just me or are the features I mentioned above only working for main domains and not subdomains?

[Lyndsay]

Sorry to drag up an old topic, but I don't even have the new "page with highest PR" stats. I do see the new robots.txt tab though.

Any ideas on why I'm not seeing some of the new features? Sitemap is updated, downloaded, and crawled regularly.

[BillyS]

>>but I don't even have the new "page with highest PR" stats.

Some of the new features are on old pages. The above information is on the Crawl Stats tab.

[jadebox]

The robots.txt page in Google Sitemaps is misleading because it makes it appear that Google's crawler was not able to find the robots.txt file.

I was confused (and concerned) because the page said I didn't have a robots.txt ("We didn't find a robots.txt file on your site.") even though I have one. Finally I discovered that clicking the "Check" button forces the utility to load the robots.txt file. The page is displaying information based on the tool's downloading of robots.txt, not Google's bots.

-- Roger

[Lyndsay]

Some of the new features are on old pages. The above information is on the Crawl Stats tab.

Yeah I know... but it's not there.

Thanks anyway :)

[suzie250]

FYI - this feature does not catch an error like this:

Disallow:/ example.php