Welcome to WebmasterWorld Guest from 23.22.46.195

Robots.txt added to Google Sitemaps

New feature for sitemaps...

   
12:41 am on Feb 7, 2006 (gmt 0)

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Gives you the ability to test out changes to your robots.txt file and the response given for each of your webpages as you make changes to the robots.txt.

You can also test different user agents. The key here is to test changes before going live with a new robots.txt

Pretty cool feature!

8:04 pm on Feb 7, 2006 (gmt 0)

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Geeze, I thought this would generate some dialog...
9:12 pm on Feb 7, 2006 (gmt 0)

10+ Year Member



Interesting!

Also on the "crawl stats" page at the bottom I now see:
"Your page with the highest PageRank"

With a three month history.

And "Page analysis" now has lists of word commonly used on your site and in links to your site.

11:46 pm on Feb 7, 2006 (gmt 0)

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member



>>And "Page analysis" now has lists of word commonly used on your site and in links to your site.

There are a couple of words in that list that I know I used (intentionally) but I'm surprised they made the list. I don't think they are the most frequently used words.

2:21 am on Feb 8, 2006 (gmt 0)

5+ Year Member



The feature hasnt been working for me for the last several hours.. anyone having trouble with it?
2:47 am on Feb 8, 2006 (gmt 0)

10+ Year Member



Is it just me or are the features I mentioned above only working for main domains and not subdomains?
4:20 pm on Feb 17, 2006 (gmt 0)

5+ Year Member



Sorry to drag up an old topic, but I don't even have the new "page with highest PR" stats. I do see the new robots.txt tab though.

Any ideas on why I'm not seeing some of the new features? Sitemap is updated, downloaded, and crawled regularly.

5:05 pm on Feb 17, 2006 (gmt 0)

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member



>>but I don't even have the new "page with highest PR" stats.

Some of the new features are on old pages. The above information is on the Crawl Stats tab.

5:17 pm on Feb 17, 2006 (gmt 0)

10+ Year Member



The robots.txt page in Google Sitemaps is misleading because it makes it appear that Google's crawler was not able to find the robots.txt file.

I was confused (and concerned) because the page said I didn't have a robots.txt ("We didn't find a robots.txt file on your site.") even though I have one. Finally I discovered that clicking the "Check" button forces the utility to load the robots.txt file. The page is displaying information based on the tool's downloading of robots.txt, not Google's bots.

-- Roger

4:00 pm on Feb 21, 2006 (gmt 0)

5+ Year Member



Some of the new features are on old pages. The above information is on the Crawl Stats tab.

Yeah I know... but it's not there.

Thanks anyway :)

8:36 pm on Mar 24, 2006 (gmt 0)

5+ Year Member



FYI - this feature does not catch an error like this:

Disallow:/ example.php

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month