Welcome to WebmasterWorld Guest from 54.158.28.10

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Robots.txt added to Google Sitemaps

New feature for sitemaps...

     
12:41 am on Feb 7, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 1, 2004
posts:3181
votes: 0


Gives you the ability to test out changes to your robots.txt file and the response given for each of your webpages as you make changes to the robots.txt.

You can also test different user agents. The key here is to test changes before going live with a new robots.txt

Pretty cool feature!

8:04 pm on Feb 7, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 1, 2004
posts:3181
votes: 0


Geeze, I thought this would generate some dialog...
9:12 pm on Feb 7, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Aug 22, 2003
posts:333
votes: 0


Interesting!

Also on the "crawl stats" page at the bottom I now see:
"Your page with the highest PageRank"

With a three month history.

And "Page analysis" now has lists of word commonly used on your site and in links to your site.

11:46 pm on Feb 7, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 1, 2004
posts:3181
votes: 0


>>And "Page analysis" now has lists of word commonly used on your site and in links to your site.

There are a couple of words in that list that I know I used (intentionally) but I'm surprised they made the list. I don't think they are the most frequently used words.

2:21 am on Feb 8, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 21, 2006
posts:436
votes: 0


The feature hasnt been working for me for the last several hours.. anyone having trouble with it?
2:47 am on Feb 8, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Aug 22, 2003
posts:333
votes: 0


Is it just me or are the features I mentioned above only working for main domains and not subdomains?
4:20 pm on Feb 17, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:May 30, 2005
posts:118
votes: 0


Sorry to drag up an old topic, but I don't even have the new "page with highest PR" stats. I do see the new robots.txt tab though.

Any ideas on why I'm not seeing some of the new features? Sitemap is updated, downloaded, and crawled regularly.

5:05 pm on Feb 17, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 1, 2004
posts:3181
votes: 0


>>but I don't even have the new "page with highest PR" stats.

Some of the new features are on old pages. The above information is on the Crawl Stats tab.

5:17 pm on Feb 17, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:June 24, 2004
posts:147
votes: 1


The robots.txt page in Google Sitemaps is misleading because it makes it appear that Google's crawler was not able to find the robots.txt file.

I was confused (and concerned) because the page said I didn't have a robots.txt ("We didn't find a robots.txt file on your site.") even though I have one. Finally I discovered that clicking the "Check" button forces the utility to load the robots.txt file. The page is displaying information based on the tool's downloading of robots.txt, not Google's bots.

-- Roger

4:00 pm on Feb 21, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:May 30, 2005
posts:118
votes: 0


Some of the new features are on old pages. The above information is on the Crawl Stats tab.

Yeah I know... but it's not there.

Thanks anyway :)

8:36 pm on Mar 24, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:July 21, 2005
posts:84
votes: 0


FYI - this feature does not catch an error like this:

Disallow:/ example.php