homepage Welcome to WebmasterWorld Guest from 54.237.38.30
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Robots.txt added to Google Sitemaps
New feature for sitemaps...
BillyS

WebmasterWorld Senior Member billys us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 33049 posted 12:41 am on Feb 7, 2006 (gmt 0)

Gives you the ability to test out changes to your robots.txt file and the response given for each of your webpages as you make changes to the robots.txt.

You can also test different user agents. The key here is to test changes before going live with a new robots.txt

Pretty cool feature!

 

BillyS

WebmasterWorld Senior Member billys us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 33049 posted 8:04 pm on Feb 7, 2006 (gmt 0)

Geeze, I thought this would generate some dialog...

abates

10+ Year Member



 
Msg#: 33049 posted 9:12 pm on Feb 7, 2006 (gmt 0)

Interesting!

Also on the "crawl stats" page at the bottom I now see:
"Your page with the highest PageRank"

With a three month history.

And "Page analysis" now has lists of word commonly used on your site and in links to your site.

BillyS

WebmasterWorld Senior Member billys us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 33049 posted 11:46 pm on Feb 7, 2006 (gmt 0)

>>And "Page analysis" now has lists of word commonly used on your site and in links to your site.

There are a couple of words in that list that I know I used (intentionally) but I'm surprised they made the list. I don't think they are the most frequently used words.

Tearabite

5+ Year Member



 
Msg#: 33049 posted 2:21 am on Feb 8, 2006 (gmt 0)

The feature hasnt been working for me for the last several hours.. anyone having trouble with it?

abates

10+ Year Member



 
Msg#: 33049 posted 2:47 am on Feb 8, 2006 (gmt 0)

Is it just me or are the features I mentioned above only working for main domains and not subdomains?

Lyndsay

5+ Year Member



 
Msg#: 33049 posted 4:20 pm on Feb 17, 2006 (gmt 0)

Sorry to drag up an old topic, but I don't even have the new "page with highest PR" stats. I do see the new robots.txt tab though.

Any ideas on why I'm not seeing some of the new features? Sitemap is updated, downloaded, and crawled regularly.

BillyS

WebmasterWorld Senior Member billys us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 33049 posted 5:05 pm on Feb 17, 2006 (gmt 0)

>>but I don't even have the new "page with highest PR" stats.

Some of the new features are on old pages. The above information is on the Crawl Stats tab.

jadebox

10+ Year Member



 
Msg#: 33049 posted 5:17 pm on Feb 17, 2006 (gmt 0)

The robots.txt page in Google Sitemaps is misleading because it makes it appear that Google's crawler was not able to find the robots.txt file.

I was confused (and concerned) because the page said I didn't have a robots.txt ("We didn't find a robots.txt file on your site.") even though I have one. Finally I discovered that clicking the "Check" button forces the utility to load the robots.txt file. The page is displaying information based on the tool's downloading of robots.txt, not Google's bots.

-- Roger

Lyndsay

5+ Year Member



 
Msg#: 33049 posted 4:00 pm on Feb 21, 2006 (gmt 0)

Some of the new features are on old pages. The above information is on the Crawl Stats tab.

Yeah I know... but it's not there.

Thanks anyway :)

suzie250

5+ Year Member



 
Msg#: 33049 posted 8:36 pm on Mar 24, 2006 (gmt 0)

FYI - this feature does not catch an error like this:

Disallow:/ example.php

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved