homepage Welcome to WebmasterWorld Guest from 54.161.166.171
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Robots.txt added to Google Sitemaps
New feature for sitemaps...
BillyS




msg:759680
 12:41 am on Feb 7, 2006 (gmt 0)

Gives you the ability to test out changes to your robots.txt file and the response given for each of your webpages as you make changes to the robots.txt.

You can also test different user agents. The key here is to test changes before going live with a new robots.txt

Pretty cool feature!

 

BillyS




msg:759681
 8:04 pm on Feb 7, 2006 (gmt 0)

Geeze, I thought this would generate some dialog...

abates




msg:759682
 9:12 pm on Feb 7, 2006 (gmt 0)

Interesting!

Also on the "crawl stats" page at the bottom I now see:
"Your page with the highest PageRank"

With a three month history.

And "Page analysis" now has lists of word commonly used on your site and in links to your site.

BillyS




msg:759683
 11:46 pm on Feb 7, 2006 (gmt 0)

>>And "Page analysis" now has lists of word commonly used on your site and in links to your site.

There are a couple of words in that list that I know I used (intentionally) but I'm surprised they made the list. I don't think they are the most frequently used words.

Tearabite




msg:759684
 2:21 am on Feb 8, 2006 (gmt 0)

The feature hasnt been working for me for the last several hours.. anyone having trouble with it?

abates




msg:759685
 2:47 am on Feb 8, 2006 (gmt 0)

Is it just me or are the features I mentioned above only working for main domains and not subdomains?

Lyndsay




msg:759686
 4:20 pm on Feb 17, 2006 (gmt 0)

Sorry to drag up an old topic, but I don't even have the new "page with highest PR" stats. I do see the new robots.txt tab though.

Any ideas on why I'm not seeing some of the new features? Sitemap is updated, downloaded, and crawled regularly.

BillyS




msg:759687
 5:05 pm on Feb 17, 2006 (gmt 0)

>>but I don't even have the new "page with highest PR" stats.

Some of the new features are on old pages. The above information is on the Crawl Stats tab.

jadebox




msg:759688
 5:17 pm on Feb 17, 2006 (gmt 0)

The robots.txt page in Google Sitemaps is misleading because it makes it appear that Google's crawler was not able to find the robots.txt file.

I was confused (and concerned) because the page said I didn't have a robots.txt ("We didn't find a robots.txt file on your site.") even though I have one. Finally I discovered that clicking the "Check" button forces the utility to load the robots.txt file. The page is displaying information based on the tool's downloading of robots.txt, not Google's bots.

-- Roger

Lyndsay




msg:759689
 4:00 pm on Feb 21, 2006 (gmt 0)

Some of the new features are on old pages. The above information is on the Crawl Stats tab.

Yeah I know... but it's not there.

Thanks anyway :)

suzie250




msg:759690
 8:36 pm on Mar 24, 2006 (gmt 0)

FYI - this feature does not catch an error like this:

Disallow:/ example.php

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved