Welcome to WebmasterWorld Guest from 18.104.22.168
Hi reseller (and everyone!),
I'm here and reading what everyone has to say (and taking lots of notes about what you would like). I'll do my best to answer questions, although I can't promise snazzy videos like Matt put together. :)
Matt and I think Google Sitemaps is a good vehicle for webmaster communication because it's a scalable way to get input from you and give information back to you. You know how we like scalable things at Google.
There's definitely a lot more we can do regarding communication, much of which has been touched on in this thread. I know that with the name "Google Sitemaps", it may sound like the product is about creating and submitting Sitemaps, and these other communication features are an afterthought. But in reality, our whole team is looking at better ways to communicate with webmasters, and the creating and submitting Sitemaps is just one small piece of that overall communication.
We want this product to include all the tools and diagnostics you need to learn everything we have available about your site's crawling and indexing, as well as include ways for you to provide input to us (beyond submitting Sitemaps and filing reinclusion requests). So keep your ideas coming...
[edited by: tedster at 5:37 am (utc) on Aug. 2, 2006]
Its a symbiotic relationship, both gain as long as both work together.
the more you work with Google, the more you gain from visitor count
Nope, wrong argument for this discussion.
As per google's suggestions, whitehat webmasters build their sites for visitors (not Google algo's) and Google gets free content and the ad revenue from that. And good sites "supposedly" get higher rankings and more "free" visitors in return.
That's an even business exchange.
Google gets marketing research and user behavior metrics worth billions by giving out "free" analytics and sitemaps.
Again, an even business exchange.
What's the return value to webmasters giving advice, making G's results more relevant, enhancing Google's visitors experience and ultimately their revenue?
Let's take the usual scenario. A webmaster builds an "authority" website. It doesn't rank for a year and then suddenly ranks #1 for 3 years straight. An update happens and suddenly it's nowhere to be found. The site was always an authority site so what happened the first year it didn't rank and why is it gone now?
Either G was wrong when it was ranking it #1 or it is wrong to not rank it now. Either way, their visitors somehow are getting a sub-par experience.
Why should webmasters even deal with this when they can use the same time and energy to build blackhat sites that cost Google time and money as they do building whitehat sites that make Google money?
The question remains:
Is Google going to only take from the content providers who make them money and make the web a better place or will they spend a little of that time and/or money to communicate in a meaningful way to that same community and create a win-win relationship for themselves and their visitors?
Edited to add - In fact, many of the blackhatters I know were whitehatters who built authority sites, lost rankings, got fed up, and decided to "guarantee" their income.
Novice webmasters rarely know how to build 5 million page spam that gets past filters, ya know? (which I see has re-entered some of the DCs)
Instead of spending the summer working on so called new innovations and other changes, try fixing every thing you've broken. For example, if
>>22.214.171.124 has some newer infrastructure that makes site: queries more accurate
why don't you make Google.com get it's results from there?
Right now, when you do a site: search on google.com for a lot of domains, the results are screwed up while they are great in the datacenters.
And how about addressing and fixing the sub-domain SPAM stuff. That's sad when it has to get to the front page of Digg for you to do anything about them, and the SPAMers are still having no trouble making new sites and getting massive listings in SPAMoogle using there sub-domains.
And if you claim to be working on better communication, how about branching out to some of the other popular webmaster message boards. Then you can get even more ideas and suggestions.
">>126.96.36.199 has some newer infrastructure that makes site: queries more accurate
why don't you make Google.com get it's results from there?"
While awaiting a response from the Googlers, allow me to give my own opinion on that one.
As you know GoogleGuy speak Googlish and that means one have to "translate" some of what he says sometimes. GG mentioned recently that he will talk Datacenters at the end of this summer. Which could means that at present they are still working on developing-testing-tweaking algos and filters for the new infrastructure. Which means that nothing is final on the serps at present. And which means that GG expect that to finish at the end of this summer where we will see more stable serps and "functional" operators.
However, we have a little problem here. Which summer GG was talking about? USA summer or the Danish summer :-)
Personally I do not think that webmasters should demand or expect to get personalised answers to their questions. This is not possible, as their support team will get flooded with messages each time they make a significant update to their algo.
Also such a communications method will be open to abuse, as people will start sending emails even with minor algo changes and it simply can not work (given the amount of webmasters)...
Another idea is to make the spam/contact forms much more complex, so that those who do not provide very specific information about the nature or the source of the problem are automatically filtered out in favour of those who make the effort and provide clear evidence.
I am sure it is in the best interest of google to improve it's search results, but the current communications channels available are (I suspect) flooded with reports and questions from frustrated webmasters (such as myself). There needs to be a different approach, unless google are up to doing a "telephone-center"-like office with hundreds if not thousands of googlers answering webmaster questions.
When a company holds the biggest market share and the second player is far behind, they don't even need to worry about customer complaints. You know why, because they don't see financial hit on their spreadsheets. So, even if a good employee tries to tell the management that they should spend more money in training customer service people and researching and fixing the communication gaps, the immediate response is going to be we don't see any financial hit and therefore this is not a significant problem. Lets not spend resources on it.
You take any big company, online or offline, the problem is the same. The guy who takes decisions doesn't know what is going on with us simple and small customers and doesn't even care most of the time. The employees who know our problems can't and sometimes won't report to him the problems.
That is why I laugh whenever someone calls a customer service rep and yells at them and tells them that they are going to give the businnes to their competitor. Because, I know that if you do the same with just a wall, you might get a better response.
So, now Google is just in the same boat/path as any other big company. They are making their money. They will address issues of all the big big clients (I don't know who they are, though) and not small webmasters.
You can not say they are ignorant. Also to those who say that adwords/adsense affect SERPs, I think you are wrong here...
I can not believe that Google favours sub/parked domains with adsense over a site with unique content. They are much more likely to make more money from a successfull site than they do from parked ones.
"Good morning Reseller... (cloudy and chilly in London unfortunately)..."
Good morning to you too. Sunshine over Denmark.. though expect some rain late in the afternoon :-)
"Personally I do not think that webmasters should demand or expect to get personalised answers to their questions. This is not possible, as their support team will get flooded with messages each time they make a significant update to their algo."
IMO, the most important "support" communication channels to seek are the following 3, which could be included in Sitemaps:
- Filing Reinclusion Requests. That also covers sites which are not deindexed but rather being affected such as lost big parts of pages, turned to supplementals or having cannonical problems (as that of my good UK friend Dayo_UK ;-) )
- Reporting problems with Sitemaps
- Spam Reporting
Have I forgotten to mention anything else ;-)
It is important to have it in sitemaps, as googlers can establish which is the site affected by this, provided that generally sitemaps are related to a specific account. I think they can easily determine which accounts not to trust (thus reducing their workload).
Also for serious search quality issues, the support team does pay attention...
(reseller it is hard to come to terms when you're from greece with this weather... ;) )
What I'd like to suggest is that there be a way of enquiring about apparent anomalies in the sitemaps reports which might then lead to better or more useful information so we can interpret them better. In the long run this would mean less need for direct contact which is obviously very difficult to handle on the scale that you're working on.
I'll give you an example. On a client's site which we assessed for SEO there had previously been a duplicate content problem. Over a period of time we managed to eradicate this (we think!) but while the site has decent rankings in MSN and Yahoo the Google rankings have been very slow to appear, and those that have have not included their two most important keywords. Looking at their sitemaps report the "Page analysis - Common words in your site's content" section doesn't include those two words despite them being (justifiably) the two most common on the site.
We've speculated that this might be a reflection of some sort of penalty that is still in operation but we have no way of confirming if that is the case (we did email Google support but haven't received any reply). Certainly we can see no other reason for the two words being ommitted.
If there was some method for enquiring about this type of anomaly then the site reports would become very much more useful, while the feedback might well alert you to aspects that aren't working as intended.
I don't believe that it aided me in any way in the year which I used it, now I'll find out if there actually was any value in beating my head repeatedly against the Google wall.
What a sad, sick joke. Just like most of Google's other notable failures (see Froogle and Google Base).
I think the Googlers at the plex - who deal extensively with code and algos and abstract concepts - have to begin to realize that webmasters are real people with real world issues running real businesses on the internet with real bills to pay. They need to stabilize their operation and stop yanking our chains.
It's also fair to say that dealing with an entity as large and as secretive as Google is frustrating and generally a waste of time. For an example, try the US federal beaurocracy. SS:DD
Lets face it, communication with G is awful, it must and will improve to some degree because G knows if it doesnt address this soon, it is going to get hurt. However, dont get too excited at the improvement! This is more likely to take the form of "Glasnost" - Google opening up to known issues and when they intend to implement/fix them.
Personal help, advice to correct problems or website analysis is unlikely ever to happen because the cost in man hours could never bring a return on investment. Also the logical secrets as to how sites end up ranked are likely to be kept between a close inner circle and a stack of code. Lots of humans in the know would eventually give out little tidbits of advice and all hell would break loose around the world trying to use this information in one form or another and G would suffer.
This just leaves us one hope. The automated response option. By this I mean we can expect to see enhancements to the Sitemaps interface introduced over the next year or so.
These enhancements could eventually lead to new page options similar to the Page Analysis pages but with little red bars creeping across to show negative level indicators for various indisgressions. e.g. Repetitive, spamming, links warning, domain issues etc...
As Google has already calculated all this information during analysis of your pages it is easy to simply update the values it wants to reveal in your Sitemap statistics...
...this will be a whole lot better than nothing.
We're definitely looking at how we, as a company, can communicate more with the webmaster community. Google Sitemaps lets us do that in a scalable way. For instance, we show all errors we had crawling a site to anyone who signs up and verifies ownership.
We just fixed a problem with the display of robots.txt status. Let me know if you continue to see "not found". As for verification trouble, could you give me more details? If possible, it would be great if you could post your site URL in the Sitemaps group so I can look into it further.
Just saying you're going to communicate with us isn't going to cut it. For example, you can start out by talking about issues that almost every webmaster is concerned about. For example, the site: search which is used almost 100% by webmasters would be a great place to start. As you work on fixing that bug, geting google.com to show what the datacenters show, give us updates and estimates as to when it'll be fixed. Just like you got a blog for the SiteMap, maybe make a blog that allowed comments, where you give up updates on fixes to search results, which would include for example the sub-domain SPAM stuff, and explain what a 'bad data push' is, stuff we're all interested in but Matt Cutts won't even talk about. That would give us way more confidence in Google than what you're all doing right now, which looks like it's just trying to fix your reputation from all the major bugs BigDaddy has caused.
And as a bonus just for fun, give us a few days notice for PR updates. It's possible, as Matt Cutts did that once. And do a PR update once a month. :)
[edited by: Jesse_Smith at 11:40 pm (utc) on Aug. 3, 2006]
Something like the MS Knowledgebase would be cool.
Given Googles global reach and impact, what is there is either too vague or simply not there.
Or a simple update notification page ... lone employees posting in forums and in videos is nice but doesn't reach imo everybody who should know when Step Daddy goes awol.
Very simple like my ISP, .. server outage will occur at x time etc etc .. new data update pending.. The way it's now is inefficient.
"Knowledge base Google" etc typed into Google doesn't bring anything up so if it's there, it would be interesting to know where.
In addition to the latest suggestions contributed by fellow members, I wish to summarize the previous suggestions which could be included in Sitemaps.
- Filing Reinclusion Requests. That also covers sites which are not deindexed but rather being affected such as lost big parts of pages, turned to supplementals or having cannonical problems.
- Reporting problems with Sitemaps
- Spam Reporting
- DMCA complaint form
You are more welcome to suggest other items, of course ;-)
Wish you all a great weekend!
The green bars in the sitemaps statistics seem a little bit outdated. For instance, my robots.txt is
for more than two years now. the sitemaps statistics still shows a significant green bar, but the details say "no problems."
These green bars also show more than half of my pages as HTTPerrors/not found. A request on the details shows less than 10 pages in total. One is a directory, giving a 403. How can I tell googlebot that this directory is not indented to be indexed, but only that one widget.html-page in that directory? And this leads to my final question:
You offer a csv-download of the error-tables. For very long urls with a number of subdirectories this table lists data (urls)abbreviated by dots.... Particularly for webmasters working with script-languages it would be very helpful if these tables were downloadable in a very simple but complete format, so that they can be imported into an error-correcting script.
"Spam reporting? You must have missed my reference to Sisyphus. Maybe if I phrased it as a "fruitless and hopeless labor" it would make more sense."
Its just because I wish that fellow webmasters who still wish to report spam, to be given such option on the Sitemaps. The right to choose, you may call it :-)
Nice to see you participating. While I would really like to have a better way to comunicate with Google, I just do not understand why it has to be sitemaps. I really do not want to use it at all. Nothing against you, or your department, it's just something that I do not think i need.
I use Google Analytics and I feel that is more than enough for me. I have mentioned in previous posts that I do not feel comfortable using something that so many people have said something happened to their site and many think its because of sitemaps.
Below is my understanding of beta.
"1. Mostly working, but still under test; usu. used with “in”: in beta. In the Real World, hardware or software systems often go through two stages of release testing: Alpha (in-house) and Beta (out-house?). Beta releases are generally made to a group of lucky (or unlucky) trusted customers."
Sitemaps is still beta.
My question is this.
If there are thousands of webmasters that have been running sites that are not in any kind of testing phase, why should it be required to have these thousands of webmasters who may or may not make a living from those sites have to subject them selves to a testing enviroment to submit possible problems?
Correct me if i am wrong, but I am pretty sure sitemaps is still beta.
I have a good understanding of beta as I am in the software field. I know before we ask hundreds of thousands of people to use something we offer, we have an inhouse alpha (mostly dev) then beta inhouse and we do offer beta to a very small amount of users. Then after the beta period has expired, we then offer the working version to hundreds of thousands of people.
Having web masters as a whole is not really fair on them because you are basically forcing them to be a guniea pig for your beta if they want to have a resolve to something that might not even be the fault of their website? Does that seem fair at all?
** Enable webmasters to provide detailed critiques of Google’s search results for specific search terms (keywords).**
Instead of an essay style form, this should be a carefully structured, detailed survey form – using check boxes and radio buttons, which would provide data that Google can easily compare with data obtained from other sources (e.g. ordinary user data) using entirely automated processes.
The console could require the webmaster to cut and paste a specific page of SERP results into the form. That would allow Google to figure out exactly which algorithm set was used to generate the SERPs. As well, it would allow the data to be parsed and used to ask the webmaster a series of more and more specific questions on the aspects of these SERPs that are of particular interest to Google.
For instance, if Google knows that certain documents appear in the result set due to an ongoing test it is currently conducting, the sitemaps console might ask the webmaster a series of detailed follow-up questions about that specific portion of the result set, perhaps including an opportunity for the webmaster to visit the site in question and tell Google if there is a different page within that site which would be superior to the one selected by Google for display, etc.
Similarly, it could allow the webmaster to provide some feedback concerning the document, and suggest other documents which are more relevant, or have superior content. For instance, it could ask the webmaster whether there is a page on their own site that they think would better satisfy users who are conducting that particular search.
The console could then compare the two documents, and analyze various data Google has collected concerning the webmaster's own page, and then ask a series of specific questions concerning why the webmaster thinks their document is superior (e.g. the information on my site is more accurate; my competitor hasn't updated his page in 3 years...)
Note that Google would benefit from the receipt of very detailed, structured quality control data concerning its SERPs. This data would help fill weaknesses in its current data collection efforts at minimal cost to Google, and without requiring it to hire more employees.
There would also be some benefits for Webmasters:
Webmasters would benefit by having a mechanism available that they can use to complain to Google about specific flaws in the SERPs. Some of these flaws might be affecting their own niche(e.g. those junky sites above them) some might be affecting their experience as an ordinary user (e.g. I was trying to find unbiased information about this disease my friend’s been diagnosed with, and the entire first page was filled with junky sites selling pills).
Webmasters will benefit when their feedback leads to improvements in specific SERPs they care about (once their feedback data is correlated with various other data sources, the underlying source of the problem is identified and algorithmic solutions are developed).
In effect, all the players in Google's game will have an incentive to keep each other honest. And, this would help level the playing field between those who try to play by Google’s rules, and those who would rather focus on trying to break the rules as profitably as possible.
Webmasters will also benefit from being able to build up their trust rating with google (assuming their feedback is honest). In turn, this would increase the chances that Google will treat them like one of the “good guys” and be willing to give them a bit of personal attention, or their automated response system in the site console will provide clearer hints about what they are doing wrong (e.g. 57% of your inbound links are coming from sites we don't trust; you need to make some new friends or move into a better neighborhood... )
Answering Vanessa, I deleted my sitemap and removed the file from my root directory. After 6 months of trying to verify, I had enough.
Let's get this straight. Google, primarily a search company, and a public company, has an obligation to its shareholders to deliver quality search results. I would expect them to spider or crawl my site with the utmost in efficiency without assistance from me or from hundreds of thousands of webmasters.
Google's crawlers, bots, agents and algos should adjust to the vagueries and variations of websites, not the other way around. That's why investors pumped billions of $$$ into Google stock, figuring Google would hire the best talent to press their already enormous advantage in search technology and market share by hiring the best talent and advancing their product.
But what has Google done with those billions? How should we measure their performance (hint: not by the price of their stock)?
From my point of view (admittedly, I am not an engineer or management expert, but a lowly small business owner, webmaster and writer), Google has squandered their advantage and some of the investor's money by flying off in all directions with new applications like Google Base, Google Mobile, Payments, Sitemaps, Pages, etc., etc., etc., ad nauseum, instead of focuing their efforts on SEARCH (and developing an income stream other than AdWords).
Maybe I'm too critical, or cynical, but I think Google has not focused their mission well enough and despite the hype and hoopla, was a better company before they went public.
Public companies are subject to much closer scrutiny and G is not immune from perceptions by the investment market, but what's telling are threads like this and hundreds of others, containing complaints, unanswered questions and a continuing flow of somewhat indecipherable information and concern from Google reps with little substantive change.
The SERPs are still messy, MFAs proliferate, sites drop in and out of the SERPs without explanation, yet we're supposed to trust Google with our sites, our business, our livlihoods, and use all of their tools, including sitemaps. Unfortunately, not all of this is working for everybody.
Some of us aren't exactly buying the corporate line and are getting just a touch cranky and impatient. Focus on your core competency, if you can still find it.
Also (not to take away from WebmasterWorld)they have greatly expanded their google groups pages which have several new categories and replies from actual Named Google employees.
I say, Bravo, a great step to better communication.
As for this morning the way sitemaps presents information about my website seems to have changed a bit: the crawling statistics now only shows page-rank distribution, and detailed diagnostics of the crawling process now shows a correct "zero" for robots.txt restriction.
Maybe coincidential, but I assume someone has taken a closer look at these things due to the feedback given in this thread.
I'm still thinking about that 403-error on a directory. Does it really make sense to crawl a directory? Shouldn't the spider at best search for an alternative index-page, and, in case no such index-page exists, skip the whole URL?
I there any mechanism in the crawling/indexing process, by means of which a permanent broken/wrong link from another website (pointing to a page on my site, that doesn't exist) will some day be excluded from the error statistics?
I searched for some of the 404ed pages in quotes, but got no results, so that I can't even find out whether there is something wrong on my own site, nor contact another webmaster. Could you additionally report in the diagnostics from what source the attempt to crawl such 404-page is drawn?
So it seems our discussion on this thread has already improved Google-Webmasters communications a great deal on Sitemaps. Remarkable changes!
And of course there is that link at right side on top which I like most :-)
- Download data for all sites
- Report spam in our index
- Submit a reinclusion request
Hopefully another item would be added to above list
- DMCA complaint form
BIG Thank You to my WebmasterWorld fellow members for great contributions and generous sharing spirit.
Thanks a bunch to our Googlers friends for taking the time to participate in this thread and for the efforts to improve Google-Webmasters communication!
Special thanks to Matt "The Kentuckian" Cutts for his great Videos/info contributions.
Wish you all a great day and a positive constructive discussion.
[edited by: reseller at 7:10 am (utc) on Aug. 5, 2006]
Now if we choose the preferred domain would we still need the 301 redirect? I know we should be safe and keep it but just throwing it out there. Also would any links to the non www version be included in the calculations for the www version and not just followed in the correct version.