Forum Moderators: Robert Charlton & goodroi
Figured I'd make a post here since I haven't in a while. I recieved an email from someone at google telling me the benifits of allowing the google spider to index my site. The site in question has a robots.txt file disallowing all search engines.
The email includes the URL in question and states that atleast 50% of internet searches come from google and its important to allow google to spider the site.
First reaction was, automated responce? Did I send an inquiree to them? Nopes certainly not for this domain so I sent an email back asking why the email and I expected it to bounce. To my surprise this morning I got a reply back saying they want to include the site because it was identified as being highly ranked.
New sort of team at google trying to get people uninformed and disallowing google to switch back and allow google to spider the data?
C
In the last two years the traffic from google was steadily rising. It's been exciting to see the growth. Then the update came and bam! O visitors, nothing!
I'm tired of this game. There are now three other major search engines to work with, I've chosen to give them my information.
Since then I've banned the googlebot on all 12 of my sites. No use in them taking up bandwidth for nothing.
Google could simply program its crawlers to ignor robots.txt if the going got tough...
Doubtful - someone would unleash a program to ignore their robots.txt and submit multiple requests at a rapid rate - and for each time they have to server a request there is a lot more calculation involved... that would be exactly the same.
Also - I'm sure they'd leave themselves open to a lot of scrutiny and possibly even a court case.
Na, they won't do that.
I am also seriously considering blocking the g-bot, not just in robots.txt - not that it obeys it anyway, just by restricting an entire IP Block. It caches the pages, and that is it. No results, what so ever, so why bother. And No, I am not a spammer or use any bad tactics to encrease position in SERPS - and to my standards do not do anything to confuse the Bot - it confuses it self. I don’t like confusion.
Perhaps I will too get an e-mail invite one day that I could Smile at.
When you bring that serious problem to G$'s attention, they then sabotagingly recommend to you that you should instead use an ALLOW command -- even as they tell you that other SE's might not recognize the command!
ALLOW is not part of the Robots Exclusion Protocoal -- and G$ knows it!
So, G$'s patheticly faulty bot causes a problem and instead of fixing that, they then try to sabotage you in other SEs by recommending you do something they know can hurt you in other SEs.
ALLOW
I may do the same (ban the bot).
What will happen to the content when we allow gbot again? Will google think that sites that copied or hijacked our sites had our content before it was posted on ours?
This would surely make an interesting story to report. I don't understand why Google isn't saying SOMETHING about this, it's pretty obvious there's a problem, even to the casual searcher. My Brother even commented to me that he's had friends say Google seems to be broken.
Andy
How will ALLOW hurt you in other search engines?
The ALLOW command is not part of the robots.txt standards.
DISALLOW is.
While some SEs like G$ might use this non-standard protocol, it will not work in others.
So, if you DISALLOW everything else to only ALLOW something specific, then those SEs, who do not recognize the non-standard ALLOW command, will fully ignore what you had actually wanted to ALLOW.
Why should I allow Google to count all of my pages as part of their index, and include them in their total pages indexed on THEIR site, when doing relevant searches won't list any of my pages?
Just because you're not getting the high listings you want, what good does it do you to remove yourself from the listings altogether?
Just because you're not getting the high listings you want, what good does it do you to remove yourself from the listings altogether?
However, when a site isn't doing anything unethical, or against the Google guidelines, and its pages relate to the search query, those pages should be available to searchers. When there is a page on the internet with a unique title, and that title is searched for, if it's the only one with that title, it should be listed first! It is the most relevant, assuming the page title properly describes the contents of the page.
As it stands now, Google has my pages in its index, and no one can find them when doing applicable searches. I receive no benefit, but Google can add my pages to its total of pages indexed, and is using my bandwidth to do it.
Google needs to be held accountable for what it does. Regardless of whether or not what it does is its own opinion, if it is to be relevant it needs to return relevant results, that is what a SE is supposed to do. Right now, when I search for my unique page, "Antique Egyption Blue Widgets" I'm not finding it in the SERPs.
I do see directories that mention the page, I see newsfeed sites that mention the page, I see other pages linking to my page, I see pages with "Blue Widgets" "Antique Blue" "Antique Widgets" "Egyptian Widgets", etc., but I don't see MY PAGE, which is the only complete match for the search query. In fact I see everything BUT my page, it seems. And it's the only one with that title.
By saying goodbye to Google, it no longer is a consideration, or a concern. I can maximize my site for the other SEs with more stable results. Google is like a big drama queen who has to make everything a big deal. The other SEs just seem to go about their business without all the fan fare, and today, they are returning more relevant results. So, six months from now when everyone is screaming about the latest Google update, it won't have any effect on me at all.
When you get in your car and turn the key, you expect the engine to start. Not sometimes, not every once in awhile, but every time. When I go to Google, and type in a search query, I expect to see relevant results, the best ones first, every time.
No one would keep a car that won't start, why keep a SE that doesn't work? Google hasn't "started" for me in 2 months! And they won't even communicate. All they have to say is, "Webmasters, we have a problem. We know what it is and we're trying to find the best way to fix it." No big deal.
I can understand it from Google's point of view. They want to continue being the dominant SE, so they need to cover as much of the net as possible.
There are plenty of sites out there for Google to index--those in the sandbox for instance.
On the whole Google has been good to my content sites, so I'm not about to ban it. But I'm starting to wonder about the benefits of putting in large bot-free sections on my sites.
What you have experienced is the G$ blackmail of authority sites into paying for AW.
G$ finds the true authority sites and then pushes them down the SERPs, knowing that most of such sites will find it important to pay G$ their AW blackmail.
Beware the G$ cult worshippers here who will lay all kinds cult mind control tactics to suggest that that doesn't happen, but it does. It absolutely does. G$ is simply a liar, and nothing they say can be taken as honest these days. Others of us have experienced that blackmail firsthand too, so the facts are the facts. (And you can ignore the G$ cultists and their silly irrelevant accusations of supposed "conspiracy theory making" etc -- they are only trying to promote their cult using whatever lies they can come up with.)
The community topic I deal with has been purposely sabotaged by G$ ever since Oct 2003 -- and despite numerous efforts to correct whatever might be "wrong," such efforts are all a complete waste of time. G$ is simply playing with sites trying to "correct" things -- if anything was ever really wrong anyway. (There really is no correction possible to be accomplished when G$ decides they want to play with an authority site.) In this keyword, G$ still gives the most useless and dishonest "natural" SERPs for the keyword, and still prevents all the on-topic sites to appear.
Anyway, the choice for you is, do you have the freedom to not pay that blackmail or not? If you are fortunate enough to have the kind of audience who will indeed go to another search engine to find you, then perhaps you can avoid the G$ extortion into AW. But if you can't (or until you do), then G$ has purposely given you no choice, showing how truly that G$ can "do only evil."
They rather say, have your SEO company look into this and this, and that, etc etc.
After the sandbox of my site,
I stopped AdWord advertisement six months back. (click fraud).
I blocked googlebot last month. It was eating 300 MB/month and not any decent traffic.
This month I have contacted other ad companies for the replacement of AdSense. (unreliable pay)
I have grown to love msn(yuck) and yahoo search.
Just when did google become such money wh*re?
Sometimes I just wonder what these computer geeks must be thinking. Its just history repeating itself.
Only google product I really like is gmail. But I wouldn't be suprised if google screwed up that one in the future.
I wish more webmaster followed. This is the only way we can get back our Internet.
Power corrupts and absolute power corrupts absolutely.
IMHO bandwidth costs are not usually an issue, (to other posting, geez, if there are that many pages being spidered, and you cannot moneterise it to pay for the bandwidth, drop me a line, I might be able to help)
Privacy I can understand, as bhartzer said.
Why do Google want your site in the index?
Which seems to puzzle you :)
can we keep on track?
Sad but true and these are non SEO's
Hollywood
Bonus Question (20 points)
G$ already ignores/disobeys proper Robots Exclusion Protocols. Googblebot is such an amateur that it can not read multiple lines of DISALLOW commandments in a robots.txt file.
When you bring that serious problem to G$'s attention, they then sabotagingly recommend to you that you should instead use an ALLOW command -- even as they tell you that other SE's might not recognize the command!ALLOW is not part of the Robots Exclusion Protocoal -- and G$ knows it!
question:
what is the most likely cause of the observed behaviour?
answer:
[ ] a) a shortage of phd's out there
[ ] b) less interesting than personal free time projects
[ ] c) less important than gmail, orkut, what have you
[ ] d) all of the above
wildcard answer:
[ ] *) defer for three more updates
+++
Would others see your site as leading?
Could it be that others would use the Google toolbar to tell Google that they like your site?
Could it be that others are giving Google some kind of feedback telling that they miss your site in the SERPS if it is not there?
The google bot is on my site right NOW! Indexing every page. It visits every now and then but I hardly get ANY hits from google. My SE hits top three stands at:
MSN - 49%
Yahoo - 21%
Google - 12% [including adwords - about $10 a day]
Others - 18%
I am a bit worried though cause my income from my website is dependent on adsense and G seems to be loosing ground everyday. Waiting for a better paying ad network... And yes... last cheque received from google was 2 month's ago... so I'm not particularly happy.
If you're spending $10/day in adwords and your last adsense check was 2 months ago, then it would appear to be obvious that your income is less than $100 per month (min payout) yet your expenses would be 30x10=$300 per month.. so it's time to stop using adwords!
It is also the first month ever that yahoo tipped the scale referral wise (against google)...
Is Kruoogle losing its charm?
Are more average joe's resort to other SE's to find what they are really looking for?
I think they do. It's just too tedious to shift through the piles of junk Kruoogle is serving lately.
Who's in for a "Google Free Site" with its own little logo, webring and a page explaining why more and more webmasters are blocking the google bot….this should/will hit prime time news in no time.
Maybe we should start a yahoo group on this topic. Lets teach them PHD's to stop messing with the web's heart & soul....us webmasters.
My point was that I get visitors from Google ONLY from adwords which is not good at all... I expect better returns from google search engine. I have no complaints from adsense [except for the late payment - the cheque probably got lost (AGAIN) this is the fourth time in my ten months with the program.
Some how I am getting the feeling that Google is changing into an ebay... a Bid war with most promoting adsense and some promoting products and some promoting adsense for the adwords for those products :-D
Regards,
R
[edited by: roycerus at 4:59 pm (utc) on Feb. 20, 2005]
That's what I've heard about that. Does Overture match up to Adwords?
So they’ll have:
1) Gmail
2) Bad SERPs
3) Autolinking in toolbar
4) Groups
5) Images
6) Maps
7) Thousands of books
8) Blogger
9) Googlepedia
And the list goes on....
They'll be crazy to continue and pay Adsense publishers at the same rate they are paying now once all their services are in place. Why would they need us? Their shareholders will just vote us out (un-needed liability).
We Publishers currently doing the PR work for google and pave the way, once they feel it is done and their entire master plan is in place you will see the end of Adsense.
Enjoy it while it lasts and don't quit your day job.
Whatever happened to Adsense earnings starting 1st of Feb is a good example of how kruogle can cut publishers earning by half, overnight and with no prior warning.
Kruogle at its best.
