Welcome to WebmasterWorld Guest from 54.80.198.173

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Inclusion in the Google Index

E-Mail sent to me from Google.

     
6:46 pm on Feb 16, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 5, 2001
posts:2730
votes: 8


Greetings WebmasterWorld :)

Figured I'd make a post here since I haven't in a while. I recieved an email from someone at google telling me the benifits of allowing the google spider to index my site. The site in question has a robots.txt file disallowing all search engines.

The email includes the URL in question and states that atleast 50% of internet searches come from google and its important to allow google to spider the site.

First reaction was, automated responce? Did I send an inquiree to them? Nopes certainly not for this domain so I sent an email back asking why the email and I expected it to bounce. To my surprise this morning I got a reply back saying they want to include the site because it was identified as being highly ranked.

New sort of team at google trying to get people uninformed and disallowing google to switch back and allow google to spider the data?

3:59 am on Feb 19, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2002
posts:2579
votes: 0


I may do the same (ban the bot). The hijacknig problems will apparently not be resolved, because they don't care. They lose out on a great resource.

C

6:12 am on Feb 19, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 12, 2003
posts:84
votes: 0


This last update I got hit pretty hard. Content on the site was being added daily, there was no SEO involved, the site was growing naturally with webmasters linking to the site because of the resources that were found on it.

In the last two years the traffic from google was steadily rising. It's been exciting to see the growth. Then the update came and bam! O visitors, nothing!

I'm tired of this game. There are now three other major search engines to work with, I've chosen to give them my information.

Since then I've banned the googlebot on all 12 of my sites. No use in them taking up bandwidth for nothing.

7:29 am on Feb 19, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 14, 2004
posts:52
votes: 0


Google could simply program its crawlers to ignor robots.txt if the going got tough...
7:59 am on Feb 19, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 20, 2002
posts:813
votes: 1


Google could simply program its crawlers to ignor robots.txt if the going got tough...

Doubtful - someone would unleash a program to ignore their robots.txt and submit multiple requests at a rapid rate - and for each time they have to server a request there is a lot more calculation involved... that would be exactly the same.

Also - I'm sure they'd leave themselves open to a lot of scrutiny and possibly even a court case.

Na, they won't do that.

1:59 pm on Feb 19, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1951
votes: 61


Some one once said: "what good is the phone call if you are unable to speak".

I am also seriously considering blocking the g-bot, not just in robots.txt - not that it obeys it anyway, just by restricting an entire IP Block. It caches the pages, and that is it. No results, what so ever, so why bother. And No, I am not a spammer or use any bad tactics to encrease position in SERPS - and to my standards do not do anything to confuse the Bot - it confuses it self. I don’t like confusion.

Perhaps I will too get an e-mail invite one day that I could Smile at.

2:31 pm on Feb 19, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Mar 22, 2004
posts:338
votes: 0


G$ already ignores/disobeys proper Robots Exclusion Protocols. Googblebot is such an amateur that it can not read multiple lines of DISALLOW commandments in a robots.txt file.

When you bring that serious problem to G$'s attention, they then sabotagingly recommend to you that you should instead use an ALLOW command -- even as they tell you that other SE's might not recognize the command!

ALLOW is not part of the Robots Exclusion Protocoal -- and G$ knows it!

So, G$'s patheticly faulty bot causes a problem and instead of fixing that, they then try to sabotage you in other SEs by recommending you do something they know can hurt you in other SEs.

3:18 pm on Feb 19, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1951
votes: 61


I just sent them 2 page invite letter myself, lets see what they say.
3:38 pm on Feb 19, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:May 23, 2003
posts:586
votes: 0


ALLOW

Multiman, How will ALLOW hurt you in other search engines?

I may do the same (ban the bot).

We could really start something here to get Google to wake up to their current problems!

What will happen to the content when we allow gbot again? Will google think that sites that copied or hijacked our sites had our content before it was posted on ours?

3:46 pm on Feb 19, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 11, 2005
posts:513
votes: 0


There seems to be a growing anti-Google sentiment now. During my keyword check this morning (during which I determined my site is STILL not showing up on Google searches for most keywords), I came across a site that had a familiar red circle with a slash through it - but instead of No Smoking or whatever, it said "NO GOOGLE"! The site further said to not look for them in Google - they didn't want to be there! And it went on to explain how the Google search pages were bogged down by spam and directory sites, authority sites that everyone already knew about (mentioned Amazon and eBay), and that pages with helpful, informative, original content were in many cases bured under all the garbage! It sounded rather convincing to me, and this is a pretty high traffic site, so I wonder how long before word of mouth hits the news media?

This would surely make an interesting story to report. I don't understand why Google isn't saying SOMETHING about this, it's pretty obvious there's a problem, even to the casual searcher. My Brother even commented to me that he's had friends say Google seems to be broken.

Andy

4:39 pm on Feb 19, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Mar 22, 2004
posts:338
votes: 0


How will ALLOW hurt you in other search engines?

The ALLOW command is not part of the robots.txt standards.

DISALLOW is.

While some SEs like G$ might use this non-standard protocol, it will not work in others.

So, if you DISALLOW everything else to only ALLOW something specific, then those SEs, who do not recognize the non-standard ALLOW command, will fully ignore what you had actually wanted to ALLOW.

6:32 pm on Feb 19, 2005 (gmt 0)

New User

10+ Year Member

joined:Feb 19, 2005
posts:5
votes: 0


Why should I allow Google to count all of my pages as part of their index, and include them in their total pages indexed on THEIR site, when doing relevant searches won't list any of my pages?

Just because you're not getting the high listings you want, what good does it do you to remove yourself from the listings altogether?

8:23 pm on Feb 19, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 11, 2005
posts:513
votes: 0


Just because you're not getting the high listings you want, what good does it do you to remove yourself from the listings altogether?

What good does it do to stay? Google uses up my bandwidth, gets to claim my pages in its index, but I get nothing from it. There needs to be some accountability here. If a site is intentionally trying to influence the SERPs with black hat techniques, then they deserve to be caught and dealt with. I have no problem with that.

However, when a site isn't doing anything unethical, or against the Google guidelines, and its pages relate to the search query, those pages should be available to searchers. When there is a page on the internet with a unique title, and that title is searched for, if it's the only one with that title, it should be listed first! It is the most relevant, assuming the page title properly describes the contents of the page.

As it stands now, Google has my pages in its index, and no one can find them when doing applicable searches. I receive no benefit, but Google can add my pages to its total of pages indexed, and is using my bandwidth to do it.

Google needs to be held accountable for what it does. Regardless of whether or not what it does is its own opinion, if it is to be relevant it needs to return relevant results, that is what a SE is supposed to do. Right now, when I search for my unique page, "Antique Egyption Blue Widgets" I'm not finding it in the SERPs.

I do see directories that mention the page, I see newsfeed sites that mention the page, I see other pages linking to my page, I see pages with "Blue Widgets" "Antique Blue" "Antique Widgets" "Egyptian Widgets", etc., but I don't see MY PAGE, which is the only complete match for the search query. In fact I see everything BUT my page, it seems. And it's the only one with that title.

By saying goodbye to Google, it no longer is a consideration, or a concern. I can maximize my site for the other SEs with more stable results. Google is like a big drama queen who has to make everything a big deal. The other SEs just seem to go about their business without all the fan fare, and today, they are returning more relevant results. So, six months from now when everyone is screaming about the latest Google update, it won't have any effect on me at all.

When you get in your car and turn the key, you expect the engine to start. Not sometimes, not every once in awhile, but every time. When I go to Google, and type in a search query, I expect to see relevant results, the best ones first, every time.

No one would keep a car that won't start, why keep a SE that doesn't work? Google hasn't "started" for me in 2 months! And they won't even communicate. All they have to say is, "Webmasters, we have a problem. We know what it is and we're trying to find the best way to fix it." No big deal.

10:37 pm on Feb 19, 2005 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 4, 2002
posts:1872
votes: 3


I can understand it from Google's point of view. They want to continue being the dominant SE, so they need to cover as much of the net as possible.

There are plenty of sites out there for Google to index--those in the sandbox for instance.

1:13 am on Feb 20, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 16, 2003
posts:992
votes: 0


For every real page in the index, there are multiple scrapers. For some pages there are also copyright-infringing copies around the place. Ban Google, and do you also put your site beyond the radar of most of these?

On the whole Google has been good to my content sites, so I'm not about to ban it. But I'm starting to wonder about the benefits of putting in large bot-free sections on my sites.

1:18 am on Feb 20, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Mar 22, 2004
posts:338
votes: 0


AndyA,

What you have experienced is the G$ blackmail of authority sites into paying for AW.

G$ finds the true authority sites and then pushes them down the SERPs, knowing that most of such sites will find it important to pay G$ their AW blackmail.

Beware the G$ cult worshippers here who will lay all kinds cult mind control tactics to suggest that that doesn't happen, but it does. It absolutely does. G$ is simply a liar, and nothing they say can be taken as honest these days. Others of us have experienced that blackmail firsthand too, so the facts are the facts. (And you can ignore the G$ cultists and their silly irrelevant accusations of supposed "conspiracy theory making" etc -- they are only trying to promote their cult using whatever lies they can come up with.)

The community topic I deal with has been purposely sabotaged by G$ ever since Oct 2003 -- and despite numerous efforts to correct whatever might be "wrong," such efforts are all a complete waste of time. G$ is simply playing with sites trying to "correct" things -- if anything was ever really wrong anyway. (There really is no correction possible to be accomplished when G$ decides they want to play with an authority site.) In this keyword, G$ still gives the most useless and dishonest "natural" SERPs for the keyword, and still prevents all the on-topic sites to appear.

Anyway, the choice for you is, do you have the freedom to not pay that blackmail or not? If you are fortunate enough to have the kind of audience who will indeed go to another search engine to find you, then perhaps you can avoid the G$ extortion into AW. But if you can't (or until you do), then G$ has purposely given you no choice, showing how truly that G$ can "do only evil."

5:42 am on Feb 20, 2005 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member

joined:May 13, 2003
posts:723
votes: 0


They do have a team in the New England area which helps people understand all those SEO things. The good thing is I have not heard that saying "DO NOT HIRE an SEO).

They rather say, have your SEO company look into this and this, and that, etc etc.

6:54 am on Feb 20, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 11, 2005
posts:143
votes: 0


OMG few weeks back I also banned googlebot from some of my unique pages.

After the sandbox of my site,
I stopped AdWord advertisement six months back. (click fraud).

I blocked googlebot last month. It was eating 300 MB/month and not any decent traffic.

This month I have contacted other ad companies for the replacement of AdSense. (unreliable pay)

I have grown to love msn(yuck) and yahoo search.

Just when did google become such money wh*re?
Sometimes I just wonder what these computer geeks must be thinking. Its just history repeating itself.

Only google product I really like is gmail. But I wouldn't be suprised if google screwed up that one in the future.

I wish more webmaster followed. This is the only way we can get back our Internet.

Power corrupts and absolute power corrupts absolutely.

7:28 am on Feb 20, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 24, 2001
posts:653
votes: 0


eliteweb,
Very interesting, thanks for posting. This rases to questions for me.
Why do you want to be out of the index?

IMHO bandwidth costs are not usually an issue, (to other posting, geez, if there are that many pages being spidered, and you cannot moneterise it to pay for the bandwidth, drop me a line, I might be able to help)
Privacy I can understand, as bhartzer said.

Why do Google want your site in the index?
Which seems to puzzle you :)
can we keep on track?

7:29 am on Feb 20, 2005 (gmt 0)

Preferred Member from US 

10+ Year Member

joined:Nov 27, 2002
posts:417
votes: 0


I keep hearing from others as well the same thing, "Google is Broken", or "google *(&^%"

Sad but true and these are non SEO's

Hollywood

8:30 am on Feb 20, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 12, 2003
posts:772
votes: 0


2005 FRESHMAN SEO THEORY 101 FINAL EXAM

Bonus Question (20 points)


G$ already ignores/disobeys proper Robots Exclusion Protocols. Googblebot is such an amateur that it can not read multiple lines of DISALLOW commandments in a robots.txt file.
When you bring that serious problem to G$'s attention, they then sabotagingly recommend to you that you should instead use an ALLOW command -- even as they tell you that other SE's might not recognize the command!

ALLOW is not part of the Robots Exclusion Protocoal -- and G$ knows it!

question:

what is the most likely cause of the observed behaviour?

answer:

[ ] a) a shortage of phd's out there
[ ] b) less interesting than personal free time projects
[ ] c) less important than gmail, orkut, what have you
[ ] d) all of the above

wildcard answer:

[ ] *) defer for three more updates

+++

2:29 pm on Feb 20, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 11, 2002
posts:504
votes: 0


Why does Google want your site in the index?

Would others see your site as leading?

Could it be that others would use the Google toolbar to tell Google that they like your site?

Could it be that others are giving Google some kind of feedback telling that they miss your site in the SERPS if it is not there?

2:57 pm on Feb 20, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Jan 21, 2005
posts:298
votes: 2


I have a small site with niche content and I get loads of emails from my forum members who love the site.

The google bot is on my site right NOW! Indexing every page. It visits every now and then but I hardly get ANY hits from google. My SE hits top three stands at:

MSN - 49%
Yahoo - 21%
Google - 12% [including adwords - about $10 a day]
Others - 18%

I am a bit worried though cause my income from my website is dependent on adsense and G seems to be loosing ground everyday. Waiting for a better paying ad network... And yes... last cheque received from google was 2 month's ago... so I'm not particularly happy.

4:27 pm on Feb 20, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 20, 2002
posts:813
votes: 1


roycerus,

If you're spending $10/day in adwords and your last adsense check was 2 months ago, then it would appear to be obvious that your income is less than $100 per month (min payout) yet your expenses would be 30x10=$300 per month.. so it's time to stop using adwords!

4:33 pm on Feb 20, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 9, 2004
posts:139
votes: 0


I'm noticing more then usual AV visitors on my logs recently for keywords that always ranked well on AltaVista...most keywords are in the same position they've been at for some months now but all of a sudden there are more hits from this SE. Anyone else noticed that?

It is also the first month ever that yahoo tipped the scale referral wise (against google)...

Is Kruoogle losing its charm?
Are more average joe's resort to other SE's to find what they are really looking for?

I think they do. It's just too tedious to shift through the piles of junk Kruoogle is serving lately.

Who's in for a "Google Free Site" with its own little logo, webring and a page explaining why more and more webmasters are blocking the google bot….this should/will hit prime time news in no time.

Maybe we should start a yahoo group on this topic. Lets teach them PHD's to stop messing with the web's heart & soul....us webmasters.

4:39 pm on Feb 20, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 9, 2003
posts:735
votes: 0


I remember a time when CruelGle was loved in these very forums. Times have changed, it seems :(
4:51 pm on Feb 20, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Jan 21, 2005
posts:298
votes: 2


Hi Chico_loco,
No my adsense earnings are actually much more than that so that's not an issue but yes I agree that I may not be getting enough revenue from adsense (from the adwords hits) but I don't want to loose the share of visitors from Google.

My point was that I get visitors from Google ONLY from adwords which is not good at all... I expect better returns from google search engine. I have no complaints from adsense [except for the late payment - the cheque probably got lost (AGAIN) this is the fourth time in my ten months with the program.

Some how I am getting the feeling that Google is changing into an ebay... a Bid war with most promoting adsense and some promoting products and some promoting adsense for the adwords for those products :-D

Regards,
R

[edited by: roycerus at 4:59 pm (utc) on Feb. 20, 2005]

4:59 pm on Feb 20, 2005 (gmt 0)

Senior Member

joined:Dec 29, 2003
posts:5428
votes: 0


if a company is being unjust to you (in yoru opinion), why would you support them by giving them your money? Especially when you have other choices.
5:01 pm on Feb 20, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Jan 21, 2005
posts:298
votes: 2


Unfortunately there are no replacements for Adsense... not the kind which is trustworthy and high paying.
5:03 pm on Feb 20, 2005 (gmt 0)

Senior Member

joined:Dec 29, 2003
posts:5428
votes: 0


"Unfortunately there are no replacements for Adsense... not the kind which is trustworthy and high paying."

That's what I've heard about that. Does Overture match up to Adwords?

5:29 pm on Feb 20, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 9, 2004
posts:139
votes: 0


Rest assured, Adsense is not here to stay (one read between the TOC lines tells it all). Krugle are working hard on tripling and quadrupling the amount of impression across their own pages. They will not need publishers page impressions soon (except maybe for the really big ones). And as for content, they are creating their own by publishing thousands of books online (shortly).

So they’ll have:
1) Gmail
2) Bad SERPs
3) Autolinking in toolbar
4) Groups
5) Images
6) Maps
7) Thousands of books
8) Blogger
9) Googlepedia

And the list goes on....

They'll be crazy to continue and pay Adsense publishers at the same rate they are paying now once all their services are in place. Why would they need us? Their shareholders will just vote us out (un-needed liability).

We Publishers currently doing the PR work for google and pave the way, once they feel it is done and their entire master plan is in place you will see the end of Adsense.

Enjoy it while it lasts and don't quit your day job.

Whatever happened to Adsense earnings starting 1st of Feb is a good example of how kruogle can cut publishers earning by half, overnight and with no prior warning.

Kruogle at its best.

This 85 message thread spans 3 pages: 85
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members