lets try this for a month or three...

last recourse against rogue bots

[Brett_Tabke]

[webmasterworld.com...]

required login the real story here...
MSN and yahoo bots were blocked in October. This does everyone else.

[incrediBILL]

Well Brett, you certainly have made a fine mess of things - can't search the site, webmasterworld missing in Google - lovely PR 0. The upside is that people booted out of AdSense can't find us for their first post anymore so I'm torn about this change of events as the positives are pretty good so far.

[DaveN]

vince vince vince ... if that's the case remaining open to friend-recommended .. the community you have today which you dislike will just get bigger and bigger.

< Mod hat OFF

It seems that the message that WebmasterWorld is sending out is we Don't need search! we don't need Google or Yahoo, Well .. I Do need search , I do need Google and Yahoo.
if you don't need Google or yahoo then fine, you will be over the moon... but I wouldn't expect someone like GoogleGuy to stick around when you have just banned his company form you site, And would you go to a search Conference run by a company that does not embrace SEARCH!

mod hat ON >

DaveN

[oddsod]

User-agent: *
Disallow: /

is disallowing listing in SERPs. By Brett's own admission it doesn't stop Google crawling the site. So if the recognised spiders are still crawling the site and the bad bots don't follow robots.txt anyway ... what exactly does the "nuclear option" (thanks DS) achieve apart from reducing the influx of the n00bs by removing all SE traffic?

[DaveN]

the upside is that people booted out of AdSense can't find us for their first post anymore :

yer and thats what mods are here for .. dealing with newbies, i guess thats why we have google reps here too..

you BILL you think thats an upside.. cool

DaveN

[DaveN]

for the record Oct 5, 2001 i was a noob here ..

DaveN

[incrediBILL]

Dave, I didn't say noobs not finding us was a good thing, just the noobs that don't understand why they got banned from AdSense because their friends clicked their ads ;)

[Brett_Tabke]

> By Brett's own admission it doesn't stop Google crawling the site.

but that isn't the core action here - required cookies is... which stops most bots dead...

New site search engine is in alpha and looking good. We'll fire it up some time for some live beta testing over the next few days in some off hours. If it holds up - we'll give it a bit more and more air time. Not in any real big hurry for it.

[oddsod]

Another plus side, Brett, is that you're giving your members a chance to rank for terms where WW always took #1 :)

Can't wait for your search. I bet it'll be an improvement on using Google to find stuff.

[Brett_Tabke]

So few people use site search oddsod, that it really isn't worth any extra ordinary effort. Less than 1 in 1k users use site search - not much roi in having it. Of those - it is mostly people looking for their own name or company name.

[oddsod]

>> Less than 1 in 1K use site search
Is it because they were using Google instead?

You must have an idea what percentage of people finding you in Google were already members.

[Brett_Tabke]

i mean from google... we had a lot from yahoo too because yahoo consistently had more pages indexed faster - however, slurp was so aggressive that it was too much load.

You take g, hoo, aj, msn bot together, and you are looking at a quarter million pvs a day most days. yeow...

[bird]

Less than 1 in 1k users use site search - not much roi in having it.

That's because it doesn't actually find anything in the current implementaiton. Make it useable and it will get much more exposure. But I'm sure you already knew that...

[Brett_Tabke]

THe part that would surprise people, is where referrals actually went...

The highest referring forums:
- javascript.
- html browsers.
- css.
- and the .net forum was also huge.

Those 4 accounted for 60-75% of all referrals.

[incrediBILL]

So few people use site search

That is sooo wrong, we all use site search, just not YOUR site search as I couldn't find anything with it. Now that you've whacked Google indexing WW there's no way to find a damn thing so it's like WW has become a large oracle of information that's next to impossible to use, much like the National Archives if you've ever been there.

WW world is now like a slot machine, a one-way repository with no access to it's contents.

[bird]

i mean from google...

Your logs won't tell you when I used a site: search on Google, because I'm dragging the links into a second window (or tab in FF).

[Brett_Tabke]

I wonder if somepeople are just wanked that their profile will no longer pass pr? So it comes back to their inability to game google through webmasterworld. interesting... this must be what G feels like during an update. lol

[reseller]

Brett_Tabke

>>So few people use site search oddsod, that it really isn't worth any extra ordinary effort. Less than 1 in 1k users use site search - not much roi in having it. Of those - it is mostly people looking for their own name or company name. <<

As a "GoogleGuyologist" I use the site search very often to refer other kind fellow members to what GG said in connection to specific questions or updates.

Call me a retired GoogleGuyologist from now on (:(

[DaveN]

Those 4 accounted for 60-75% of all referrals.

really ... that does suprise me hitwise traffic form the Uk looks a tad different to that..

DaveN

[edited by: DaveN at 5:31 pm (utc) on Nov. 23, 2005]

[Robin_reala]

No, I don't really care about that - you could stick rel="nofollow" on everything for all I care. But I do use google to search WebmasterWorld on a daily (if not hourly) basis.

Of course, now's the time I should shut up and pay for membership ;)

[Brett_Tabke]

The js forum accounts for almost 47% of all referrals to the site for the last year.

[incrediBILL]

profile will no longer pass pr

Considering I never put a URL in the profile, I'm purely annoyed from the fact I can't find anything. That's ok, will just have to download the whole site for offline indexing. ;)

[Stefan]

I assume that this technique won't be included in the "26 steps" ;-)

[Brett_Tabke]

I don't know how to explain it any better - but ya'll need to get a clue at how much of a problem rogue bots are... you are not getting it or appreciating it. This is a serious and costly problem that has threatened the very existence of the site.

[whoisgregg]

Gigablast is the slowest to respond, still returning 80,000 results for [css] [gigablast.com] where alltheweb and yahoo return about 9,000.

Added: Jeez everybody, it's Brett's decision. This site is a community, not just a database of knowledge. The community will continue without any kind of site search.

[Powdork]

Yeah, but what about this [powdork.com]

[oddsod]

I know some members are upset and I'm already feeling the pinch of the Google search function going (I did a whole load of searches and saved them!) but, as they say, you've got to walk a mile in the man's boots I guess.

[DaveN]

.. I did a whole load of searches and saved them ..

if i was you i would delete those .. WebmasterWorld is copyrighted.. and i would expect a letter from legal .. and the bad bots ... i guess you are one of them

DaveN

[Brett_Tabke]

> I assume that this technique won't be included in the "26 steps" ;-)

Actually, don't you wish your site could kick it's search engine addiction?

Admitting you have a problem, is the first step.

aka: this step? is the entire purpose of the first 26 steps. Please consider it...

Happy Thanksgiving everyone... they just called for boarding... i'll be in the air till late late...

[jimbeetle]

looking at a quarter million pvs a day most days. yeow...

Am I doing the math right here? There's just too many zeros for me to wrap my head around.

Assuming an average page size of about 40K (using the pages in this thread as a sample), does that actually come out to 10 gigs per day just for bots?

Whew!

[balam]

> This is a serious and costly problem that has threatened the very existence of the site.

B-b-but... that's your problem. To make a unilateral decision that has a most pronounced effect on us members who pay nothing for WebmasterWorld (ok, paying members too) is a gross abuse of power. After all, Brett, what do you think your website is? Your own personal fiefdom?

;)