homepage Welcome to WebmasterWorld Guest from 54.227.77.237
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Local / Foo
Forum Library, Charter, Moderators: incrediBILL & lawman

Foo Forum

This 223 message thread spans 8 pages: < < 223 ( 1 2 [3] 4 5 6 7 8 > >     
lets try this for a month or three...
last recourse against rogue bots
Brett_Tabke




msg:329347
 1:21 am on Nov 19, 2005 (gmt 0)

[webmasterworld.com...]

required login the real story here...
MSN and yahoo bots were blocked in October. This does everyone else.

 

incrediBILL




msg:329407
 4:08 pm on Nov 23, 2005 (gmt 0)

Well Brett, you certainly have made a fine mess of things - can't search the site, webmasterworld missing in Google - lovely PR 0. The upside is that people booted out of AdSense can't find us for their first post anymore so I'm torn about this change of events as the positives are pretty good so far.

DaveN




msg:329408
 4:09 pm on Nov 23, 2005 (gmt 0)

vince vince vince ... if that's the case remaining open to friend-recommended .. the community you have today which you dislike will just get bigger and bigger.

< Mod hat OFF

It seems that the message that WebmasterWorld is sending out is we Don't need search! we don't need Google or Yahoo, Well .. I Do need search , I do need Google and Yahoo.
if you don't need Google or yahoo then fine, you will be over the moon... but I wouldn't expect someone like GoogleGuy to stick around when you have just banned his company form you site, And would you go to a search Conference run by a company that does not embrace SEARCH!

mod hat ON >

DaveN

oddsod




msg:329409
 4:12 pm on Nov 23, 2005 (gmt 0)

User-agent: *
Disallow: /

is disallowing listing in SERPs. By Brett's own admission it doesn't stop Google crawling the site. So if the recognised spiders are still crawling the site and the bad bots don't follow robots.txt anyway ... what exactly does the "nuclear option" (thanks DS) achieve apart from reducing the influx of the n00bs by removing all SE traffic?

DaveN




msg:329410
 4:14 pm on Nov 23, 2005 (gmt 0)

the upside is that people booted out of AdSense can't find us for their first post anymore :

yer and thats what mods are here for .. dealing with newbies, i guess thats why we have google reps here too..

you BILL you think thats an upside.. cool

DaveN

DaveN




msg:329411
 4:18 pm on Nov 23, 2005 (gmt 0)

for the record Oct 5, 2001 i was a noob here ..

DaveN

incrediBILL




msg:329412
 4:28 pm on Nov 23, 2005 (gmt 0)

Dave, I didn't say noobs not finding us was a good thing, just the noobs that don't understand why they got banned from AdSense because their friends clicked their ads ;)

Brett_Tabke




msg:329413
 4:34 pm on Nov 23, 2005 (gmt 0)

> By Brett's own admission it doesn't stop Google crawling the site.

but that isn't the core action here - required cookies is... which stops most bots dead...

New site search engine is in alpha and looking good. We'll fire it up some time for some live beta testing over the next few days in some off hours. If it holds up - we'll give it a bit more and more air time. Not in any real big hurry for it.

oddsod




msg:329414
 4:49 pm on Nov 23, 2005 (gmt 0)

Another plus side, Brett, is that you're giving your members a chance to rank for terms where WW always took #1 :)

Can't wait for your search. I bet it'll be an improvement on using Google to find stuff.

Brett_Tabke




msg:329415
 4:53 pm on Nov 23, 2005 (gmt 0)

So few people use site search oddsod, that it really isn't worth any extra ordinary effort. Less than 1 in 1k users use site search - not much roi in having it. Of those - it is mostly people looking for their own name or company name.

oddsod




msg:329416
 5:09 pm on Nov 23, 2005 (gmt 0)

>> Less than 1 in 1K use site search
Is it because they were using Google instead?

You must have an idea what percentage of people finding you in Google were already members.

Brett_Tabke




msg:329417
 5:18 pm on Nov 23, 2005 (gmt 0)

i mean from google... we had a lot from yahoo too because yahoo consistently had more pages indexed faster - however, slurp was so aggressive that it was too much load.

You take g, hoo, aj, msn bot together, and you are looking at a quarter million pvs a day most days. yeow...

bird




msg:329418
 5:19 pm on Nov 23, 2005 (gmt 0)

Less than 1 in 1k users use site search - not much roi in having it.

That's because it doesn't actually find anything in the current implementaiton. Make it useable and it will get much more exposure. But I'm sure you already knew that...

Brett_Tabke




msg:329419
 5:21 pm on Nov 23, 2005 (gmt 0)

THe part that would surprise people, is where referrals actually went...

The highest referring forums:
- javascript.
- html browsers.
- css.
- and the .net forum was also huge.

Those 4 accounted for 60-75% of all referrals.

incrediBILL




msg:329420
 5:21 pm on Nov 23, 2005 (gmt 0)

So few people use site search

That is sooo wrong, we all use site search, just not YOUR site search as I couldn't find anything with it. Now that you've whacked Google indexing WW there's no way to find a damn thing so it's like WW has become a large oracle of information that's next to impossible to use, much like the National Archives if you've ever been there.

WW world is now like a slot machine, a one-way repository with no access to it's contents.

bird




msg:329421
 5:23 pm on Nov 23, 2005 (gmt 0)

i mean from google...

Your logs won't tell you when I used a site: search on Google, because I'm dragging the links into a second window (or tab in FF).

Brett_Tabke




msg:329422
 5:25 pm on Nov 23, 2005 (gmt 0)

I wonder if somepeople are just wanked that their profile will no longer pass pr? So it comes back to their inability to game google through webmasterworld. interesting... this must be what G feels like during an update. lol

reseller




msg:329423
 5:28 pm on Nov 23, 2005 (gmt 0)

Brett_Tabke

>>So few people use site search oddsod, that it really isn't worth any extra ordinary effort. Less than 1 in 1k users use site search - not much roi in having it. Of those - it is mostly people looking for their own name or company name. <<

As a "GoogleGuyologist" I use the site search very often to refer other kind fellow members to what GG said in connection to specific questions or updates.

Call me a retired GoogleGuyologist from now on (:(

DaveN




msg:329424
 5:28 pm on Nov 23, 2005 (gmt 0)

Those 4 accounted for 60-75% of all referrals.

really ... that does suprise me hitwise traffic form the Uk looks a tad different to that..

DaveN

[edited by: DaveN at 5:31 pm (utc) on Nov. 23, 2005]

Robin_reala




msg:329425
 5:30 pm on Nov 23, 2005 (gmt 0)

No, I don't really care about that - you could stick rel="nofollow" on everything for all I care. But I do use google to search WebmasterWorld on a daily (if not hourly) basis.

Of course, now's the time I should shut up and pay for membership ;)

Brett_Tabke




msg:329426
 5:30 pm on Nov 23, 2005 (gmt 0)

The js forum accounts for almost 47% of all referrals to the site for the last year.

incrediBILL




msg:329427
 5:30 pm on Nov 23, 2005 (gmt 0)

profile will no longer pass pr

Considering I never put a URL in the profile, I'm purely annoyed from the fact I can't find anything. That's ok, will just have to download the whole site for offline indexing. ;)

Stefan




msg:329428
 5:32 pm on Nov 23, 2005 (gmt 0)

I assume that this technique won't be included in the "26 steps" ;-)

Brett_Tabke




msg:329429
 5:32 pm on Nov 23, 2005 (gmt 0)

I don't know how to explain it any better - but ya'll need to get a clue at how much of a problem rogue bots are... you are not getting it or appreciating it. This is a serious and costly problem that has threatened the very existence of the site.

whoisgregg




msg:329430
 5:36 pm on Nov 23, 2005 (gmt 0)

Gigablast is the slowest to respond, still returning 80,000 results for [css] [gigablast.com] where alltheweb and yahoo return about 9,000.

Added: Jeez everybody, it's Brett's decision. This site is a community, not just a database of knowledge. The community will continue without any kind of site search.

Powdork




msg:329431
 5:45 pm on Nov 23, 2005 (gmt 0)

Yeah, but what about this [powdork.com]

oddsod




msg:329432
 5:45 pm on Nov 23, 2005 (gmt 0)

I know some members are upset and I'm already feeling the pinch of the Google search function going (I did a whole load of searches and saved them!) but, as they say, you've got to walk a mile in the man's boots I guess.

DaveN




msg:329433
 5:49 pm on Nov 23, 2005 (gmt 0)

.. I did a whole load of searches and saved them ..

if i was you i would delete those .. WebmasterWorld is copyrighted.. and i would expect a letter from legal .. and the bad bots ... i guess you are one of them

DaveN

Brett_Tabke




msg:329434
 5:51 pm on Nov 23, 2005 (gmt 0)

> I assume that this technique won't be included in the "26 steps" ;-)

Actually, don't you wish your site could kick it's search engine addiction?

Admitting you have a problem, is the first step.

aka: this step? is the entire purpose of the first 26 steps. Please consider it...

Happy Thanksgiving everyone... they just called for boarding... i'll be in the air till late late...

jimbeetle




msg:329435
 5:52 pm on Nov 23, 2005 (gmt 0)

looking at a quarter million pvs a day most days. yeow...

Am I doing the math right here? There's just too many zeros for me to wrap my head around.

Assuming an average page size of about 40K (using the pages in this thread as a sample), does that actually come out to 10 gigs per day just for bots?

Whew!

balam




msg:329436
 5:54 pm on Nov 23, 2005 (gmt 0)

> This is a serious and costly problem that has threatened the very existence of the site.

B-b-but... that's your problem. To make a unilateral decision that has a most pronounced effect on us members who pay nothing for WebmasterWorld (ok, paying members too) is a gross abuse of power. After all, Brett, what do you think your website is? Your own personal fiefdom?

;)

kevinpate




msg:329437
 5:58 pm on Nov 23, 2005 (gmt 0)

Safe landings Brett, for you and for the site. :)

This 223 message thread spans 8 pages: < < 223 ( 1 2 [3] 4 5 6 7 8 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Local / Foo
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved