Welcome to WebmasterWorld Guest from 54.144.243.34

Forum Moderators: incrediBILL & lawman

lets try this for a month or three...

last recourse against rogue bots

   
1:21 am on Nov 19, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



[webmasterworld.com...]

required login the real story here...
MSN and yahoo bots were blocked in October. This does everyone else.

4:08 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Well Brett, you certainly have made a fine mess of things - can't search the site, webmasterworld missing in Google - lovely PR 0. The upside is that people booted out of AdSense can't find us for their first post anymore so I'm torn about this change of events as the positives are pretty good so far.
4:09 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



vince vince vince ... if that's the case remaining open to friend-recommended .. the community you have today which you dislike will just get bigger and bigger.

< Mod hat OFF

It seems that the message that WebmasterWorld is sending out is we Don't need search! we don't need Google or Yahoo, Well .. I Do need search , I do need Google and Yahoo.
if you don't need Google or yahoo then fine, you will be over the moon... but I wouldn't expect someone like GoogleGuy to stick around when you have just banned his company form you site, And would you go to a search Conference run by a company that does not embrace SEARCH!

mod hat ON >

DaveN

4:12 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



User-agent: *
Disallow: /

is disallowing listing in SERPs. By Brett's own admission it doesn't stop Google crawling the site. So if the recognised spiders are still crawling the site and the bad bots don't follow robots.txt anyway ... what exactly does the "nuclear option" (thanks DS) achieve apart from reducing the influx of the n00bs by removing all SE traffic?

4:14 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



the upside is that people booted out of AdSense can't find us for their first post anymore :

yer and thats what mods are here for .. dealing with newbies, i guess thats why we have google reps here too..

you BILL you think thats an upside.. cool

DaveN

4:18 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



for the record Oct 5, 2001 i was a noob here ..

DaveN

4:28 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Dave, I didn't say noobs not finding us was a good thing, just the noobs that don't understand why they got banned from AdSense because their friends clicked their ads ;)
4:34 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



> By Brett's own admission it doesn't stop Google crawling the site.

but that isn't the core action here - required cookies is... which stops most bots dead...

New site search engine is in alpha and looking good. We'll fire it up some time for some live beta testing over the next few days in some off hours. If it holds up - we'll give it a bit more and more air time. Not in any real big hurry for it.

4:49 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Another plus side, Brett, is that you're giving your members a chance to rank for terms where WW always took #1 :)

Can't wait for your search. I bet it'll be an improvement on using Google to find stuff.

4:53 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



So few people use site search oddsod, that it really isn't worth any extra ordinary effort. Less than 1 in 1k users use site search - not much roi in having it. Of those - it is mostly people looking for their own name or company name.
5:09 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>> Less than 1 in 1K use site search
Is it because they were using Google instead?

You must have an idea what percentage of people finding you in Google were already members.

5:18 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



i mean from google... we had a lot from yahoo too because yahoo consistently had more pages indexed faster - however, slurp was so aggressive that it was too much load.

You take g, hoo, aj, msn bot together, and you are looking at a quarter million pvs a day most days. yeow...

5:19 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Less than 1 in 1k users use site search - not much roi in having it.

That's because it doesn't actually find anything in the current implementaiton. Make it useable and it will get much more exposure. But I'm sure you already knew that...

5:21 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



THe part that would surprise people, is where referrals actually went...

The highest referring forums:
- javascript.
- html browsers.
- css.
- and the .net forum was also huge.

Those 4 accounted for 60-75% of all referrals.

5:21 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



So few people use site search

That is sooo wrong, we all use site search, just not YOUR site search as I couldn't find anything with it. Now that you've whacked Google indexing WW there's no way to find a damn thing so it's like WW has become a large oracle of information that's next to impossible to use, much like the National Archives if you've ever been there.

WW world is now like a slot machine, a one-way repository with no access to it's contents.

5:23 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



i mean from google...

Your logs won't tell you when I used a site: search on Google, because I'm dragging the links into a second window (or tab in FF).

5:25 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I wonder if somepeople are just wanked that their profile will no longer pass pr? So it comes back to their inability to game google through webmasterworld. interesting... this must be what G feels like during an update. lol
5:28 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Brett_Tabke

>>So few people use site search oddsod, that it really isn't worth any extra ordinary effort. Less than 1 in 1k users use site search - not much roi in having it. Of those - it is mostly people looking for their own name or company name. <<

As a "GoogleGuyologist" I use the site search very often to refer other kind fellow members to what GG said in connection to specific questions or updates.

Call me a retired GoogleGuyologist from now on (:(

5:28 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Those 4 accounted for 60-75% of all referrals.

really ... that does suprise me hitwise traffic form the Uk looks a tad different to that..

DaveN

[edited by: DaveN at 5:31 pm (utc) on Nov. 23, 2005]

5:30 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



No, I don't really care about that - you could stick rel="nofollow" on everything for all I care. But I do use google to search WebmasterWorld on a daily (if not hourly) basis.

Of course, now's the time I should shut up and pay for membership ;)

5:30 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The js forum accounts for almost 47% of all referrals to the site for the last year.
5:30 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



profile will no longer pass pr

Considering I never put a URL in the profile, I'm purely annoyed from the fact I can't find anything. That's ok, will just have to download the whole site for offline indexing. ;)

5:32 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I assume that this technique won't be included in the "26 steps" ;-)
5:32 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I don't know how to explain it any better - but ya'll need to get a clue at how much of a problem rogue bots are... you are not getting it or appreciating it. This is a serious and costly problem that has threatened the very existence of the site.
5:36 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member whoisgregg is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Gigablast is the slowest to respond, still returning 80,000 results for [css] [gigablast.com] where alltheweb and yahoo return about 9,000.

Added: Jeez everybody, it's Brett's decision. This site is a community, not just a database of knowledge. The community will continue without any kind of site search.

5:45 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member powdork is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Yeah, but what about this [powdork.com]
5:45 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I know some members are upset and I'm already feeling the pinch of the Google search function going (I did a whole load of searches and saved them!) but, as they say, you've got to walk a mile in the man's boots I guess.
5:49 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



.. I did a whole load of searches and saved them ..

if i was you i would delete those .. WebmasterWorld is copyrighted.. and i would expect a letter from legal .. and the bad bots ... i guess you are one of them

DaveN

5:51 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



> I assume that this technique won't be included in the "26 steps" ;-)

Actually, don't you wish your site could kick it's search engine addiction?

Admitting you have a problem, is the first step.

aka: this step? is the entire purpose of the first 26 steps. Please consider it...

Happy Thanksgiving everyone... they just called for boarding... i'll be in the air till late late...

5:52 pm on Nov 23, 2005 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member



looking at a quarter million pvs a day most days. yeow...

Am I doing the math right here? There's just too many zeros for me to wrap my head around.

Assuming an average page size of about 40K (using the pages in this thread as a sample), does that actually come out to 10 gigs per day just for bots?

Whew!

5:54 pm on Nov 23, 2005 (gmt 0)

10+ Year Member



> This is a serious and costly problem that has threatened the very existence of the site.

B-b-but... that's your problem. To make a unilateral decision that has a most pronounced effect on us members who pay nothing for WebmasterWorld (ok, paying members too) is a gross abuse of power. After all, Brett, what do you think your website is? Your own personal fiefdom?

;)

This 223 message thread spans 8 pages: 223
 

Featured Threads

Hot Threads This Week

Hot Threads This Month