Welcome to WebmasterWorld Guest from 54.211.17.91

Forum Moderators: incrediBILL & lawman

Message Too Old, No Replies

lets try this for a month or three...

last recourse against rogue bots

   
1:21 am on Nov 19, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



[webmasterworld.com...]

required login the real story here...
MSN and yahoo bots were blocked in October. This does everyone else.

5:41 pm on Nov 24, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Pulling the plug on the SEs prior to launching an alternative site search is my only complaint. The number of repeat questions by competent members will likely increase, the number by the ask-before-looking-crowd is already maxed. Brett says internal site search will be available soon - an inconvenient wait but not critical. Should the wait become indefinite or the result be substandard we can say something then.

I don't know the backend code, server capability, etc. Brett does so I will take his word on what is necessary, possible, or not. I have appreciated his efforts (and those of everyone else involved) for several years and believe that on the past record he deserves, at the least, the benefit of any doubt.

Ban The Bots!
Nasty little creepy crawlies.

6:03 pm on Nov 24, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I'll add that Brett's said publically more than once that he dreamed fo the day he could ban all bots and WebmasterWorld would stand on its own. I guess I just didn't believe he'd actually do it. Ballsy move - we'll be watching closely to see how WebmasterWorld fares going forward. All the complaining and #*$! folks aside I know none of us wish ill upon WebmasterWorld.
6:26 pm on Nov 24, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



it was all your fault todd
6:28 pm on Nov 24, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>>he dreamed fo the day he could ban all bots and WebmasterWorld would stand on its own

yeah and then he fell over ;)

7:17 pm on Nov 24, 2005 (gmt 0)

10+ Year Member



Forgive me if this is a noob response.

Couldn't you let bots in for one day a week only, in the middle of the night, and ban them the rest of the time? This, combined with bandwidth-throttling, and a bad bot repellent script, would help.

Unlike most other websites, Bots will come back to WebmasterWorld time and again due to all the links pointing to it. So let them Monday in the early AM only, to get the latest pages.

And bar them from all save the latest pages using mod_rewrite.

Greedy pigs behave one way, polite visitors another. Must be some way 'round this.

Interesting problem. Wish I had it :)

7:30 pm on Nov 24, 2005 (gmt 0)

10+ Year Member



Removing google search ability before having something else in place is the worst. And from earlier casual posts, that is not even close to being done since the direction isn't set. It should be already in place for supporters at a minimum. Now no one can complain about all the new posts we'll have because people can't do any research on their own. We'll need to have the same old posts and questions over and over again. It makes it hard to pay to support a site that doesn't support it's members but says it's all about the members. I hope it all works out for the best!

Maybe this is to get us all so frustrated that we'll welcome some kind of advertising to sponsor the search function instead of complaining.

10:19 pm on Nov 24, 2005 (gmt 0)

10+ Year Member



Just for fun I'm going to share an old crackpot theory I had about WW's search.

The thought was that the poor functionality of WW's site search was intentional and that encouraging users to use Google's 'site:' search had the side effect of increasing WW's rankings.

I theorized that Brett and co. knew some superstar SEO secret that the more 'site:' specific searches G sees for a particular domain, the higher it might rank it.

To me (I'm not an SEO pro) it would be logical to conclude that seeing tens of thousands of 'site:' queries per day for a particular domain would be an excellent indicator of a site's popularity/authority.

Now I wonder if all those searches had perhaps a negative effect: increasing the frequency and depth of indexing by the bots. It would seem to me that Google's (and Yahoo's?) algo might do this based on the same logic "hey, this site is really popular, we better index it like crazy to maintain accurate and thorough results".

Obviously, this wasn't WW's intention but I'd be interested to hear what people think. Could the number of 'site:' searches seen by the SE's possibly affect ranking or indexing frequency/depth?

10:28 pm on Nov 24, 2005 (gmt 0)

10+ Year Member



brett
can you use a program like Fluid Dynamics Search Engine?
i use it on my site, works great for us
10:38 pm on Nov 24, 2005 (gmt 0)

10+ Year Member



IMHO there is no reason that ww cannot use YPN adsense selectively on some forums or on a % of pages say 25 % to pay for the immense bendwidth charges.

Turning away traffic and Google I believe would lead to ultimate downfall of WW

How do you think will new webmasters find you...? I found ww by search and feel lucky now

and there is a crowd joining the webmaster band everyday given what adsense / ypn does to your pocket

Please do not take anything personally, this is a well meant advise.

10:47 pm on Nov 24, 2005 (gmt 0)

10+ Year Member



Hey Brett,
Ever thought of burning the whole site to DVD and selling it? Then I could copy to my hard drive and search locally for all the old threads. No server load to you and you could make some bucks...

Just a thought, maybe WW members could then get an update DVD once a quarter as an incentive for people to suscribe.

10:50 pm on Nov 24, 2005 (gmt 0)

10+ Year Member



In support of those who think its arrogant in the extreme to remove the only reasonable way to search the content provided by the good citizens of this community.

'doh I didnt think that would happen' oh yea homer?

If webmaster world of all places cant find solutions to the load/bandwidth issues, then may your god help the internet.

I conclude its been done purely and simply to stop paying supporters searching for the unsubscribe instructions.

but otoh maybe I too missed the point

l.

11:22 pm on Nov 24, 2005 (gmt 0)

10+ Year Member



I remember a couple of years back when the ODP was basically brought to its knees by over-agressive scrapers and crawlers. Not sure how it got fixed, but it did.

We're not seeing the picture that Brett sees.. I think we should trust his judgement on this one. Sure, the missing site search is a PITA but I'm sure we'll all cope. Although personally I would monetise the site more and then invest in the biggest servers I could lay my hands on.

You know though, I've always found that as a site GROWS then then the average VALUE of the visitor decreases (either in terms of revenue or other value.) Look at the threads about the last Google update if you want to see what the signal/noise problem is like. So perhaps this (temporary) change will help to refocus things a little.

As a site note, keep an eye on the Alexa stats. For a busy site like this they'll be fairly accurate. Sure, there's a dip today but then I understand it's this "Thanksgiving" holiday the Americans like so much.

And a final thought.. this is perhaps an experiment that none of us would want to try ourselves - it'll be interesting to see how it pans out.

12:08 am on Nov 25, 2005 (gmt 0)

10+ Year Member



Being without a search facility for webmasterworld has made me realise just how much I use it to find answers to questions. So first and foremost thanks for creating such a great resource!

Bots are a pain - as are spam email. I wish someone had the answer on how to effectively deal with them.

I look forward to the new search but think that we should give Brett a break - it sounds like he's been putting in the hours trying to find a solution to misbehaving bots.

I also think that WebmasterWorld.com could do some advertising, whether its affiliate links or adwords. I'm not overly found of advertising - but lets face it most other sites we visit have advertising.

1:34 am on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



One of the silver linings here is that for members who post valuable info, then reproduce it on their sites, its no longer dupe content.

Searching for some stuff on WW is now more of a redirect thru blogs and sites that reference many of the topics.

>outlive

One thing to remember is that everything has a life cycle, and when we look at that in an online perspective, we see that it isnt always as long as we think. With the logarithmic increase in blogs, particularly by those who are contributors to some of the more valuable content, a google search will take us to other resources without the noise.

I hope you know what you are doing Brett, because without a site search soon, the value of WW as a community will be measured against the value of WW as a resource. I personally care more about the resources than the community, but that's just me. I send a lot of people in my niche here for answers to questions. I frequently use Google to quickly find the url, then I post that for them to go to. Dont underestimate the resource/reference side of WW.

2:22 am on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The ODP had all functions (public, editor, forum, etc) all on one box, and bots killed it.

It was fixed by then having 3 load-balanced public servers, 1 editor server, 1 forum server, etc.

6:43 am on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Seeing as this is the Foo forum, Brett's timing for killing these spiders is interesting. November is the eighth canonical month (Ecclesiastical year; astronomical also, as it happens, starting 21 March) and Brett announced his decision to kill them on this site just as that month finished.

I've been steadily adding new spiders each month to my robots.pm file since getting AWStats in May, and had already noticed that there were far more new spiders this last 4 weeks than at any other period since I started. Synchronicity of 8, huh?.

8:06 am on Nov 25, 2005 (gmt 0)

10+ Year Member



I don't understand the first post:

required login the real story here...

What story?

I read through the posts, and what I get out of this, the site owner is blocking all bots because they consume too much bandwidth? But why block even Google? I am confused, where is the story?

I am curious to know the story because I run a very high bandwidth consuming forum (no monetization yet), and I would like to know the motivations for this act.

9:00 am on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Come on Brett you have kept us all waiting now, tell us the real reason why you have banned all bots..?
9:27 am on Nov 25, 2005 (gmt 0)

10+ Year Member



If I were Brett I would want to monetize this site, we all want to make our own sites profitable so why not Brett.

Remove content from the web, make the search available to members only. There could be a landing page to direct searchers for WebmasterWorld to the site (like the pubcon page)

This is the greatest resource for those involved in websites, and as such, very valuable both to us and Brett.

I honestly feel it is time that people start to accept that everything on the 'net is not free.

I do not suggest the above is what BT has in mind, nor is it in any way critical.

11:21 am on Nov 25, 2005 (gmt 0)

10+ Year Member



I'd be cool with that... The reason I pay a subscription to WebmasterWorld is becuase I see it a useful resouce that I want to be able to access. Giving Brett some $$$ will keep this site going... (I also think the threads I've read in the Supporters forum are some of the best info & advice I've read on the web...)

Up until now I've taken a step back because I know that whatever info I need is in WebmasterWorld and I could use Google to find it. But without a search function I will have to spend more time reading the forums. Time I really don't have.

12:14 pm on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member play_bach is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Regarding monetizing, here's a quote from Jenstar (don't know which post I get this from awhile back because I can't search for it anymore...)

"I have seen many sites try to switch to a
subscription model in order to make the site
either pay for itself or generate revenue, and I
have recently seen several die a painful death
while trying to implement it. Why? Because unless
it is information you cannot find anywhere else,
people can find another site with the same or
similar information for free. If it comes up
unexpectedly, your current visitor base can
become upset. Or worse, one of the people who
refused to pay the subscription fee goes and
starts a competitive non-subscription site. Some
sites can pull it off, but many can't."

--Jenstar (moderator, WebmasterWorld)

1:51 pm on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member play_bach is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Getting back to "rogue bots" as Brett's stated reason for having to shut down search, I keep coming back to the question:

"If rogue robots are such a problem, then how do the big sites deal with it?"

Somehow they all must, or they'd all be out of business, right? So how come WebmasterWorld can't?

1:55 pm on Nov 25, 2005 (gmt 0)

WebmasterWorld Administrator lawman is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Hello Play_Back. How did you find Webmaster World?
2:01 pm on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member play_bach is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Hi lawman,

> Hello Play_Back. How did you find Webmaster World?

Probably through a Google search. And you?

2:05 pm on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



if found it via google btw ... maybe Brett does not want any more problem makes like me finding the place :)

DaveN

[edited by: DaveN at 2:19 pm (utc) on Nov. 25, 2005]

2:14 pm on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member lammert is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Probably through a Google search. And you?

One of the things that scares me with the current situation. I also found WebmasterWorld with a Google search. After the site popped up a number of times for webmaster related queries I decided to join and eventually become a paying member.

By not being visible anymore in the major search engines, the number of new users may decrease. For some forums--like the AdSense forum--this might lead to increased quality in the short term, but I am not sure how the community will evolve if WW remains unfindable from the search engines for a longer period of time. A good thread is often not started by a senior who knows it all, but by a newbie who happens to ask the good questions. A community is an organic entity and by cutting one of its sources for new blood, it can change into something unwanted which is difficult to reverse.

2:24 pm on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



chopin2256,

Brett said it was a server load issue not one of bandwidth.

I can relate to that.

I just hope that his new search system doesn't cause the same issue for him.

2:25 pm on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



For some topics, maybe WebmasterWorld doesn't need the same 20 questions asked again and again every single day, when each question has already been answered in several hundred previous repetitive threads?
2:32 pm on Nov 25, 2005 (gmt 0)

10+ Year Member




> For some topics, maybe WebmasterWorld doesn't need the same
> 20 questions asked again and again every single day, when
> each question has already been answered in several hundred
> previous repetitive threads?

Without a search function this is just going to spiral upwards. One of things that did irk me was the elitist way there wasn't a set search function. Newbies to search wouldn't be able to work out how to use Google to search WebmasterWorld and so ended up asking repetive questions.

2:50 pm on Nov 25, 2005 (gmt 0)

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Hey Brett,
Ever thought of burning the whole site to DVD and selling it

I'd be interested in this. Would be hard to stop it becoming a plug-and-play clone or scraper site. Seems it would need encryption of some type and raster display on decryption, allowing only [ code ] blocks to be selected for copy & paste. Major undertaking.
This 223 message thread spans 8 pages: 223