Forum Moderators: open
required login the real story here...
MSN and yahoo bots were blocked in October. This does everyone else.
I don't know the backend code, server capability, etc. Brett does so I will take his word on what is necessary, possible, or not. I have appreciated his efforts (and those of everyone else involved) for several years and believe that on the past record he deserves, at the least, the benefit of any doubt.
Ban The Bots!
Nasty little creepy crawlies.
yeah and then he fell over ;)
Couldn't you let bots in for one day a week only, in the middle of the night, and ban them the rest of the time? This, combined with bandwidth-throttling, and a bad bot repellent script, would help.
Unlike most other websites, Bots will come back to WebmasterWorld time and again due to all the links pointing to it. So let them Monday in the early AM only, to get the latest pages.
And bar them from all save the latest pages using mod_rewrite.
Greedy pigs behave one way, polite visitors another. Must be some way 'round this.
Interesting problem. Wish I had it :)
Maybe this is to get us all so frustrated that we'll welcome some kind of advertising to sponsor the search function instead of complaining.
The thought was that the poor functionality of WW's site search was intentional and that encouraging users to use Google's 'site:' search had the side effect of increasing WW's rankings.
I theorized that Brett and co. knew some superstar SEO secret that the more 'site:' specific searches G sees for a particular domain, the higher it might rank it.
To me (I'm not an SEO pro) it would be logical to conclude that seeing tens of thousands of 'site:' queries per day for a particular domain would be an excellent indicator of a site's popularity/authority.
Now I wonder if all those searches had perhaps a negative effect: increasing the frequency and depth of indexing by the bots. It would seem to me that Google's (and Yahoo's?) algo might do this based on the same logic "hey, this site is really popular, we better index it like crazy to maintain accurate and thorough results".
Obviously, this wasn't WW's intention but I'd be interested to hear what people think. Could the number of 'site:' searches seen by the SE's possibly affect ranking or indexing frequency/depth?
Turning away traffic and Google I believe would lead to ultimate downfall of WW
How do you think will new webmasters find you...? I found ww by search and feel lucky now
and there is a crowd joining the webmaster band everyday given what adsense / ypn does to your pocket
Please do not take anything personally, this is a well meant advise.
Just a thought, maybe WW members could then get an update DVD once a quarter as an incentive for people to suscribe.
'doh I didnt think that would happen' oh yea homer?
If webmaster world of all places cant find solutions to the load/bandwidth issues, then may your god help the internet.
I conclude its been done purely and simply to stop paying supporters searching for the unsubscribe instructions.
but otoh maybe I too missed the point
l.
We're not seeing the picture that Brett sees.. I think we should trust his judgement on this one. Sure, the missing site search is a PITA but I'm sure we'll all cope. Although personally I would monetise the site more and then invest in the biggest servers I could lay my hands on.
You know though, I've always found that as a site GROWS then then the average VALUE of the visitor decreases (either in terms of revenue or other value.) Look at the threads about the last Google update if you want to see what the signal/noise problem is like. So perhaps this (temporary) change will help to refocus things a little.
As a site note, keep an eye on the Alexa stats. For a busy site like this they'll be fairly accurate. Sure, there's a dip today but then I understand it's this "Thanksgiving" holiday the Americans like so much.
And a final thought.. this is perhaps an experiment that none of us would want to try ourselves - it'll be interesting to see how it pans out.
Bots are a pain - as are spam email. I wish someone had the answer on how to effectively deal with them.
I look forward to the new search but think that we should give Brett a break - it sounds like he's been putting in the hours trying to find a solution to misbehaving bots.
I also think that WebmasterWorld.com could do some advertising, whether its affiliate links or adwords. I'm not overly found of advertising - but lets face it most other sites we visit have advertising.
Searching for some stuff on WW is now more of a redirect thru blogs and sites that reference many of the topics.
>outlive
One thing to remember is that everything has a life cycle, and when we look at that in an online perspective, we see that it isnt always as long as we think. With the logarithmic increase in blogs, particularly by those who are contributors to some of the more valuable content, a google search will take us to other resources without the noise.
I hope you know what you are doing Brett, because without a site search soon, the value of WW as a community will be measured against the value of WW as a resource. I personally care more about the resources than the community, but that's just me. I send a lot of people in my niche here for answers to questions. I frequently use Google to quickly find the url, then I post that for them to go to. Dont underestimate the resource/reference side of WW.
It was fixed by then having 3 load-balanced public servers, 1 editor server, 1 forum server, etc.
I've been steadily adding new spiders each month to my robots.pm file since getting AWStats in May, and had already noticed that there were far more new spiders this last 4 weeks than at any other period since I started. Synchronicity of 8, huh?.
required login the real story here...
What story?
I read through the posts, and what I get out of this, the site owner is blocking all bots because they consume too much bandwidth? But why block even Google? I am confused, where is the story?
I am curious to know the story because I run a very high bandwidth consuming forum (no monetization yet), and I would like to know the motivations for this act.
Remove content from the web, make the search available to members only. There could be a landing page to direct searchers for WebmasterWorld to the site (like the pubcon page)
This is the greatest resource for those involved in websites, and as such, very valuable both to us and Brett.
I honestly feel it is time that people start to accept that everything on the 'net is not free.
I do not suggest the above is what BT has in mind, nor is it in any way critical.
Up until now I've taken a step back because I know that whatever info I need is in WebmasterWorld and I could use Google to find it. But without a search function I will have to spend more time reading the forums. Time I really don't have.
"I have seen many sites try to switch to a
subscription model in order to make the site
either pay for itself or generate revenue, and I
have recently seen several die a painful death
while trying to implement it. Why? Because unless
it is information you cannot find anywhere else,
people can find another site with the same or
similar information for free. If it comes up
unexpectedly, your current visitor base can
become upset. Or worse, one of the people who
refused to pay the subscription fee goes and
starts a competitive non-subscription site. Some
sites can pull it off, but many can't."
--Jenstar (moderator, WebmasterWorld)
"If rogue robots are such a problem, then how do the big sites deal with it?"
Somehow they all must, or they'd all be out of business, right? So how come WebmasterWorld can't?
> Hello Play_Back. How did you find Webmaster World?
Probably through a Google search. And you?
DaveN
[edited by: DaveN at 2:19 pm (utc) on Nov. 25, 2005]
Probably through a Google search. And you?
One of the things that scares me with the current situation. I also found WebmasterWorld with a Google search. After the site popped up a number of times for webmaster related queries I decided to join and eventually become a paying member.
By not being visible anymore in the major search engines, the number of new users may decrease. For some forums--like the AdSense forum--this might lead to increased quality in the short term, but I am not sure how the community will evolve if WW remains unfindable from the search engines for a longer period of time. A good thread is often not started by a senior who knows it all, but by a newbie who happens to ask the good questions. A community is an organic entity and by cutting one of its sources for new blood, it can change into something unwanted which is difficult to reverse.
Brett said it was a server load issue not one of bandwidth.
I can relate to that.
I just hope that his new search system doesn't cause the same issue for him.
Without a search function this is just going to spiral upwards. One of things that did irk me was the elitist way there wasn't a set search function. Newbies to search wouldn't be able to work out how to use Google to search WebmasterWorld and so ended up asking repetive questions.
Hey Brett,
Ever thought of burning the whole site to DVD and selling it
