Forum Moderators: open
required login the real story here...
MSN and yahoo bots were blocked in October. This does everyone else.
no one is looking forward to the inevitable compatability problems
Judging by the quality of support in forum83 that should really be an issue :)
So, if this site can work without SE's all the more power and respect to my fellow members, to Brett and to the mods throughout the years for that. Well done!
It's not the first site to achieve that status, but it's certainly one of the largest.
--
An internal site search is *badly needed* though, and a very good one at that. But that's apparently in the works, so I'll just have to be patient (like any good doctor will tell you to be).
Just imagine forgetting Search Engines all together we could bang out sites left right and center without the worry.
I must say I'm going to miss the Google search, specially in the Php section looking up previous posts and syntax tips.
Looking forward to help test the new search function.
I only wish I used my Add Bookmark button more, never mind.
Hands up all those who would love to close the door on search engines!
I remember finding WW in Google for the very first time many years ago. When I searched Google for php answers etc I got WW and Co and 99% of the time the answers.
WW is like a family on the web you can go about your business each day knowing that the support or community is right there when you need it, thats why WW doesn't require Google or Yahoo any longer!
Good Luck Brett on the new search function!
> It's not - system load is.
> Sooner or later, you are going to kick the nighbors out of the house and build a fence.
OK, well my question still stands. How does eBay, Amazon, craigslist, Yahoo! or any of the other big portals deal with this bot problem? Somehow they are all able to be spidered by Google (et al) and also provide site search - anybody know?
Thanks
[edited by: DaveN at 2:13 pm (utc) on Nov. 24, 2005]
Uhhhh. Load balancing? Dells are cheap.
How does eBay, Amazon, craigslist, Yahoo! or any of the other big portals deal with this bot problem? Somehow they are all able to be spidered by Google (et al) and also provide site search - anybody know?
Its called a multi-million dollar investment in load balancing, servers, bandwidth etc.
WW is a free forum with donations from regular supporters and I guess only 1 server that has to handle a huge load.
surley that will bring the server to it's knees
DaveN
Hands up all those who would love to close the door on search engines!
Exactly! If you're playing high risk games with SEs you want them around 'cause that's how you get your buzz. Everyone else would like to give up the addiction but won't admit it in public and cetainly won't go the whole hog and ban all bots.
if anything just Disallow forums /forum30/ (forum 1-XX) and you probably would have cut down robot load by 99% while still maintaining your homepage.
If your gone for 6 months, thats pretty bad... and self-inflicted is even worse. One thing about pubcon's is you see old friends you've made and you see fresh faces at each one...
But I thought Brett said the problem was "rogue bots," right? So how do the big portals deal with it? Does anybody know?
Thanks.
Load balancing is easy on the send - syncing writes is very difficult and requires software specific to your app to deal with. That software is started, but not complete here. I'd guess 50 files a minute are updated. Probably 8-10k files a day are changed. Some how, the software has to sync those files across all the server simultaneously.
> how are we going to handle a site search?
Good point - yet to be seen. I am working under the theory that we'll do as we did before and put it on sew [searchengineworld.com] - or another server altogether. Also, plan B is aspseek, which isn't too bad load wise via a sql server on the same box (although, I think the results are pretty poor).
But ya know - of all the people that would have issue, or voice support for the action - yours is completely baffling Dave. You talk often about being so "black hat", eschewing scrapper sites, loathing the engines, and ripping them of traffic, that you should be the first to support this!?
> So, if this site can work without SE's
> all the more power and respect to my fellow
> members, to Brett and to the mods throughout
> the years for that. Well done!
Claus - thanks!
> server load is
there is also the issue of ripped content... that is another independent story though that I don't think is prudent to discuss in public.
> system load is this a inerrant problem with the BestBBS
I believe it is the most system friendly forum on the web.
/. was last heard to be running about .75 our page views and uniques, but on 8 load balanced servers. Eg: bestbbs is 10-15 times more efficient than / code. Which is pretty good considering I wrote the software to handle 1,000 members and 10,000 page views a day (take times 100 to get into the ball park of where we were last week). Who ever freakin believed we would have 600k files in just threads? yeow...
[edited by: Brett_Tabke at 3:20 pm (utc) on Nov. 24, 2005]
I see a revolution in what you have done :-)
Lets assume that everything will go successfuly as you have thought and planned. And WebmasterWorld grow and nurish without being listed in search engines, and I do hope that that exactly whats gonna happen.
Have you ever thought about what that means for the SEO industry?
In fact you are showing the owners of big sites that they don't need SEO and search engines to survive!
Leaving those famous 26 steps to the webmasters of mini-sites, small and medium sites.
And I can already see 100s of SEO specialists starting shining their resumes :-)
Pretty early to be making that claim...
lol, Maybe I'm getting soft in my old age :)
DaveN
syncing writes is very difficult and requires software specific to your app to deal with.
OK, I could explain to you the concept of time invariant data models and how you could easily update write synchs seamlessly across multiple servers as each server would always have it's correct snapshot "in time" as they all update, but it only works if your data architecture is designed properly.
Been there, done that, I'm not cheap ;)
Of course I came to WebmasterWorld from the advice of a friend.
hey Brett .. look back to my old evil blackhat self ;)
DaveN
With regard to new members, I found this via word of mouth 3 years ago. With blogs, WW gets even more play, so that is probably a wash.
Its not like this place will fall apart in 60-90 days, but I hope you are testing this with a goal in mind. Not sure though, since you and DaveN seem to be on different pages and I would have expected your mods to be in this with you from the start, or at least willing to give it a chance. Making decisions of this magnitude without your mods is foolish and arrogant, especially when you want this to out live you. Without good mods it doesnt stand a chance, and without search, you will wear out your mods. Losing your mods from this community, but past and present, is not something you can afford.
I wonder if somepeople are just wanked that their profile will no longer pass pr? So it comes back to their inability to game google through webmasterworld. interesting... this must be what G feels like during an update. lol
This is the only thing that really irked me in this thread. We have good reasons to be wanked, so that was a cheap shot. Like many others, I havent put anything in a profile. Up until last week at PubCon, and outside of my supporters registration, my ID was completely seperate from my online nic.
Best of luck with this and Happy Thanksgiving.
The first I knew this was happening is when a member MSN IMed me ..lol
DaveN
I used Google to find Suzy's css tutorial. It was bookmarked in my laptop someone stole. This is more than a community to me. It is a resource. Its like someone just took away my library card.
Best of luck to all those who decide to hide their sites from now on, but I'm going to leave mine out there where everyone can find them. (And if that's an addiction, what the hec - I also like a few beers everyday, so this won't be the first one I'm dealing with.)
Brett_Tabke #150:
> if bandwidth is a problem
It's not - system load is.
slurp was so aggressive that it was too much load
Brett_Tabke #67:
New site search engine is in alpha ... Not in any real big hurry for it
Less than 1 in 1k users use site search
It is now a resource who's history cannot be accessed.
Later: Heheh, WebmasterWorld does too now, it seems.
