homepage Welcome to WebmasterWorld Guest from 54.197.147.98
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Local / Foo
Forum Library, Charter, Moderators: incrediBILL & lawman

Foo Forum

This 223 message thread spans 8 pages: < < 223 ( 1 [2] 3 4 5 6 7 8 > >     
lets try this for a month or three...
last recourse against rogue bots
Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 9593 posted 1:21 am on Nov 19, 2005 (gmt 0)

[webmasterworld.com...]

required login the real story here...
MSN and yahoo bots were blocked in October. This does everyone else.

 

walkman



 
Msg#: 9593 posted 4:42 am on Nov 22, 2005 (gmt 0)

I wish I could afford to turn away free traffic. It must feel great :)

Josk

10+ Year Member



 
Msg#: 9593 posted 10:16 am on Nov 22, 2005 (gmt 0)

> Incidentally, any chance of getting a better site-search now
> that Google and AllTheWeb won't be indexing new content?

This is the killer app. If I can't search WebmasterWorld like I can now with Google I wouldn't come back so often. I've found that mostly any query can be answered now with a properly formed query.

Perhaps time to invest in one of those Google Boxes...?

ukgimp

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 11:54 am on Nov 22, 2005 (gmt 0)

[webmasterworld.com...]

so how do people find stuff now?

RockyB

5+ Year Member



 
Msg#: 9593 posted 1:24 pm on Nov 22, 2005 (gmt 0)

I must admit I find this a shame. I've personally had the experiance where searching for information on a specific, technical subject on Google I've come across a webmasterworld post which perfectly answers what I needed. It's a shame this won't be avaliable any more.

Having said that, I can see where you are coming from, and I hope it works out for you.

oddsod

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 3:43 pm on Nov 22, 2005 (gmt 0)

Brett, I've no doubt you know what you're doing.

I'll live with a short period of no site search + having to log in.

I wonder what the boys in the plex had to say about this. I can't believe it didn't even reach the water cooler. :)

balam

10+ Year Member



 
Msg#: 9593 posted 6:04 pm on Nov 22, 2005 (gmt 0)

The sky is falling! The sky is falling!

Sorry, wrong thread, I think.

Allergic

10+ Year Member



 
Msg#: 9593 posted 6:47 pm on Nov 22, 2005 (gmt 0)

If it's working why not but a crawl-delay like Slashdot?

User-agent: Googlebot
Crawl-delay: 100

User-agent: Slurp
Crawl-delay: 100
Disallow:

User-agent: Yahoo-NewsCrawler
Disallow:

User-Agent: msnbot
Crawl-delay: 100
Disallow:

User-agent: *
Crawl-delay: 100

balam

10+ Year Member



 
Msg#: 9593 posted 7:43 pm on Nov 22, 2005 (gmt 0)

"Crawl-delay", while nice, is not "standard" or universally recognized or supported.

In the case of Google, there is no mention of support for it on their bot page, Googlebot: Google's Web Crawler [google.com]. In fact, I believe Google would suggest the use of SiteMaps if you wish to throttle Googlebot.

From the above link:
3. Googlebot is crawling my site too fast. What can I do?

Please contact us with the URL of your site and a detailed description of the problem. Please also include a portion of the weblog that shows Google accesses so we can track down the problem quickly.

[Emphasis mine]

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 9593 posted 7:47 pm on Nov 22, 2005 (gmt 0)

> It must feel great

Honestly, yes it does. I have had 4 very nice nights of sleep.

AlexK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 8:01 pm on Nov 22, 2005 (gmt 0)

<over-earnest-effort-to-see-the-bright-side>

Better to have to deal with the side-effects of success than those of failure.

</over-earnest-effort-to-see-the-bright-side>

arbitrary

5+ Year Member



 
Msg#: 9593 posted 6:19 am on Nov 23, 2005 (gmt 0)

The rogue bots that are spidering the site, what are they doing with the pages?

I figured people would be smart enough not to repost WW pages on the web (maybe I am wrong) so I am wondering what they are doing with them.

reseller

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 9:13 am on Nov 23, 2005 (gmt 0)

Brett

Do we have any other possibility than google search boxes (at bottom of WebmasterWorld pages) to search for previous posts?

Thanks.

DaveN

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 9:14 am on Nov 23, 2005 (gmt 0)

[google.co.uk...] in the uk webmasterworld is gone in google

DaveN

ncw164x

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 9:38 am on Nov 23, 2005 (gmt 0)

>>Ya, the site is as fast as it has ever been
It will be even faster when you loose even more of your members

>>Honestly, yes it does. I have had 4 very nice nights of sleep
Brett why are you loosing sleep over a website?

You must know something that all of us have missed Brett, it's gone already in google UK the rest will follow shortly..

Josk

10+ Year Member



 
Msg#: 9593 posted 9:54 am on Nov 23, 2005 (gmt 0)

I'm not seeing any results in google.com either. (And this is from London) This kind-of leaves the knowledge in WebmasterWorld unsearchable...

netscan

10+ Year Member



 
Msg#: 9593 posted 11:42 am on Nov 23, 2005 (gmt 0)

Throwing out baby with the bathwater comes to mind...

stubear

5+ Year Member



 
Msg#: 9593 posted 11:46 am on Nov 23, 2005 (gmt 0)

Pagerank is now 0 too...

digicam

10+ Year Member



 
Msg#: 9593 posted 11:46 am on Nov 23, 2005 (gmt 0)

Will this work on my site where I have a problem with supplemental results - if I ban all robots for a week will google remove all my content.

I can then fix a few problems that have been bothering me.

If I then reallow robots.txt after a week will everything be OK

Any thoughts please.

limbo

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 12:05 pm on Nov 23, 2005 (gmt 0)

I can't find any of the threads I have been looking for.

http://www.google.co.uk/search?q=webmasterworld.com in the uk webmasterworld is gone in google

Same MSN. Still OK in Yahoo but showing less results than expected and I guess it's a matter of time before that is the same.

The vast collective of archived WebmasterWorld threads only exists if you can find them - via the search engines - now we can't find a single post unless we hunt through reams and reams of links...

:(

vincevincevince

WebmasterWorld Senior Member vincevincevince us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 9593 posted 12:35 pm on Nov 23, 2005 (gmt 0)

I'm wondering if Forum30 will become de-pre-moderated at some point now?

DaveN

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 12:49 pm on Nov 23, 2005 (gmt 0)

Brett Said : a solution is being tested and worked on. It will probably take atleast 60 days for the old pages to be purged from the engines.

I guessed at 5 days, I have had a client question me about a quote that GG said in a one of the Jagger Threads, it's kinda Cryptic so I need to read it a few times more.. NOW I can't even find the dam thing

MSN has 1 pages
Google has 0 pages
Yahoo is dropping them fast than I can search!

so :

a solution is being tested and worked on : which means? we have a framework in place.. how long before we will see some results..

because if we allow Google back in today it still going to be 180 days

from google :

URL removal system will cause a temporary, 180 day removal of your site from the Google index, regardless of whether you remove the robots.txt file after processing your request.

AlexK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 12:58 pm on Nov 23, 2005 (gmt 0)

Brett, you cannot both ban the SEs *and* not have a viable search facility on the site. One or the other, maybe, but not both.

Brett_Tabke:
12million page views (by rogue bots) last week while we were away at the conference ... it is not uncommon to have more than 1000 visitors that will view more than 500 pages a day or 200 visitors that will visit more than 1000 pages in an 8hr day or 50 visitors that view more than 2000 pages a day ... a solution is being tested and worked on

And I thought I had problems with rogue bots! [webmasterworld.com] - I wish you the best in sorting this.

PS

No single human can read and digest 500 pages in 8 hours. Now, I understand that there may be more than one human behind a single IP, but it seems reasonable to me to insist that any single login has a daily limit of pages.

balam

10+ Year Member



 
Msg#: 9593 posted 1:28 pm on Nov 23, 2005 (gmt 0)

> because if we allow Google back in today it still going to be 180 days

Brett changed robots.txt - he didn't use the URL removal tool. BIG difference.

> I have had a client question me about a quote that GG said in a one of the Jagger Threads [...] NOW I can't even find the dam thing

Did you ask the client where it was? You can't be blamed for not finding it. If the client tries that, and they're not aware of this thread, point it out to them - "circumstances beyond my control" sorta thing.

<aside>
Fortune cookie of the day: Client who visit WebmasterWorld soon boss.
</aside>

bateman_ap

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 1:48 pm on Nov 23, 2005 (gmt 0)

Brett changed robots.txt - he didn't use the URL removal tool. BIG difference.

I don't think that stops anyone else using it though...

Do you think this will reduce the amount of Spam in the forum? Noticed a lot recently, especially around the weekends.

MatthewHSE

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 2:00 pm on Nov 23, 2005 (gmt 0)

I hope something for the site search can be found quickly. There's one thread I need right now and can't find. Google is already returning no results, and this is apparently affecting AllTheWeb, too - their results are no help at all.

This may make flagging threads much more important. That, and saving pages with ScrapBook.

DaveN

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 2:39 pm on Nov 23, 2005 (gmt 0)

balam the removal tool was used

ken_b

WebmasterWorld Senior Member ken_b us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 9593 posted 2:58 pm on Nov 23, 2005 (gmt 0)

How does this affect the ability to attract new members? It seems like it would be a lot harder.

randle

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 3:00 pm on Nov 23, 2005 (gmt 0)

It would appear the fundamental symbiotic relationship between webmaster and search engine has been inexorably tilted in this situation to the point of saying Iím going my own way. Just pulling out of the crawl and rank game; pretty courageous move.

vincevincevince

WebmasterWorld Senior Member vincevincevince us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 9593 posted 3:34 pm on Nov 23, 2005 (gmt 0)

How does this affect the ability to attract new members? It seems like it would be a lot harder.

I don't know, but I have a sneaking suspicion that this may have been part of the move. Perhaps cutting down on walk-in-members for a while, whilst remaining open to friend-recommended (and hence pre-qualified) members could well shift the community to the way it was last year or the year before. There have been countless discussions about the changes in community ethos and the influx of spammers or low quality posts. I'd give you links but I can't search for them :-) You can see some attempts to deal with them in the pre-moderation.

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 9593 posted 4:02 pm on Nov 23, 2005 (gmt 0)

Personally, if Brett has the manpower, it wouldn't bother me a bit if all the forums were pre-moderated. There does seem to be a lot more junk coming in then there was a few years ago. That's what happens when a site becomes as popular as this site. I swear that I have seen multiple new users that are really the same person re-registrating again and posting. Sometimes to spread confusion, sometimes to rant... over and over and over again. People know the plex is looking and they wll do just about anything for a chance to be heard (or seen as is the case here).

Bold move Brett.

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 9593 posted 4:08 pm on Nov 23, 2005 (gmt 0)

Well Brett, you certainly have made a fine mess of things - can't search the site, webmasterworld missing in Google - lovely PR 0. The upside is that people booted out of AdSense can't find us for their first post anymore so I'm torn about this change of events as the positives are pretty good so far.

This 223 message thread spans 8 pages: < < 223 ( 1 [2] 3 4 5 6 7 8 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Local / Foo
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved