Welcome to WebmasterWorld Guest from 188.8.131.52
This is the killer app. If I can't search WebmasterWorld like I can now with Google I wouldn't come back so often. I've found that mostly any query can be answered now with a properly formed query.
Perhaps time to invest in one of those Google Boxes...?
Having said that, I can see where you are coming from, and I hope it works out for you.
In the case of Google, there is no mention of support for it on their bot page, Googlebot: Google's Web Crawler [google.com]. In fact, I believe Google would suggest the use of SiteMaps if you wish to throttle Googlebot.
From the above link:
3. Googlebot is crawling my site too fast. What can I do?
Please contact us with the URL of your site and a detailed description of the problem. Please also include a portion of the weblog that shows Google accesses so we can track down the problem quickly.
>>Honestly, yes it does. I have had 4 very nice nights of sleep
Brett why are you loosing sleep over a website?
You must know something that all of us have missed Brett, it's gone already in google UK the rest will follow shortly..
http://www.google.co.uk/search?q=webmasterworld.com in the uk webmasterworld is gone in google
Same MSN. Still OK in Yahoo but showing less results than expected and I guess it's a matter of time before that is the same.
The vast collective of archived WebmasterWorld threads only exists if you can find them - via the search engines - now we can't find a single post unless we hunt through reams and reams of links...
I guessed at 5 days, I have had a client question me about a quote that GG said in a one of the Jagger Threads, it's kinda Cryptic so I need to read it a few times more.. NOW I can't even find the dam thing
MSN has 1 pages
Google has 0 pages
Yahoo is dropping them fast than I can search!
a solution is being tested and worked on : which means? we have a framework in place.. how long before we will see some results..
because if we allow Google back in today it still going to be 180 days
from google :
URL removal system will cause a temporary, 180 day removal of your site from the Google index, regardless of whether you remove the robots.txt file after processing your request.
12million page views (by rogue bots) last week while we were away at the conference ... it is not uncommon to have more than 1000 visitors that will view more than 500 pages a day or 200 visitors that will visit more than 1000 pages in an 8hr day or 50 visitors that view more than 2000 pages a day ... a solution is being tested and worked on
No single human can read and digest 500 pages in 8 hours. Now, I understand that there may be more than one human behind a single IP, but it seems reasonable to me to insist that any single login has a daily limit of pages.
Brett changed robots.txt - he didn't use the URL removal tool. BIG difference.
> I have had a client question me about a quote that GG said in a one of the Jagger Threads [...] NOW I can't even find the dam thing
Did you ask the client where it was? You can't be blamed for not finding it. If the client tries that, and they're not aware of this thread, point it out to them - "circumstances beyond my control" sorta thing.
Fortune cookie of the day: Client who visit WebmasterWorld soon boss.
This may make flagging threads much more important. That, and saving pages with ScrapBook.
How does this affect the ability to attract new members? It seems like it would be a lot harder.
I don't know, but I have a sneaking suspicion that this may have been part of the move. Perhaps cutting down on walk-in-members for a while, whilst remaining open to friend-recommended (and hence pre-qualified) members could well shift the community to the way it was last year or the year before. There have been countless discussions about the changes in community ethos and the influx of spammers or low quality posts. I'd give you links but I can't search for them :-) You can see some attempts to deal with them in the pre-moderation.
Bold move Brett.