Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: open
when a site that was just deep botted show up in the index?
one site is new, and it was fresbotted like last week, disappeared( I know why cause I read here about freshbots)
now when will it be re-indexed?
also on another site,
which is older and bigger, it it showing a really old index page in google's index. ( this domain is available page).. I snatched it up someone let it expire.
this site was debotted too, when will the real page show?
The deepbot has been hanging around my page for a couple of days now but seems to be crawling extremely slow. My site has a low PR (it is very new) but has about 5,000 or so pages. This is my second deepcrawl and I was hoping google would gobble all of it...but judging by the freqeuncy of it hitting me it wont. Is this a good way to judge how much the bot likes me...should it be going faster?
>>I deleted hundreds of old pages yesterday in the middle of deep crawl.
Some may have been indexed, some may not have been. You might have a few 404's for an update, no big deal.
Looking through the logs, at one point we have 87 of them together with 4 freshbots in the site for 4 hours continuous at about 4 hits a second.
Still getting deepbotted now, but only 3 of them in there and being reasonably well behaved and gentle. So I think they fixed the problem, but I haven't received a phone call or email yet. I'll let you know the outcome.
I never thought I'd see the day where I had my finger hovering over the "ban googlebot" button.
A few questions about bot algos...
How does a bot move through your site, does one goto a page and then grab all the links then other bots come for those links and so-on? Or does one bot come and visit more than one page at a time, I seem to have about a dozen coming at once, but not sure of their direction :)
My site is ossau dot net if one of you can notice a problem... maybe in the robots.txt
I don't think because other bots like Pompos have no problem to crawl my site.
IP Address 18.104.22.168
Reverse DNS crawl9.googlebot.com
From / Via Biznessonline
Origin Location New York, USA.
Time Spent 12 min
Hits / Kilobytes 1 / 2.64Kb
Browser Tag Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Each bot seems to take one file but it's a real page, not robots.txt .