Forum Moderators: open
teeceo.
I have been seeing alot of Inktomi deep craw’s to that is a first for me. About 600 hit's so far on this crawl
Freshbot has been hitting me up for two days then WHAM... Deepbot is tearing through with Inktomi bringing up the rear.
Smiles.
My terrible unreliable host that I can't regain my domain name from went down today and hasn't come up yet. I want to scream.
I understand it could take 10 days to crawl.
-Adrian
[edited by: Marcia at 5:19 am (utc) on April 19, 2003]
[edit reason] No sigs, please. [/edit]
question-
when a site that was just deep botted show up in the index?
one site is new, and it was fresbotted like last week, disappeared( I know why cause I read here about freshbots)
now when will it be re-indexed?
also on another site,
which is older and bigger, it it showing a really old index page in google's index. ( this domain is available page).. I snatched it up someone let it expire.
this site was debotted too, when will the real page show?
The deepbot has been hanging around my page for a couple of days now but seems to be crawling extremely slow. My site has a low PR (it is very new) but has about 5,000 or so pages. This is my second deepcrawl and I was hoping google would gobble all of it...but judging by the freqeuncy of it hitting me it wont. Is this a good way to judge how much the bot likes me...should it be going faster?
Thanks,
Kevin
You need to read the rules for this forum, you can not have a URL in your post.:)
>>I deleted hundreds of old pages yesterday in the middle of deep crawl.
Some may have been indexed, some may not have been. You might have a few 404's for an update, no big deal.
Looking through the logs, at one point we have 87 of them together with 4 freshbots in the site for 4 hours continuous at about 4 hits a second.
Madness.
Still getting deepbotted now, but only 3 of them in there and being reasonably well behaved and gentle. So I think they fixed the problem, but I haven't received a phone call or email yet. I'll let you know the outcome.
I never thought I'd see the day where I had my finger hovering over the "ban googlebot" button.
;-)
TJ
On what day of the month should you plan to have changes completed on your site to be ready for the crawl?
And how can you monitor the crawl on your site?
Thanks.
Thanks
A few questions about bot algos...
How does a bot move through your site, does one goto a page and then grab all the links then other bots come for those links and so-on? Or does one bot come and visit more than one page at a time, I seem to have about a dozen coming at once, but not sure of their direction :)
Thx
My site is ossau dot net if one of you can notice a problem... maybe in the robots.txt
I don't think because other bots like Pompos have no problem to crawl my site.
help!
Peter
I've seen the deep bot hit only the homepage of a PR6 site this time round - very strange indeed.
Or maybe it's not over...
Is there anything I should be worried about?
Leo
e.g.
IP Address 216.239.46.239
Reverse DNS crawl9.googlebot.com
From / Via Biznessonline
Origin Location New York, USA.
Time Spent 12 min
Hits / Kilobytes 1 / 2.64Kb
Browser Tag Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Each bot seems to take one file but it's a real page, not robots.txt .
*dream-modus on* just index all and every page and put them on no. 1 position next month *dream-modus off*)
