Googlebot is back!

Forum Moderators: open

Message Too Old, No Replies

Googlebot is back!

It looks like the deep crawl is starting.

RoadRash

9:28 am on Mar 9, 2003 (gmt 0)

Google started to deep crawl my site in the last 15 minutes. Anyone else notice this?

PaulPaul

5:23 pm on Mar 15, 2003 (gmt 0)

Oaf357: I think he means that his robots.txt file which resides on his development server, which he uses to disallow all robots from the development server, got copied over to his production server when he implemented his recent changes.

Gibble: You might have a problem..

On a different note: I am still waiting to see the bot to grab 1000 more pages.. She only picked up about 100 so far for me :(

Alternative Future

5:26 pm on Mar 15, 2003 (gmt 0)

Deepbot finally arrived here (UK) and has found 400 pages so far Yahoo ;-)

-gs

Jesse_Smith

6:26 pm on Mar 15, 2003 (gmt 0)

It's been bombing me for three days now.

Alternative Future

6:29 pm on Mar 15, 2003 (gmt 0)

Well here is hoping that it continues to *bomb* me for another few days also ;-) as it has a lot more work to do before its finished...

-gs

teeceo

10:54 pm on Mar 15, 2003 (gmt 0)

I'm getting hit by freshbot(on new sites)will this count the same as a deepbot visit in getting my sites in for the next update?

teeceo.

Jesse_Smith

11:02 pm on Mar 15, 2003 (gmt 0)

It pushed my server load up to 24.56, 10.49, 5.14 for about five minutes, slowed the server down and looked like it would crash. Normally it's under 3! That's the kind of bombing I'm geting! Very good bomb!

The freshbot is only a few days listing, unless it keeps visiting the pages.

freshBot: 64.68.82.* bah, listed for only a few days, short dinner date then it dumps you.
deepcrawler: 216.239.46.* Good bot, it likes you, listed until death do you depart. So don't make him mad or it will divorce your site. You can divorce the Googlebot by using your robot.txt file. It's much cheaper and faster than going to the courts.

bignooz

12:21 pm on Mar 16, 2003 (gmt 0)

I can see googlebot crawling quite fast since march 15th.
CRAWL BABY CRAWL! :))

ScottM

4:04 pm on Mar 16, 2003 (gmt 0)

Interesting. I'm seeing a fresh cache for my website and yet I haven't seen either bots in 13 days...

plasma

4:14 pm on Mar 16, 2003 (gmt 0)

>Interesting. I'm seeing a fresh cache for my website and yet I haven't seen either bots in 13 days...

Maybe you look in the wrong place :p

bignooz

1:30 pm on Mar 17, 2003 (gmt 0)

Googlebot IP 216.239.X.X still crawling my server.
May it suppose an other late update for MARCH?

skipfactor

1:47 pm on Mar 17, 2003 (gmt 0)

I hope so. It's hitting my pages like 1 page every 4 hours with a few hundred to go.

Gibble

2:09 pm on Mar 17, 2003 (gmt 0)

Well Googlebot has grabbed a page since the robots.txt was fixed...hopefully she grabs more!

This 102 message thread spans 4 pages: 102