Welcome to WebmasterWorld Guest from 54.235.25.206

Forum Moderators: open

Message Too Old, No Replies

Googlebot is Crawling

Starting seeing some action

     
5:17 am on Jul 31, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:May 14, 2002
posts:378
votes: 0


Seeing a few requests for the home pages of a few sites! Starting a couple of hours ago.
9:08 am on July 31, 2002 (gmt 0)

New User

10+ Year Member

joined:Apr 29, 2002
posts:33
votes: 0


Yes!

Though it seems I've banned (or 20k droped) I've seen in my log yesterday it came to my website and got 365 pages! That's good, 'cause many of the URL are not on google index today, so it's seems some of us dropped on last update are going to be "taken" again.

Or not? :)

Regards (and sorry for my English)

9:21 am on July 31, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member rfgdxm1 is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 12, 2002
posts:4479
votes: 0


Google dropped my site in the garbage like a box of used Kleenex last update, even though I was crawled. My site was just again crawled by Googlebot today. However, I'm not getting my hopes up. :(
9:31 am on July 31, 2002 (gmt 0)

New User

5+ Year Member

joined:Oct 20, 2008
posts:
votes: 0


I seem to have had a similar experience
My site was orignally PR5-6 went to PRO on update which I couldn't understand, later went grey,now I note Gogglebot visited 550 pages in last days.

Although none of my pages are in index yet I hope I may get back in sometime and I wasn't banned.

One hopeful Google fan

10:09 am on July 31, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:June 3, 2002
posts:142
votes: 0


Please tell us your home page PR and high/low estimate if you see crawling. That data is of interest to a few of us out here.
10:12 am on July 31, 2002 (gmt 0)

Moderator from GB 

WebmasterWorld Administrator brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 30, 2002
posts:4842
votes: 1


nice timing, ive just been uploading those fancy absolute divs and light CSS, hopefully google will find the newer slim code much more tasty food to eat :)

For once, google has crawled at a convenient time.

10:23 am on July 31, 2002 (gmt 0)

New User

10+ Year Member

joined:Apr 29, 2002
posts:33
votes: 0


KakenBetaal:

PR 6. If someone is interested, can see googlebot evolution on my website (see my profile) /logs/googlebot_log

If I see some change on www2 or www3, I'll tell you

10:28 am on July 31, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:May 1, 2002
posts:439
votes: 0


PR7 being crawled as of late last night.

To those who feel hopeless cause they were dropped last update, don't despair. It's very important for my google referrals to have a lot of pages in the index. Two or so months ago, I got cut by some 70% of pages. Then things turned around and the latest update has made a huge difference.

When you get cut from the index or have a lot of pages get dropped, assuming you aren't doing anything nasty, take a good look at your code and your internal navigation. Get a few more inbound links from anywhere you can and maybe in a month or two you might be singin a happy tune.

Go Googlebot, slurp lots and lots of pages!

11:21 am on July 31, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:June 3, 2002
posts:142
votes: 0


Thanks Cyberdark, that was interesting. Nice looking site you have there! It looks like my crawl will be 12-24 hours behind yours as I've just had a few more pages taken. Hey, hey, hey!

I'm hoping the bot will see my newly installed forum and go take some of those pages as well.

On nav structure, I have both a site map (aimed at Googlebot more than users), and a nav bar link to all my main tier 2 pages from every page on the site. That seems to work fairly well for me, and nearly all my pages are indexed regularly. Thanks to WebmasterWorld and all you helpful users for the site map tip - it helped to get far more of my pages indexed.

What may also help my site to be indexed is that I have quite a few external deep links.

3:19 pm on July 31, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2001
posts:2145
votes: 0


Googlebot grabbed a 1000 pages last night.
6:59 pm on July 31, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 5, 2001
posts:2723
votes: 0


PR7 and crawling nonstop many sites.
One has PR 4
and other that was hit is PR 5

I dont have Low PR sites really so those are some results

7:17 pm on July 31, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 13, 2002
posts:662
votes: 0


All it's done is grab my robots.txt files...I'm waiting for the crawl now.

And I'm impatient!

7:55 pm on July 31, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 22, 2002
posts:1807
votes: 1


Does anyone have any suggestions on the best freeware tracking programs? I'm currently using extreme tracking, but I'm sure there has to be something better. I want to be able to watch for the googlebot too!
8:29 pm on July 31, 2002 (gmt 0)

New User

10+ Year Member

joined:June 26, 2002
posts:17
votes: 0


PR5 site - Googled crawled the whole thing early this morning - about 75 pages.
6:16 pm on Aug 4, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 4, 2002
posts:59
votes: 0


pr0 site. Many new incoming links.
Googlebot has been crawling all over me since 2002-07-31 05:08:37.000
and last entry on 2002-08-04 03:35:25.000
I have a dynamic page the SQL Server generates here www.hemsell.com/resources/googlebot.htm if anyone is interested
9:57 pm on Aug 4, 2002 (gmt 0)

New User

10+ Year Member

joined:Apr 29, 2002
posts:33
votes: 0


Update: up to 20000 pages crawled. It hung my SQL server (more than 500 petitions/hour) :(

Let's hope, on August update, site reapears there :)

10:26 pm on Aug 4, 2002 (gmt 0)

Full Member

10+ Year Member

joined:May 23, 2002
posts:232
votes: 0


Googlebot grabbed robots.txt on friday morning and looks like it finished spidering the entire site last night....with the help fo a new site map....thanks WW.
10:39 pm on Aug 4, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member marcia is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Sept 29, 2000
posts:12095
votes: 0


stuntdubl, dig through the Tracking forum [webmasterworld.com], you'll find a lot of them recommended.
11:41 pm on Aug 4, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:May 4, 2002
posts:103
votes: 0



PR5, and it has been crawling for the last several days
3:59 am on Aug 5, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:June 20, 2002
posts:60
votes: 0


if it actually hung your SQL server, you might want to look into using a template engine that is capable of caching repetitive SQL queries. The Smarty template engine is extremely easy to use if you have PHP on your system, and does a whole lot more than caching - but if google is halting your server, you should get some kind of query caching going ASAP to avoid losing traffic etc. or burning out your CPU ;D
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members