Forum Moderators: open

Message Too Old, No Replies

Bot doesn't crawl new pages but older ones..

         

minnie

11:26 am on Mar 6, 2003 (gmt 0)



I'm new here, I read the FAQ and many threads, but I haven't found an answer yet.

10 days ago I submitted several new pages on several sites. But Googlebot didn't visit any of them so far, and it continues to crawl already indexed pages from time to time.

Do my new pages still have a chance to appear in March listings? Seeing that Google will update in 2 days I don't have much hope :/

troels nybo nielsen

11:33 am on Mar 6, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Welcome to WebmasterWorld, minnie

The imminent update will be based on crawls that have already been made.

You may want to read this thread: [webmasterworld.com...]

[edited by: troels_nybo_nielsen at 11:35 am (utc) on Mar. 6, 2003]

lazerzubb

11:34 am on Mar 6, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Welcome to Webmaster World [webmasterworld.com]

The safest way to get them indexed is by linking to them.
This way you might be able to get a freshtag on them.
Googlebot: Deepbot and Freshbot FAQ and Information [webmasterworld.com]
Google Updates and Everflux, the Monthly Mid-Cycle Changes [webmasterworld.com]

Emma McCreary

11:34 am on Mar 6, 2003 (gmt 0)

10+ Year Member



10 days?! I wish! Minnie, Google just isn't that fast...no search engine is.

From what I understand, the deep crawler (which adds new pages) doesn't usually see a lot of activity this close to the update. Freshbot (which crawls pages already in the index), is still out there roaming around and is probably the one that crawled your other pages.

The deep crawler comes back strong soon after the update starts, so it will probably find your pages then, and they will show up on Google in a month or two.

Keep in mind as well that Google prefers to find new pages by following links rather than having them submitted.

The word for dealing with search engines is: patience.

netguy

12:11 pm on Mar 6, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Lazerzubb... reading the Googlebot FAQ you outlined, it stated that deep crawl was IP 216, while FreshBot is 64. Do you are anyone else know if this has changed?

I have consistently been visited on several sites by 216.239.**.** over the past several days (including this morning). Obviously, with just hours before the dance, Google must be using their 216 servers for FreshBot as well?

lazerzubb

12:21 pm on Mar 6, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



netguy, i know they added quite a few more ip's the last month, haven't had time to look in to it, someone here might be able to help? (Brett ;) )

minnie

12:29 pm on Mar 6, 2003 (gmt 0)



You are all great. Thank you for your answers. I read tons of pages but didn't hear about these tips before.

Roscoe

2:33 pm on Mar 6, 2003 (gmt 0)

10+ Year Member



netguy.... all my 1800 pages have been crawled over the last 3 days by 64.68.*.*

I assume this is deep crawl, since freshbot usually looks only at 10 or so pages - usually :}

netguy

2:38 pm on Mar 6, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Roscoe... sounds logical that it would be 64, so I don't know where the 216.239.**.** is coming from. Guess Google likes to keep us guessing what they're up to.

annej

3:21 pm on Mar 6, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Minnie,

I temporarily link new pages from my homepage which the freshbot visits regularly. You can title it something about new info on the site.

Not only does this get the fresh bot to the new pages fast it also gets visitors to check out new material. Plus you are updating your homepage more often and that keeps the fresh bot comming.

Anne

netguy

3:27 pm on Mar 6, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Good point Annej... Google can pick up new pages in a day or two coming off a page it regularly visits, while the Google ADD-URL can take months or longer.