I have a question as a starting seo learner

[domin2]

Hi I have set up my website and we have just seen this in our logs. I belive this is a fresh bot as it starts with 64? what is the process after this as it only looked at the front page. Will it come back for a deeper crawl? or will it put me in google? your help would be appreciated

64.68.82.39 - [06/Jun/2003:12:17:17 +0100] "GET / HTTP/1.0" 200 49187 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

[edited by: heini at 2:59 pm (utc) on June 6, 2003]
[edit reason] see your sticky mail please [/edit]

[vitaplease]

This might help.

[webmasterworld.com...]

but things tend to take longer - at the moment.

Try to use non-url specific examples - TOS

[domin2]

HI there thanks. I had a quick look through that page before but Am i right in saying that this was a fresh bot? and if it is it only went the the front page no further. is this ususal? or should it have had a look around? sorry im fairly new to this

[takagi]

Yes, 64 is the freshbot. Read more about it in the Googlebot: Deepbot and Freshbot FAQ and Information [webmasterworld.com] thread.

If you are lucky, it could show up in 2 or 3 days. Maybe with the date is was spidered. However, no guarantees. Google might just as well ignore the data from the freshbot.

[bhartzer]

I'm not sure whether or not freshbot is actually spidering from that particular IP address. Googlebot should be back, though, for a deep crawl. Give it some time and it will be back.

[domin2]

Hi thanks! should I therefore be adding new content to the front page until the deepbot spider comes back?

[takagi]

should I therefore be adding new content to the front page until the deepbot spider comes back?

No need, unless the front page misses keywords that are only to be found in the sub pages.

[domin2]

No everything is there on the front page with regards to what we sell which is mobile games but the actual game name links etc are within the categories 2 pages in. Will the deepbot when and if it arrives pick these up or will a freshbot go past the first page?

[takagi]

Will the deepbot when and if it arrives pick these up or will a freshbot go past the first page?

Freshbot will also go deeper than the first page unless there is a reason not to do so like:
1. robots.txt doesn't allow the bot to spider the sub pages
2. <meta> tells the bot to not follow the links
3. there is a problem with the links (JavaScript, session ID, etc)

Maybe deepbot will visit soon. But since the Dominic update, a lot of things are not sure anymore. There was also recently a thread about freshbot doing the task of deepbot.

[mrguy]

Whether or not the deepbot comes back remains a question. I've seen others post that they recently got deepbotted, yet I have not seen it althought I see freshbot just about everyother day going deeper than it has before.

It has been speculated that with the new system, freshbot actually acts as the deepbot.

We won't know until it's all said and done.

[domin2]

Freshbot will also go deeper than the first page unless there is a reason not to do so like:
1. robots.txt doesn't allow the bot to spider the sub pages
2. <meta> tells the bot to not follow the links
3. there is a problem with the links (JavaScript, session ID, etc)

anyone wanna helpme on this?

[takagi]

You already gave the URL (but it is now removed).

1. The robots.txt is OK

User-agent: *
Disallow:

2. meta is OK

<meta name=robots content="index,follow">

3. Links; At least some should be picked up.

[domin2]

thanks! what im trying to learn is if i should be doing anything between fresh and deep?

[bhartzer]

>should be doing anything between fresh and deep?

Yes, adding more content and more pages to your site. ;)

[domin2]

to the front page with links going back to the pages?

[domin2]

so does a freshie cach the links it sees on the front page of a site when it first visits then if you have more next time it comes round it thinks!growing and expanding" then give it a higer pr rating?

[takagi]

Netscape counted 97 links from the home page. That should be enough for now.

There is one other thing: on the home page you it says "mydomain.com" but some inbound links are pointing to "www.mydomain.com". So search engines will find all sub pages 2 times. For example: a "mydomain.com/links.php3" page and a "www.mydomain.com/links.php3". To prevent that, you could make all links from the home page absolute. Change

<a href="links.php3">related web sites</a>

into

<a href="http://mydomain.com/links.php3">related web sites</a>

[takagi]

so does a freshie cach the links it sees on the front page of a site when it first visits then if you have more next time it comes round it thinks!growing and expanding" then give it a higer pr rating?

'Freshie' is likely to visit the site more often if the page frequently changes. New links or more/other text, doesn't matter that much. And such a visit could give your site a better ranking (closer to #1 position on the SERP) for a few days. But that is not related to a higher PR.

[domin2]

Netscape counted 97 links from the home page. That should be enough for now. ........................

On my site? i can only get any of the search engines to find one link how are you finding 97?

[takagi]

> from the home page
I meant links on your home page to the sub pages. Not inbound links to your home page. Sorry for the misunderstanding (BTW Netscape is a browser, not a SE).