Getting Google to Index More than just the homepage...

[farside847]

I checked my log files and found out that Google
grabs my robots.txt and my home page every day, but
unfortunately it does not follow any of the links
on the page. I had a few easy questions that I think
I already know the answers too but - I though it would
not hurt to ask them anyway...

- Do links need to be "A HREF" or will "a href" work too?
- Can links be dynamic? (<a href=/tools/index.html>)
- Does Google only process the first 10k, or the entire file?
- Are META Refresh pages ignored? Even if they go to to another page in the same domain?

Thanks!
Shawn

[Birdman]

That's probably "freshbot". Check out the Google Knowledge Base [webmasterworld.com]. It will keep you busy for a good while;)

[farside847]

Actually, I did read it before I posted...

thanks
Shawn

[farside847]

The IP started with 216 so I think it is not the
freshbot. Any information is greatly appreciated.

thanks
Shawn