Google Won't Come

google stopped coming or keep reading index and goes away

[Penguinsnob]

Quesiton1: google bot used to come everyday and now it stopped coming over. right after I put my robots.txt, google started crawling my site (sort of), then when the label site linked me, it kept coming daily but now it went away (holiday break?). I wonder why.

Question2: Google comes over but it only read my main index page then goes away. it reads my welcome message where it has site update infomation and have navigation bars and title bars. I add a new page but it doesn't want to read. It's weired because the bot once crawled though all the site, and that was it. And some how it is selective as to which page the bot reads. I have about 9 or 10 concert posted on the site, but google only read about 4 or 5 of them. why is it doing this?

[brotherhood of LAN]

Hello penguinsnob, I have a few questions too ;)

1) Do you have links from other sites pointing to yours? Do you have a listing in DMOZ etc
2) Do you have the Google toolbar and do you know the Pagerank of the pages on your site
3) How old is the site
4) What sort of links do you use to link to your concert pages and are they on the home page or a couple of clicks from the homepage

there are tons of reasons for why the bot will behave one way or another, all depends on the cirumstances mostly to do with the way your site naviagation is set up. It's not easy giving a reason without knowing a bit more about the site googlebot is inspecting.

Lots of info can be found on googlebot using the site search [searchengineworld.com] if you want to find out more in spare time ;).....

[duncan12]

You may be surprised to learn that Google doesn't even try to crawl the WHOLE net. The net is massive, and people only visit a relatively small portion of it, so the law of diminishing returns begins to apply.

If you want Google to crawl your site, you should install the Google Toolbar, and then visit all of your pages. Uninstall it, reinstall it and do it again. Google will think multiple people are visiting your site, and they will crawl it.

[pageoneresults]

If I understand you correctly, right after you placed a robots.txt in your root directory, there were some problems?

right after I put my robots.txt, google started crawling my site (sort of)

Have you validated your robots.txt file to make sure there are no errors and you are not chasing GoogleBot away?

Robots Text Validation [searchengineworld.com]