Getting Gbot to dig deeper

Old robots file causing problems

[johnser]

I placed a robots.txt file on my server in early Sept that disallowed all spiders (as I was putting a revamped site live) & I forgot to remove it. Smart, yes I know!

Anyway, since I removed it about the 10th Oct, I'm being visited by Googlebot but my logs are only showing entries like:

=======================================
crawler12.googlebot.com - - [09/Nov/2002:04:51:31 -0500] "GET /robots.txt HTTP/1.0" 404 204 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

crawler12.googlebot.com - - [09/Nov/2002:04:51:33 -0500] "GET / HTTP/1.0" 200 26818 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
=======================================

..........& its not deep-crawling the pages like it used to.

When I search for the site, theres no "cache" entry which there was a few days ago. Indeed, the cache entry on the 17th Nov showed the home page had been picked up. Just the DMOZ listing shows up and a few external links.

Site has 170 inbound links, PR5, is 18 months old & was redesigned in keeping with best seo practice in early Sept.

Qs >>>>

Why am I not being cached? - or was the above brief cache appearance due to Everflux & I should be ok next week on the dance?

How do I encourage Gbot to crawl deeper? All 30+ pages on the site are all in the root & linked to by standard <a href> image links with a few text links thrown in for good measure.

Am living nervously at present so all clues gratefully received!
Thanks
J

[Brett_Tabke]

Give it until the next full crawl to index all the site. Right now, just "fresh bot" is out sniffing around. It's figured out that you don't have a robots.txt file now, so it's just a matter of waiting.

[johnser]

Thanks Brett - Was hoping to get back in for Dec but it looks like 2003 now. Oh well, only money ;)
J

[Grumpus]

Try submitting it to Google again. It won't hurt, though it might not help you out. With the freshbot on the make and a PR of 5 you might be able to keep a few dozen pages in the index if you keep updating all the time and tweaking it while you're waiting for the deep crawl.

G.

[Brett_Tabke]

Good point Grumpus - might do the trick. And surf your whole site with the toolbar ON ;-)

[johnser]

Hi Grumpus

Have submitted via add url (& toolbar) about 3 times over last 6 weeks. Whats getting to me is that none of my pages are showing up at all. They all have ok PR of at least 3+

Will just have to see what happens next week....
J

[Grumpus]

You can also increase your chances of getting a freshcrawl if you find a (freshed) links page on another site and get yourself a link from there. I realize that's not always the easiest prospect, but external links from pages that get freshed really seem to help give you your "kickstart".

G.

[johnser]

I've got a few good links from PR6 sites which do appear regularly with a recent date on them so I'm hoping that'll do the trick.
Thanks again people
J