Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
The URL i submitted to Google has now been listed (only this morning) and i have received a good ranking for probably the sites most important keyword. The trouble is Google only appears to have listed the root URL.
I have tried utilities to check that the site can be indexed correctly with search engines. Does it take time for the rest of the pages to show up on Google? or perhaps the pages have been dropped?
I did not use the robots.txt file as i was under the impression that if the file did not exsist then the spider was free to index the full site is this correct?
Any help or advice anyone can give me would be greatly appreciated.
The trouble is Google only appears to have listed the root URL.
I think this might be the FreshBot so your site may disappear for a few days, but dont worry the DeepBot will eventually index your site and if your SEO is good you will have good rankings.
[edited by: nileshkurhade at 6:42 pm (utc) on Dec. 22, 2003]
Some people think a robots.txt might be desirable, but you're quite right that if you don't have one, it means 'go ahead and crawl', as robots.txt is designed to exclude. My main site didn't have one for years and were listed fine. Personally I'd be pleased to get your index page in, if it sticks it's a great start!
The DC bot has been acting a lot stranger the last 3 or 4 months. I would get a regular deep crawl on all my sites quite often but now it varies greatly which site gets deep crawled if at all.
It doesn't seem as active as it used to be.