Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
How long should I wait , Did I Do something wrong?
I don't have a robots.txt & I have the meta (index,follow).
When do I get deep crawled .
5. How long does the Google robot take to index a URL once it's been submitted?
Depending on the timing of the submission and of our crawl, the entire process can take between six and eight weeks.
This has been their response for years, and even with freshbot they haven't changed it.
What I have found is that without deep links, you can expect to get your index in the first month, a moderate crawl the second month, and a complete crawl the third month. That has been for sites of a few hundred to a few thousand pages.
If you have deep links, it seems to behave that way for each link, so you can get a lot more of your site crawled sooner by getting those links.
Of course, getting good quality high PR links will urge googlebot on even more.
That's the trick.
Think of Googlebot as aimlessly following links around the web. It generally likes high PR pages better simply because they have more incoming links (that's how they got their high PR).
To get a good/fast deepcrawl you need a large number of good deep links. Then the law of averages (and PR) will come into play and Googlebot will get to your deep pages much quicker.
For a new site with only links to the index page I generally plan on 60 days for a complete crawl of 20K to 30K pages. Getting some strategic deep links can bring that time down to 20 to 30 days.
Could be 1 day, or 1 month. It depends on how many links your site has, and your timing. Sites with good PR or a high number of links get crawled frequently. Lesser linked sites need to wait for the "crawl a huge chunk of web" googlebot cycle, which isn't on any known schedule.
Took over the marketing around the same time for two sites that had zero backlinks and were indexed as 'Supplemental Results'.
Re-did the titles, etc. and added a few relevant PR4 pointers, and the index pages are pretty much freshened daily, but nothing else is crawled & all other pages still show up as 'Supplemental Results' after a month.
Perhaps it's just the newbie test bots wreaking havoc: [webmasterworld.com...]