? in URL is crawled slower (dynamic pages)

and take longer to get in.

[ogletree]

Daniel Dulitz - Google Engineer Interview [e-marketing-news.co.uk]

We deal very well with dynamic delivery per se. So
the mere presence of a question mark in the URL doesn't
throw us for a loop at all. We will crawl those pages more
slowly than we do with pages without question marks in the
URL. Purely because they've identified themselves as being
dynamic and we certainly don't want to bring anyone's site
down because we're hitting the database too hard.

He also goes on to say that Google is very careful with dynamic URL's. I and a few other members here at WW have recently changed dynamic URL's using URL rewrites and other methods to make sure the whole site looks like .html files. We all saw pages indexed that we have been trying to get indexed for quite some time and this was within a week or two. In my case all my dynamic url's were spidered a long time ago Google just never put them in. I now have the same pages indexed with .html ext. I have tried this on several different occasions to see if it was just a fluke.

I do know that there are some of you out there that have? in your URL and have tons of pages indexed and they get indexed quickly and you rank great for everything. I bet you have a high PR and/or your site is old. You may also have sites linking to you deep.

I must give credit to martinibuster for posting this article in another thread [webmasterworld.com]

[ciml]

That's right ogletree, having 'dynamic' looking URLs doesn't prevent you from being spidered by Google^(*), but it makes it less easy to be spidered deep.

^(*): unless you have id= in the URL.

[nakulgoyal]

I agree. That is why everybody on the web try to staticize their websites as much as possible and indeed it works.

[Ledfish]

We just finished doing rewrites on our URL's. We had a couple of problems, first was the? second was that we didn't have "id" alone in it, but as part of the query. For example, many times you see people asking about something like .asp?id=1234, but ours where .asp?idproduct=123. I believe from the spiders perspective, there really isn't any difference because what it see is .asp? followed by "id" in either case and once it see that id after the?, it stops and gets out.

My basis for this is that we have a page that gets called using like this viewprd.asp?idproduct=123. Now, if you just manually call viewprd.asp, you get an error message. It is only worthwhile if it has the query attached. Looking through my logs, google has been hitting viewprd.asp almost 10 time a day, yet the page isn't listed, none of our product pages are listed.

[takagi]

I must give credit to martinibuster for posting this article in another thread

Actually, dillonstars mentioned the article earlier in the thread Interview with Daniel Dulitz [webmasterworld.com]

[ogletree]

Very sorry I missed the that thread. Thank you for pointing it out.