Is it true that spiders can't crawl a URL with a query string?

[georgiek50]

I've been testing my website with various online spiders to find that this is not possible.

I also ran into some information that Apache's mod_rewrite can take care of this. What are your opinions on this matter?

[DanA]

I don't think so,
search engine robots do follow them and I know a few offline mirror utilities that follow an URL with a query
string and give very good results.

[jatar_k]

Based on a few years of doing this

1. query strings cause problems
2. keep them as short as possible, more than 2 vars is a bad idea
3. no query strings is better
4. just because something is spidered doesn't mean it will rank
5. mod_rewrite is a good option if you can't get rid of the query strings using different methods of programming

a good thread
An Introduction to Redirecting URLs on an Apache Server [webmasterworld.com]

[jmccormac]

No. Unless the search engine operator has specifically blocked '?' in an URL, URLS with? will be included.

Regards...jmcc

[georgiek50]

Thanks for the replies guys, although it seems that not everyone agrees with everyone. I should probably check out mod_rewrite just to be in on the safe side.

[somerset]

We have found consistantly, that querystring urls do eventually get spidered, but no where like as quickly as standard url's.

We have also found that if the site has good PR (6+) then it seems to almost 'push' the spiders in there.

On the other hand, we have had low PR sites who's dynamic (querystring) urls have not been spidered and included in Googles search results even after many months.

In conclusion, standard links get spidered and ranked very much quicker, unless you have a high PR large busy site.

That what we have found accross many unrelated websites.

[georgiek50]

After reading up on this a bit, it doesn't seem too difficult but I just wanted to post an example of my understanding because maybe I'm missing something obvious.

My dynamic content consists of product news and product reviews and are retrieved from MySQL with the following syntax:

/news.php?id=7&type=news or news.php?id=7&type=review
depending on the type of article. So this basically boils down to having a RewriteRule like so:

RewriteRule ^/articles/([^/]+)/([^/]+)$ [mysite.com...]

And then change all my URLs to look like so:

/articles/news/7 or /articles/review/7

Is that the basic idea?

[jatar_k]

could be, rewrite rules aren't my specialty. ;)

That is the general idea but if you had a more specific question the experts are here
Apache Web Server Forum [webmasterworld.com]