Google crawling dynamic URLs with multiple variables

mysite.com/?section=foo&sub=this+that

[Finder]

My site was deep-crawled on the 6th for the first time ever. Googlebot is taking dynamic pages with multiple variables that it previously ignored. In the past, only Fast has indexed these pages.

I am eager to see how this will affect my rankings.

[nipear]

Same here, I have some multipe variable pages that haven't been in the index since last May get crawled. And some are already showing in the serps! I haven't looked into it but I'm assuming freshbot picked them up the other day!

[DerekT]

At the moment I have seen URLs with up to 2 variables being crawled by Google. More than 2 and they are being ignored.

[Finder]

Here's an example of one it picked up from my site:

www.mysite.com/dir/?section=other&auth=Jane+Smith+writing+as+J.+N.+Doe

That's the longest URL in my entire site. I was so happy to finally see Googlebot index it.

[DerekT]

Finder.

Even though that URL is long, it still only has two variables.

#1: section=other
#2: auth=Jane+Smith+writing+as+J.+N.+Doe

On my site I use rewrites heavily to make MySQL & PHP generated pages look like .html pages. The lengths of some of the URLs are incredible with all the variables listed, and they are being crawled.

Example:
www.site.com/p2-cat781-si-page2-sort1-per12-pass-userAnna-type2.html

[bsnrjones]

DerekT,
I want to do the same thing with ModRewrite for my dynamic pages. BUT I need help! I have spent hours playing around with the code to get it to work, but I just can't.

Would you mind sharing some info with me?

Burke

[Finder]

DerekT,

Yes, you are right. I originally had three variables but shortened it to two when I realized Google wasn't indexing them. Unfortunately it still wouldn't take the page because of the multi-part second variable. Now it is.

If anyone wants help with mod_rewrite, do a site search for the term and you'll hit a goldmine. There are lots of helpful folks here but they aren't going to comb through the busy Google News forum. Try the Perl forum.

[DerekT]

bsnrjones

Here is an example :

This would be contained in your .htaccess file on your Apache server.

<Files .htaccess> 
deny from all 
</Files>
RewriteEngine On
Options -indexes
RewriteRule ^profile(.*).html /member.php?ppaction=profile&uid=$1 [L]
RewriteRule ^album(.*)-user(.*).html /showgallery.php?cat=$1&ppuser=$2 [L]

I will explain each line.

The first three lines prevent people from viewing the contents of your .htaccess file. (Security)

The 4th line turns the rewrite engine to on. This must be turned on in order to do the rewrites.

The 5th line turns off autmatic index creation on directories without an index file. (Security)

Now on the the actual rewrites.

The 6th line uses only one variable. So if I call www.site.com/profile10.html the server extract the value between "profile" and the ".html" and makes $1 the value. So now member.php?ppaction=profile&uid=$1 becomes member.php?ppaction=profile&uid=10

The 7th line uses two variables. I call www.site.com/album23-user4.html the server extracts the value between "album" and "-user" making this value variable $1. The next portion of the url is between "-user" and ".html" taking this value variable $2.
So now album www.site.com/album23-user4.html becomes www.site.com/showgallery.php?cat=23&ppuser=4

You can take these examples and make .html pages with as many variables as you wish.

Hope this helps.

Derek