homepage Welcome to WebmasterWorld Guest from 54.166.84.82
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Google crawling dynamic URLs with multiple variables
mysite.com/?section=foo&sub=this+that
Finder

10+ Year Member



 
Msg#: 9188 posted 7:54 pm on Feb 9, 2003 (gmt 0)

My site was deep-crawled on the 6th for the first time ever. Googlebot is taking dynamic pages with multiple variables that it previously ignored. In the past, only Fast has indexed these pages.

I am eager to see how this will affect my rankings.

 

nipear

10+ Year Member



 
Msg#: 9188 posted 8:02 pm on Feb 9, 2003 (gmt 0)

Same here, I have some multipe variable pages that haven't been in the index since last May get crawled. And some are already showing in the serps! I haven't looked into it but I'm assuming freshbot picked them up the other day!

DerekT

10+ Year Member



 
Msg#: 9188 posted 12:19 am on Feb 10, 2003 (gmt 0)

At the moment I have seen URLs with up to 2 variables being crawled by Google. More than 2 and they are being ignored.

Finder

10+ Year Member



 
Msg#: 9188 posted 12:32 am on Feb 10, 2003 (gmt 0)

Here's an example of one it picked up from my site:

www.mysite.com/dir/?section=other&auth=Jane+Smith+writing+as+J.+N.+Doe

That's the longest URL in my entire site. I was so happy to finally see Googlebot index it.

DerekT

10+ Year Member



 
Msg#: 9188 posted 12:49 am on Feb 10, 2003 (gmt 0)

Finder.

Even though that URL is long, it still only has two variables.

#1: section=other
#2: auth=Jane+Smith+writing+as+J.+N.+Doe

On my site I use rewrites heavily to make MySQL & PHP generated pages look like .html pages. The lengths of some of the URLs are incredible with all the variables listed, and they are being crawled.

Example:
www.site.com/p2-cat781-si-page2-sort1-per12-pass-userAnna-type2.html

bsnrjones

10+ Year Member



 
Msg#: 9188 posted 4:53 pm on Feb 10, 2003 (gmt 0)

DerekT,
I want to do the same thing with ModRewrite for my dynamic pages. BUT I need help! I have spent hours playing around with the code to get it to work, but I just can't.

Would you mind sharing some info with me?

Burke

Finder

10+ Year Member



 
Msg#: 9188 posted 5:20 pm on Feb 10, 2003 (gmt 0)

DerekT,

Yes, you are right. I originally had three variables but shortened it to two when I realized Google wasn't indexing them. Unfortunately it still wouldn't take the page because of the multi-part second variable. Now it is.

If anyone wants help with mod_rewrite, do a site search for the term and you'll hit a goldmine. There are lots of helpful folks here but they aren't going to comb through the busy Google News forum. Try the Perl forum.

DerekT

10+ Year Member



 
Msg#: 9188 posted 8:26 pm on Feb 10, 2003 (gmt 0)

bsnrjones

Here is an example :

This would be contained in your .htaccess file on your Apache server.


<Files .htaccess>
deny from all
</Files>
RewriteEngine On
Options -indexes
RewriteRule ^profile(.*).html /member.php?ppaction=profile&uid=$1 [L]
RewriteRule ^album(.*)-user(.*).html /showgallery.php?cat=$1&ppuser=$2 [L]

I will explain each line.

The first three lines prevent people from viewing the contents of your .htaccess file. (Security)

The 4th line turns the rewrite engine to on. This must be turned on in order to do the rewrites.

The 5th line turns off autmatic index creation on directories without an index file. (Security)

Now on the the actual rewrites.

The 6th line uses only one variable. So if I call www.site.com/profile10.html the server extract the value between "profile" and the ".html" and makes $1 the value. So now member.php?ppaction=profile&uid=$1 becomes member.php?ppaction=profile&uid=10

The 7th line uses two variables. I call www.site.com/album23-user4.html the server extracts the value between "album" and "-user" making this value variable $1. The next portion of the url is between "-user" and ".html" taking this value variable $2.
So now album www.site.com/album23-user4.html becomes www.site.com/showgallery.php?cat=23&ppuser=4

You can take these examples and make .html pages with as many variables as you wish.

Hope this helps.

Derek

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved