Welcome to WebmasterWorld Guest from 107.23.176.162

Forum Moderators: Ocean10000 & phranque

Message Too Old, No Replies

Non-Existent Pages in Google. Is it due to my .htaccess file?

.htaccess format

     
10:07 pm on Mar 20, 2012 (gmt 0)

New User

5+ Year Member

joined:Mar 19, 2012
posts: 19
votes: 0


I'm using MediaWiki. Instead of long URL's like Site.com/pagename.php?123 I get shorts URLs like Site.com/PageName. I added the hotlink protection line via the Hotlink Protection feature in cPanel.

My problem is that I have a directory in the root of my site called "Game Music" which has subfolders with MP3s in (there are no webpages inside it). Google is showing non-existant pages from that directory in search results when I search for an album name. For example, try searching for the following on Google:
Baroque (Saturn) Original Soundtrack

On the 1st results page you'll see the real page on my site. The 2nd results page has a link to a non-existant page called:
"Game Music/B/Baroque/001. Baroque (Saturn) Original soundtrack"

The 1st page has download links to MP3s in the following location:
"Game Music/B/Baroque/001. Baroque (Saturn) Original soundtrack"

So is this non-existant page problem got something to do with my .htaccess file being wrong? Here's what I've got:

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)$ index.php?title=$1 [L,QSA]

Options +FollowSymlinks
RewriteEngine on

# Link for the Sitemap
RewriteRule ^sitemap(.*)\.xml$ sitemap.php?page=$1 [L,NC]

RewriteCond %{HTTP_REFERER} !^http://example.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://example.com$ [NC]
RewriteRule .*\.(jpg|jpeg|gif|png|bmp|mp3)$ http://example.com/Hotlink_Protection [R,NC]


[edited by: Holygamer at 10:28 pm (utc) on Mar 20, 2012]

[edited by: incrediBILL at 10:38 pm (utc) on Mar 23, 2012]
[edit reason] site link removed [/edit]

10:23 pm on Mar 20, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


The rules are in the wrong order.

The hotlink rules should be first. Both of the .* patterns in that ruleset are errors in some way or other.

The sitemap rule should be next.

The general rewrite should be last.

You should have RewriteEngine On only ONCE at the start of the file.

Use example.com in your code here to suppress URL auto-linking in this forum.
10:43 pm on Mar 20, 2012 (gmt 0)

New User

5+ Year Member

joined:Mar 19, 2012
posts: 19
votes: 0


I don't understand code, I got all of this from someone else. This is what I've got so far from what you said:

RewriteRule .*\.(jpg|jpeg|gif|png|bmp|mp3)$ http://example.com/Hotlink_Protection [R,NC]

# Link for the Sitemap
RewriteRule ^sitemap(.*)\.xml$ sitemap.php?page=$1 [L,NC]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)$ index.php?title=$1 [L,QSA]

Options +FollowSymlinks
RewriteEngine on

RewriteCond %{HTTP_REFERER} !^http://example.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://example.com$ [NC]


Is the above correct?

How do you know the hotlink protection code is wrong? I used the Hotlink Protection button in cPanel to make it. Any idea what the correct code should be for it?

Which block of text is the general rewrite that you're talking about?

[edited by: incrediBILL at 10:39 pm (utc) on Mar 23, 2012]
[edit reason] site link removed [/edit]

2:09 pm on Mar 22, 2012 (gmt 0)

New User

5+ Year Member

joined:Mar 19, 2012
posts: 19
votes: 0


Can anyone else help me with this? Is the above code correct?
2:20 pm on Mar 22, 2012 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11566
votes: 182


welcome to WebmasterWorld, Holygamer!

the Options and RewriteEngine directives should be at the top and the last 2 RewriteCond directives should go before the RewriteRule to which they apply.
2:56 pm on Mar 22, 2012 (gmt 0)

New User

5+ Year Member

joined:Mar 19, 2012
posts: 19
votes: 0


To save me guessing, could you please just rearrange my currect htaccess file for me? I didn't understand what g1smd said and I don't imagine I'm going to get it right this time either. Here's my currect file:

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)$ index.php?title=$1 [L,QSA]

Options +FollowSymlinks
RewriteEngine on

# Link for the Sitemap
RewriteRule ^sitemap(.*)\.xml$ sitemap.php?page=$1 [L,NC]

RewriteCond %{HTTP_REFERER} !^http://example.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://example.com$ [NC]
RewriteRule .*\.(jpg|jpeg|gif|png|bmp|mp3)$ http://example.com/Hotlink_Protection [R,NC]

Also g1smd said that "The hotlink rules should be first. Both of the .* patterns in that ruleset are errors in some way or other". How do you know they are errors? What should I have instead?

[edited by: incrediBILL at 10:40 pm (utc) on Mar 23, 2012]
[edit reason] site link removed [/edit]

4:07 pm on Mar 22, 2012 (gmt 0)

New User

5+ Year Member

joined:Mar 19, 2012
posts: 19
votes: 0


Is this correct?

RewriteRule .*\.(jpg|jpeg|gif|png|bmp|mp3)$ [example.com...] [R,NC]

Options +FollowSymlinks
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{HTTP_REFERER} !^http://example.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://example.com$ [NC]
RewriteRule ^(.+)$ index.php?title=$1 [L,QSA]

# Link for the Sitemap
RewriteRule ^sitemap(.*)\.xml$ sitemap.php?page=$1 [L,NC]

[edited by: incrediBILL at 10:40 pm (utc) on Mar 23, 2012]

8:25 pm on Mar 22, 2012 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15445
votes: 739


Is this correct?

Noooo.

Sorry. But you've got the "RewriteEngine on" statement coming after your first RewriteRule.

Incidentally: It's your choice, but most people handle hotlinks with a rewrite rather than a redirect. Note too that Redirects (plain [R] flag, or Redirect in mod_alias) default to 302. You almost always want 301 instead.
10:16 pm on Mar 22, 2012 (gmt 0)

New User

5+ Year Member

joined:Mar 19, 2012
posts: 19
votes: 0


g1smd said that the hotlink rule should be first so that's what I did, I put it first. But Lucy you say that's wrong!?

The hotlink code was automatically inserted by cPanel when I actived Hotlink Protection. What difference does it make if I use a 301 or 302 for Hotlink Protection when the Hotlink Protection page that people see when they hotlink is worthless for SEO? What code do I use for a 301?

Is this correct?:

Options +FollowSymlinks
RewriteEngine on
RewriteRule .*\.(jpg|jpeg|gif|png|bmp|mp3)$ [example.com...] [R,NC]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{HTTP_REFERER} !^http://example.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://example.com$ [NC]
RewriteRule ^(.+)$ index.php?title=$1 [L,QSA]

# Link for the Sitemap
RewriteRule ^sitemap(.*)\.xml$ sitemap.php?page=$1 [L,NC]

[edited by: incrediBILL at 10:41 pm (utc) on Mar 23, 2012]
[edit reason] exemplified [/edit]

10:26 pm on Mar 22, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


The hotlinking code is easily circumvented but at least the syntax is now right. The user will also see that they have been redirected to a different URL, and that it is called "hotlink protection". I much prefer a "silent" block, serving some other error message or image without redirecting to a different URL. For this you use a RewriteRule configured as a rewrite and not as a redirect.

Add a blank line after every rule for clarity.

RewriteConds and RewriteRules may appear in groups. The conditions apply only to the rule that immediately follows.

Options +FollowSymlinks
RewriteEngine On

# hotlink code
RewriteCond %{HTTP_REFERER} !^http://(www\.)?example\.com [NC]
RewriteRule \.(jpe?g|gif|png|bmp|mp3)$ http://example.com/Hotlink_Protection [R=302,NC,L]

# add your www to non-www canonical redirect code here
# ->

# sitemap rewrite
RewriteRule ^sitemap([^.]+)\.xml$ /sitemap.php?page=$1 [NC,L]

# catch-all rewrite
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule (.+) /index.php?title=$1 [QSA,L]


Please read as much as you can about mod_rewrite and about regular expressions. This stuff cannot be guessed. A single typo can take your site offline so you need to know enough to maintain your code yourself not rely on suggestions from a forum.

[edited by: incrediBILL at 10:42 pm (utc) on Mar 23, 2012]
[edit reason] site link removed [/edit]

11:55 pm on Mar 22, 2012 (gmt 0)

New User

5+ Year Member

joined:Mar 19, 2012
posts: 19
votes: 0


What was wrong with the hotlinking syntax in the first place? It prevented hotlinking like it was supposed to do. What does your version do that the original version does not?

Lucy suggested a 301 for the Hotlink code but you've chosen 302. What is the difference and why does it matter?

How is the hotlinking code you suggested easily circumvented? Is there a better way of doing it?

What is the difference between my original line and your line?:

RewriteRule ^(.+)$ index.php?title=$1 [L,QSA]
RewriteRule (.+) /index.php?title=$1 [QSA,L]

What is the difference between my sitemap line and your line?:

RewriteRule ^sitemap(.*)\.xml$ sitemap.php?page=$1 [L,NC]
RewriteRule ^sitemap([^.]+)\.xml$ /sitemap.php?page=$1 [NC,L]

I got my line from a sitemap extension for MediaWiki. The extension page instructed me to add that line. What was wrong with the original line?

You mentioned to add my owen www to non-www canonical redirect code. Do I need to do this as it was already redirecting from www.example.com to http://example.com?

I don't have the ability to understand code. I had to pay someone to make my htaccess file but since you guys have told me it's wrong then it seems like he didn't know what he was doing. So if he, who understands code got it wrong, what chance do I have of understanding it!

[edited by: incrediBILL at 10:43 pm (utc) on Mar 23, 2012]
[edit reason] site link removed [/edit]

12:06 am on Mar 23, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


what chance do I have of understanding it

Five years ago this stuff looked like Vogon poetry to me.

After reading a few hundred threads here, several tutorials elsewhere, and playing around with code on several sites I had a reasonable idea of the basics in a few weeks and some of the more advanced stuff in a few months - and I was dedicating hardly any time to it at all.

All of the questions in your previous post have been answered here hundreds of times before. There's very little in the way of new questions these days. :)

The stuff about not using (.*) at the beginning or in the middle of a pattern I end up typing out several times per week, and occasionally several times per day.

Patterns such as (.+) and (.*) are "greedy". They don't need anchors. They keep going until they have captured everything.

Lucy suggested a 301 for the Hotlink code but you've chosen 302.

You used plain "R" in your code. That produces a 302. I just "clarified" that by adding the numbers. In real life I wouldn't redirect, I would rewrite.

MediaWiki

I have modified the MediaWiki PHP code for several sites and the htaccess file for a few more. Point me in the direction of this sitemap extension you use and I'll contribute updated code for it back to the author.
11:39 am on Mar 23, 2012 (gmt 0)

New User

5+ Year Member

joined:Mar 19, 2012
posts: 19
votes: 0


The extension is DynamicWikiSitemap. I appreciate what you're saying about repeating stuff all the time but you told me my code was wrong and you gave me new code but you won't tell me what it does compared to the original code! I need to know what it does.

Most of my questions are asking what is the difference between my code and your code. Those questions haven't been answered before so I can't find them via search.

I searched Webmasterworld for "hotlinking difference between 301 302" but found nothing. I searched for "hotlinking circumvention" but found nothing.

I looked at the library and I can't see anything which explains my questions as everyone has different kinds of problems and different code.

You mentioned to add my own www to non-www canonical redirect code. Do I need to do this as it was already redirecting from
www.example.com to http://example.com?

[edited by: incrediBILL at 10:43 pm (utc) on Mar 23, 2012]
[edit reason] site link removed [/edit]

10:45 pm on Mar 23, 2012 (gmt 0)

System Operator from US 

incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14664
votes: 99


<mod>
Please only use example.com as the domain name, actual domain names are not allowed.
Now keep it civil.
</mod>
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members