htaccess not working after changing servers

Moving from one host to another

[grandma genie]

Hello,
I just moved my website to a new host, InMotion. The htaccess file I was using with the old host was moved also. However, now the allow deny directives are not working. Other parts of the file still work. It is just blocking IPs that is not. What is happening is instead of getting the normal 403 forbidden they are getting a 302 redirect. Why would the file work on the old servers but not work on the new ones? I have contacted the host. This was their suggestion:

<Files 403.shtml>
order allow,deny
allow from all
</Files>

deny from 1.202.nnn.nn

I do not have a custom 403.shtml file, so this doesn't make sense.

I'm assuming if the file worked on one server is should work on another one and the InMotion server is set up differently. Any ideas? I need to be able to ban certain IPs.

My current htaccess file begins with this:

order allow,deny
deny from 1.202.nnn.nn
allow from all

I have many more IPs in the list, just showing one.

I know my developer when moving the site set up a 302 redirect for anyone using old links, but that shouldn't be causing my banned IPs to be redirected, should it? I haven't heard back from her yet.

The 302 redirect is in essence a block, but not one I want. These rogues need to know they are not welcome.

-- grandma

[incrediBILL]

Based on their suggestion, I'd change hosts ASAP.

[lucy24]

I know my developer when moving the site set up a 302 redirect for anyone using old links, but that shouldn't be causing my banned IPs to be redirected, should it?

Unfortunately yes, it will. Redirects happen right away-- probably in mod_alias or mod_rewrite-- before the request ever reaches the core "Deny from..." directives. So you have to ban them all over again after they arrive.

This is assuming you also changed domain names. If all you've done is change physical servers, then nothing at all should be happening to incoming requests. The DNS should just point them to the new place.

Oh, and fire your developer ;) A 302 is almost never appropriate. Certainly not when you've permanently moved your whole site!

[grandma genie]

The 302 redirects were used when I moved my site to a new host 6 months ago. At that time all my directory names changed (site redesigned) and it was easier to just send everyone back to the home page when they were arriving from one of the old links. But it's been 6 months since that change, so I asked my developer to remove the redirect. So, hopefully that will be taken care of. But this recent change (a few days ago) was just a host change. The domain name is the same as before. So, that is why it doesn't make sense that the same htaccess file shouldn't be acting the same way.

I've been with 5 hosting companies in less than 5 years. All bad.

As soon as I get my developer to get rid of the redirect, I'll see if that fixes the htaccess file. If not, I'll get back with you all.

If you know of a developer who charges $25 an hour and understands php and oscommerce, let me know.

[g1smd]

At that time all my directory names changed (site redesigned) and it was easier to just send everyone back to the home page (with a 302 redirect) when they were arriving from one of the old links.

That suicidal strategy has been discussed here over and over again for several years. It is entirely inappropriate to use a 302 redirect and certainly not to mass redirect to the root. It's a common mistake that I see frequently. Is there a popular online tutorial that recommends this? I need to go have a serious word.

Removing the redirects will kill your direct traffic from all your old links to the old site. You'll also lose the linking benefit. You should have a series of old page to new page 301 redirects.

When you redirect the old site to new, all requests are redirected unless you use RewriteRules to block the requests that should not be redirected. Those rules should be listed before the rules that perform the redirect.

If you treble your budget you might start to find people with a clue. Anyone good will be charging quite a bit more than that.

[Tim_Sisson]

Hi grandma_genie,

This is Tim S from InMotion Hosting. I'm sorry you're having issues with your account's .htaccess files. Let's see if I can help you.

302 re-directs are temporary and should not be used when moving a site. Depending on how your website was built, you may not need them. In any case, since the move is permanent you'll want to use 301's for any re-direction of old URLs.

As for blocking IP addresses, in your cPanel with InMotion you can easily re-create those bans. This is a simple tool that will write an .htaccess rule for blocking specific IP's or a range of IP's if you use wildcards.

Here's a link to using the IP Deny Manager tool in cPanel:

[inmotionhosting.com ]

Keep in mind, each web server (from different companies) are configured differently and therefore sometimes .htaccess rules function differently.

I'd like to help you further, and you can post a question on our support center [inmotionhosting.com] if you'd like. This will allow me to access your account and help you in more detail.

Thanks!

Tim S

[grandma genie]

I actually do have 301 redirects in the htaccess file, so that old links are finding their way to the correct areas in the new site. Now I just check the server logs to find any other links I'd like to add.

But its the strangeness of the allow/deny directives not working that is bugging me. Why would that htaccess file work on the old server and not on the new?

Here, I'll show you the log entries from before and after:

1.202.nnn.n - - [09/Mar/2012:08:43:17 -0500] "GET /robots.txt HTTP/1.0" 403 - "-" "\"Mozilla/5.0"

1.202.nnn.n - - [02/Apr/2012:07:54:04 -0400] "GET /robots.txt HTTP/1.0" 302 - "-" "\"Mozilla/5.0"

The site is set up the same way as with the old server, so the redirecting should not be messing up my htaccess file. It must be the way the server is set up. What about that, Tim?

grandma

[grandma genie]

I know about the IP blocking area in cPanel and if I open my htaccess file there it shows all the IPs I have blocked. But when those IPs access the server, they are getting 302s instead of 403s, like the sample shown above. I know my htaccess file is set up correctly. Why aren't the inmotionhosting servers serving the correct responses? By the way, I did post in the inmotion support area. Now I just need an answer that works.

[grandma genie]

Can I get an idea about making my 301 redirects a bit more concise? This is the way they are currently looking:

redirect 301 /old_directory/old_folder/old_page.html [mywebsite.com...]

Can I change it to this:

RewriteRule ^old_directory/(.*)$ [mywebsite.com...] [R=301,L]

I'm trying to just redirect visitors who are looking for the old directories to the new ones without having to include every page they want that reside in those directories. Once they get to the new folder in the new directory, they will be able to easily see the page they want.

[g1smd]

Yes, use RewriteRule for all of your rules.

Do not use Redirect or RedirectMatch at all.

[lucy24]

Once they get to the new folder in the new directory, they will be able to easily see the page they want.

If the pages themselves have the same names, your humans won't need to "see" at all. You can redirect them straight to the new location by capturing the last part of the URL and reattaching it.

You can use mod_alias (Redirect by that name) if-and-only-if you use it for all your Redirects. In fact it's very useful in this situation, where you want to change the beginning of the path and let the end follow naturally. But if you have even one Redirect that requires a Condition, then you're in mod_rewrite territory... and then you need to change everything to mod_rewrite* to avoid unwanted consequences.

Now backtracking...
I composed at least three answers to an earlier post but threw them out because I could never pinpoint what was nagging at me. It's this:

... log entries from before and after:

1.202.nnn.n - - [09/Mar/2012:08:43:17 -0500] "GET /robots.txt HTTP/1.0" 403 - "-" "\"Mozilla/5.0"

1.202.nnn.n - - [02/Apr/2012:07:54:04 -0400] "GET /robots.txt HTTP/1.0" 302 - "-" "\"Mozilla/5.0"

... and then what? Ordinarily if a robot gets redirected, it will say "been there, done that" and go somewhere else, or it puts the new URL on its to-do list for later. But when there's a request for robots.txt, your robot never says "### that, I'm outta here." It follows the redirect. So the 302 line will always be followed in logs by a second line, normally saying either 200 or 403.

The second and more serious question is why requests for robots.txt are getting redirected at all. I suspect one of your Redirects isn't carefully constrained, so it's picking up robots.txt when it should be ignoring it. Everything else on the site may have been rearranged, but I doubt you moved robots.txt :)


* My htaccess file contains the following commented-out lines, tucked away here so I don't have to figure them out all over again if I ever need them elsewhere (my text editor uses the \1 notation for all captures):

# meta-RegEx:
# change . to \.
#   ^(Redirect \d\d\d \S+?[^\\])\. TO \1\\.
# now change Redirect to Rewrite
#   ^Redirect(?:Match)? 301 /(.+) TO RewriteRule \1 [R=301,L]
# and
#   ^Redirect(?:Match)? 410 /(.+) TO RewriteRule \1 - [G,L]

[grandma genie]

What I wanted to do was have the redirect send visitors from the directory of the old link to the new directory, without having to list every single page. The current redirects work fine, but I have to go from the old page to the new one. I was trying to keep my list short. I thought if I just went from the old directory to the new one, that would work, but it doesn't. I just tried it. It's just redirecting to the home page and not to the new directory. Was my syntax wrong? I have about twenty different directories to deal with.

And as for the 302s on the robots.txt requests, that is the whole point. They should be getting the 403s. Yes, they always came in twos. Here's an example from the old logs:

1.202.nnn.n - - [12/Mar/2012:08:42:08 -0400] "GET /robots.txt HTTP/1.0" 403 - "-" "\"Mozilla/5.0"
1.202.nnn.n - - [12/Mar/2012:08:42:08 -0400] "GET /robots.txt HTTP/1.0" 403 - "-" "\"Mozilla/5.0"

Now they are coming in as 302s. Nothing has changed except the new server. I have that IP blocked in htaccess, so it should be getting a 403, but it's not, it's getting the 302. And all the blocked IPs are getting 302s. That's my point. Instead of getting the "go away" message, they're getting a " hi, how are you, stay and visit." Ugh!

It has to be something about the new server.

Might as well mention that my referrer blocks and my user_agent blocks are being served 500s. They should be getting 403s, too. I'm convinced it has something to do with the way the server is set up. Is there some sneaky workaround that I can use?

[lucy24]

What I wanted to do was have the redirect send visitors from the directory of the old link to the new directory, without having to list every single page.

Once you get down to pages and/or subdirectories, do your files have the same names and (relative) locations?

For example

old:
www.example.com/oldhome/foobar/filename.html
www.example.com/oldhome/foobar/otherfile.html
www.example.com/oldhome/foobar/nextfile.html

new:
www.example.com/newhome/filename.html
www.example.com/newhome/otherfile.html
www.example.com/newhome/nextfile.html

... and so on for the entire content of /oldhome/foobar/

old:
www.example.com/sillyname/filename.html
www.example.com/sillyname/otherfile.html
www.example.com/sillyname/nextfile.html

new:
www.example.com/goodname/filename.html
www.example.com/goodname/otherfile.html
www.example.com/goodname/nextfile.html

... and so on for the entire content of /oldhome/foobar/

If so, redirects with mod_rewrite can work just like redirects with mod_alias (the Redirect 301 version). You just have to capture and reuse:

RewriteRule ^sillyname/(.*) http://www.example.com/goodname/$1 [R=301,L]

RewriteRule ^oldhome/foobar/(.*) http://www.example.com/newhome/$1 [R=301,L]

If the pages are different, but the directories as a whole are equivalent, then it's

RewriteRule ^sillyname/ http://www.example.com/goodname/ [R=301,L]

RewriteRule ^oldhome/foobar/ http://www.example.com/newhome/ [R=301,L]

Here you leave out the capture-and-reuse because a request for anything in the old directory gets pointed to the index of a new directory.

But it's definitely better if you can redirect to the same page. 20 directories is a manageable number. Depending on their exact names and configurations, you may even be able to consolidate some rules:

RewriteRule ^oldplace/(foobar|gizmo|thingummy)/(.*) http://www.example.com/newplace/$1/$2 [R=301,L]

[grandma genie]

The interesting thing is my incorrect set up actually works:

redirect 301 /deserts/lizards.html [mywebsite.com...]

but this does not:
RewriteRule ^deserts/(.*)$ [mywebsite.com...] [R=301,L]

I'll try putzing around with it to see what works.

Thanks for the examples.

[lucy24]

Waitwait. Old filename lizards, new filename lizards129? Then it's no use capturing, because you're not reusing anything. If the entire name is different, then you have to redirect individually. Or ::cough-cough:: rewrite-- not redirect-- everything to a php script that looks things up in a table (either a rewrite_map or something of its own) and then issues the 301.

Capturing and reusing only works if you're recycling the same thing.

[g1smd]

mod_rewrite has no way to know that lizards129 is the new name for the lizards page so you will either have to tell it in the rule or you can just redirect to the base folder and let the vistor figure it out.

Use RewriteRule for all of your rules.

[grandma genie]

OK, thank you all for the redirect suggestions. As for the malfunctioning htaccess file, I think I figured out why it is not working and it has something to do with some php error pages. So I think I will be going to the php forum and asking them about that. Basically there is an index.php file in the public folder that is redirecting to another one in a different directory. There are also two custom php error pages with redirects. All that redirecting is confusing Apache and that's why I'm getting the wrong responses. As soon as I can set up those php files correctly, it should settle everything down.

[grandma genie]

As a follow up to this situation of a non working htaccess file, I discovered that when I removed just the 403.php file and the reference to that file in htaccess, the allow/deny directives for IPs now works. That error document had a 301 redirect in it to the home page. With it gone, no redirecting and just the server saying Forbidden, which is what I wanted.

[grandma genie]

This error message seems to be offering a clue to this whole mess:

Not Found
The requested URL /store. was not found on this server.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

When I remove the custom 404.php error page I get that error message via the browser. Notice the dot after the store folder. There is no folder named store with a dot. It is supposed to be store/index.php

So, is there a workaround in htaccess?

#301 redirects
RewriteRule ^store.$ [mywebsite.com...] [R=301,NC,L]

[lucy24]

You could redirect-- although not, ahem, using a dot . alone-- but you really do need to find out why Apache thinks it's supposed to look for a nonexistent file in the first place.

/store.

like that is the complete URL (minus domain name, duh).

You said the real URL is

/store/index.php

That is the same as

/store/

Which one is named in your code? It looks as if somewhere along the line, /store/ is getting rewritten or mistyped as /store. When this happens, Apache is no longer looking for a directory-with-index but a literal filename. Which, needless to say, doesn't exist.

This is actually pretty funny when you're talking about a 404 page. Funnier at least than when I got into an analogous mess involving a 403 that users were blocked from seeing-- because they were blocked, right?-- leading to a further request for the 403 page, et cetera.

But it will be a lot funnier once you've pinpointed the problem and got it fixed :)

[grandma genie]

Yes, Lucy, the code mainly uses the /store/ and there must be a place SOMEWHERE that is using the /store. but I haven't found it yet. At least for now the site works because of the 404.php error page that is redirecting to the correct file index.php. It is just causing the htaccess file to work wrong. I'm gonna try to see if I can figure out how to search the site files for that term. Wish me luck or send up some prayers.

[grandma genie]

Well, I was able to get the site to work with some code in htaccess without the 404.php file. That at least solves getting rid of the custom 404.php file that was causing the redirects. But I still don't know why the server wants to add a dot to my URL. It appears to be a server issue. My site always appears as www.mywebsite.com/store/ when accessed, so perhaps the server thinks that is the correct name and wants to add a dot to the end. I don't know, but at least the site is working and the htaccess file is delivering the correct messages. Here is the code added to the htaccess file:

#Forward requests with trailing dot
RewriteRule ^(.+)\.$ [mywebsite.com...] [R=301,L]

[grandma genie]

OK you guys. You can all beat me up. It was all my fault. What a dummy I am. I checked the site from my little laptop pc and there was no dot error. So I checked my Safari browser settings on my Mac and found the default opening page was set to www.mywebsite.com/store. I just changed it and everything is fine. What a dummy! I took the redirect out of htaccess and the site still works just fine. I can't stand it. I am just too dumb.

[g1smd]

Most errors are simple ones. They're just difficult to find.

One thing with redirects; never redirect to a named index file.
Don't redirect to example.com/folder/index.php
Instead redirect to example.com/folder/

[grandma genie]

So, if I want to redirect traffic from an old directory to the new one, I don't want to include the html page?

This is what I have been using:

RewriteRule ^olddirectory/page.html$ [mywebsite.com...] [R=301,NC,L]

So, I should be doing this?:

RewriteRule ^olddirectory [mywebsite.com...]

[lucy24]

No, he means specifically the /index.php (or .html or whatever) page. So it comes through as

RewriteRule blahblah/index\.php http://www.example.com/otherblahblah/ [R=301,L]

or, for existing sites without change

RewriteRule (([^/]+/)*)index\.php http://www.example.com/$1 [R=301,L]

This would probably go right before the final with/without www redirect, as it's the second-least specific rule.

And, of course, make sure all internal links on the new site point to /directory/ rather than /directory/index.php

[g1smd]

Add a RewriteCond looking at THE_REQUEST

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index.php 
RewriteRule (([^/]+/)*)index\.php http://www.example.com/$1 [R=301,L]

to prevent an infinite loop when the internal pointer points to index.php as a result of processing the DirectoryIndex directive.

[grandma genie]

OK, I removed the reference to the index.php in the redirect from one index.php to the other. It is working.

What if I want to redirect anyone who is looking for an old directory that no longer exists to just the home page. The home page has information that would lead any visitor to the page they are looking for by topic. But I am getting lots of hits from bots like Google and Yahoo and regular visitors who are looking for those old directories. They are now getting 404s. Is that OK? Should I just let them get the 404s? Or should I redirect them with a 301 to the home page? Can I use the RewriteCond looking at THE_REQUEST to do that?

But these old directories all have lots of sub folders with html files in them. I just want to redirect if someone is looking for an old directory with ANY accompanying folder, like this:

GET /oldstore/file1.html HTTP/1.1" 404
GET /oldstore/file2.html HTTP/1.1" 404
GET /oldstore/file3.html HTTP/1.1" 404

If someone is looking for /oldstore/ can I do a 301 redirect to www.example.com without having to do one for each html file?

I got the feeling from previous posts that I could not, but I thought htaccess could do just about anything if you just tell it to in the right way. Can you do it and are there a number of different methods that can be used?

By the way, g1msd, your stickymail box is full.

[grandma genie]

g1smd, I'm also dyslexic.

[lucy24]

By the way, g1msd, your stickymail box is full.

It's always full. Pity we can't swipe an idea from php/bb forums and simply make the oldest post fall off the bottom so your inbox is never full, even when it's at 100% :)

If someone is looking for /oldstore/ can I do a 301 redirect to www.example.com without having to do one for each html file?

If there's a finite number of named directories, you can list them. You don't have to list the individual files, just the directories they live(d) in.

:: mid-paragraph pause here to fish out two almost-newborn mollies and relocate them to Baby Tank ::

It would go something like

RewriteRule (olddir1|olddir2|olddir3)/ http://www.example.com/ [R=301,L]

naming all the old directories, separated by | pipes. The parentheses here are just a container; you're not capturing anything. If all the directories are at the top level, include an opening ^ anchor to speed things up a bit.

That's assuming you have soberly evaluated the options and decided a redirect to the front page is your best approach. Another possibility is to change the target to

{blahblah} - [G]

and then make a nice custom 410 page with links to all the likely places.