Forum Moderators: open

Message Too Old, No Replies

cloaking and redirection

         

notbob

5:20 am on Aug 16, 2003 (gmt 0)

10+ Year Member



I would like to cloak one of my domains for the search bots but all users that go to this cloaked page would end up on another one of my domains (the primary domain).

The reason for this is that I don't want to cloak my primary domain for the risk of it being banned in the search engines.

Would this model of cloaking and redirection work?

msr986

6:58 am on Aug 16, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you are discovered, what makes you sure that both domains won't be banned?

notbob

9:00 am on Aug 16, 2003 (gmt 0)

10+ Year Member



That's a good point, I don't know if they would ban both domains or not.

It's what I read elsewhere, never cloak your primary domain if you fear getting banned, so I don't see how to get the traffic to the primary domain without redirection.

Links would work, but redirection would ensure 100% of SE traffic would go to the primary domain.

volatilegx

3:57 pm on Aug 18, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



notbob,

There are ways to "redirect" without "redirecting". What I mean, is that you can have a server-side script make a HTTP request to your primary domain, and serve up the results under the original cloaked URL. This is transparent to the end user. It isn't technically a "redirect", and no redirect header is sent.

Dan

notbob

12:49 am on Aug 19, 2003 (gmt 0)

10+ Year Member



>There are ways to "redirect" without "redirecting". What >I mean, is that you can have a server-side script make a >HTTP request to your primary domain, and serve up the >results under the original cloaked URL. This is >transparent to the end user. It isn't technically >a "redirect", and no redirect header is sent.

That sounds really impressive, can you send me via sticky mail a name of a script that will create these "redirections"

volatilegx

9:06 pm on Aug 20, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Better than stick mail, here's a simple perl script:

#!/usr/bin/perl -w
$url="http://www.somedomain.com/index.html";
use LWP::Simple;
$HTML=get $url;
print $HTML;
exit;

claus

9:29 pm on Aug 20, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>> there are ways

Post #6 in this thread might also be interesting, don't want to repost it: [webmasterworld.com...]

/claus

dsmmg

3:08 pm on Aug 21, 2003 (gmt 0)

10+ Year Member



Better than stick mail, here's a simple perl script:
#!/usr/bin/perl -w
$url="http://www.somedomain.com/index.html";
use LWP::Simple;
$HTML=get $url;
print $HTML;
exit;

Alright, sorry to intrude into this conversation here, but I just had a quick question relating to this...

I created a file in my cgi bin entitled redirect.pl and threw this into it.

I then created a shtml doc with nothing in between the body tags but <!--#include virtual="cgi-bin/redirect.pl" -->

All I'm getting is [an error occurred while processing this directive]!

What am i doing wrong? I apoligize in advance if it's blindingly obvious to all others... I'm kinda new at this!

Thanks!

claus

3:13 pm on Aug 21, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



<!--#include virtual="cgi-bin/redirect.pl" -->

Try this, it sometimes does the trick

<!--#exec cgi="cgi-bin/redirect.pl" -->

Btw. you could also just call the script "index.pl" or "index.cgi" if your server supports such file types as index files.

>> nothing in between the body tags but

You'll get double body tags then, as the destination doc also has them i suppose.

/claus

dsmmg

4:06 pm on Aug 21, 2003 (gmt 0)

10+ Year Member



Thanks for the responce claus... much appricated!

I am still having trouble though! I tried what you suggested with the execute vs the include, that seems to stop me from getting the error message. I then took your advice and changed it into a .cgi file and named it index, which my host does infact allow... and got a 500 / Internal Server Error message!

Which led me to believe that there is a problem with my implimentation of the original code. All I did was copy and paste it into a text file (now) named index.cgi, of course changing the "somedomain" to my domain, and uploaded it. The path to perl is correct and all of that. Not sure if this matters, but I also chmodded it to both 755 and 777 and tested both. No go! Any idea what i mistyped or anything?

Thanks, DS

dsmmg

6:46 pm on Aug 21, 2003 (gmt 0)

10+ Year Member



more info to work with here... When the CHMOD is set at 777, I get an error message in my logs that says it is writable by others and when it is set set at anything other than 755, I get one that says it is not executable. Apperantly, while I am still getting 500 server errors, if it is chomded at 755, however, it doesn't record an error in my error logs. Strange...

dsmmg

7:00 pm on Aug 21, 2003 (gmt 0)

10+ Year Member



Apperently, my host requires that permissions should be set at 755 for pl and cgi files. That would explain the previous post... But not why I'm still having all this trouble!

DaveAtIFG

7:48 pm on Aug 21, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This line

#!/usr/bin/perl -w
is often called the "shebang" line and points to the directory that contains Perl. It may need to be revised to be compatible with your server. Not sure what the -w flag specifies...

Telnet to the server and, at the command prompt, type "whereis perl" to determine the correct path to Perl on your server.

dsmmg

8:09 pm on Aug 21, 2003 (gmt 0)

10+ Year Member



Thanks for the reply, DaveAtIFG...

I wish it were that easy though! That is infact the path to perl at my host, or at least they tell me it is. I can use either #!/usr/bin/perl or #!/usr/local/bin/perl. So that's not my problem, per se...

I also verified that I have support for LWP::Simple, which I do, and tried to track down what the -w stands for. As far as I can tell it is a diagnostic tool that produces some sort of results somewhere, although I havn't been able to find them (Like I said, I'm kinda new at this advanced sorta stuff!). Probably real simple, but dumb old me can't find them!

Thanks agian though!

DS

claus

10:18 pm on Aug 21, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



You might want to include this line before you print out the HTML string:

print "Content-type: text/html\n\n";

- it's a pure guess, but it's usually needed when writing to a web page. Also, you might want to put this just under the shebang until you get it to work, it will (try to) tell you where the error is:

use CGI::Carp qw(carpout fatalsToBrowser);

But be sure to remove it again when it all works.

- so try this:

-----------------------

#!/usr/bin/perl

use CGI::Carp qw(carpout fatalsToBrowser);
use LWP::Simple;
$url="http://www.somedomain.com/index.html";
$HTML=get $url;
print "Content-type: text/html\n\n";
print $HTML;
exit;

-----------------------

/claus

dsmmg

2:46 am on Aug 22, 2003 (gmt 0)

10+ Year Member



Claus, you solved it exactly 19 minutes after my cable connection went down, and a good 3 1/2 hours before the guy at my hosting support forum! Of course, I didn't get to try either of them until 10:30, but that's alright as it now works 100%!

Your 'Just a Guess' of adding the

print "Content-type: text/html\n\n";

was dead on! After doing so, it worked perfectly!

Thanks for all your help... Much appreciated!

DS

claus

7:26 am on Aug 22, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>> exactly 19 minutes after my cable connection went down

Sorry about that, i was in a hurry to get out of the door when you posted, so i did not post until i came back :)

/claus

jamesa

12:31 pm on Aug 22, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I don't know if you cloakers use PHP for anything, but just for giggles here's how to do the same thing with PHP:

<?
include("http://www.somedomain.com/index.html");
?>

dsmmg

1:37 pm on Aug 22, 2003 (gmt 0)

10+ Year Member



Tried the php approach to see if it was as seamless as the perl one... Just created a php doc with nothing but the include in it, threw it up on the server and tried it out. It actually worked ;), but when looking at the code I got this message:

<br />
<b>Warning</b>: main(): stream does not support seeking in <b>/home/sitename/public_html/testing/something.php</b> on line <b>2</b><br />

Before anything else on the page... Do I need to add something else?

Claus... Wouldn't really have made a difference at all as that the cable died as I was coming back here to check out the thread! Basicly, I wouldn't even have known about it even if you had posted it 30 seconds after my last post... Irregardless, I was estatic to finally have it working and honestly really appreciate your help!

Thanks agian, DS

volatilegx

2:02 pm on Aug 22, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



oops claus is right, i forgot my Content-type header!

I'm so embarrassed!

Dan

volatilegx

2:04 pm on Aug 22, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



OK guys, doing a "GET" call to a url is easy... just for giggles, who knows how to do a POST call, with form data? And process the reply?

hehe

I'll post code if anybody wants me to.

dsmmg

3:04 pm on Aug 22, 2003 (gmt 0)

10+ Year Member



hehehehehehe... I would love to see it! Learning all sorts of new and exciting things day by day!

claus

10:47 am on Aug 23, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>> main(): stream does not support seeking in

I've seen (on a PHP forum) that the cause of this error is that you use it with "http:". The solution, apparently, is to include a small at-sign first, like this:

<? 
@include("http://www.example.com/index.html");
?>

I don't really use php myself, so i'm not able to confirm this or explain why.

I suppose that the "main stream" is what gets sent to the browser form the server and that php is not able to perform operations (apart from "pass on/display") on content hosted elsewhere unless it's imported in another way. If the @ sign is an array, then it makes some sense somehow, but as i said i really don't know, i guess that it's not but rather a prefix for "external" or something.

/claus

added: volatilegx, i know it, i do this myself all the time, just write some code straight from memory or outside context. Your code did do the job asked for - the header was just a detail. It might as well have been that some module wasn't available or whatever. There's just so many things you can do that it happens all the time (to me at least, it does : )

jamesa

8:16 pm on Aug 23, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hmmm, don't know why the warning but what claus suggested is fine. The @ sign in this context just suppresses warning messages.

May also want to try

readfile()
which also accepts URLs the same way
include()
does (inside the parentheses).

volatilegx

4:07 pm on Aug 25, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Sending a POST request using Perl (adapted from the LWP Cookbook):

use HTTP::Request::Common qw(POST);
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$uatopasson = $ENV{"HTTP_USER_AGENT"}; # or make one up
$referertopasson = $ENV{"HTTP_REFERER"}; # or make one up
$ua->agent($uatopasson);
$req->header('referer' => $referertopasson);
my $req = POST 'http://www.somedomain.com/cgi-bin/cgiscript.cgi',
[ field1 => 'value1',
field2 => 'value2' ];
$response = $ua->request($req)->as_string; # process $response however you like...

The best part about the snippet above is that you don't have to URL-encode the field name/value pair strings.

[edited by: volatilegx at 4:15 pm (utc) on Aug. 25, 2003]

volatilegx

4:12 pm on Aug 25, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



claus,

Unfortunately, with programming, "it's all in the details".

I hate it when a script I'm working on errors out because I forgot the header. :)