Forum Moderators: coopster & phranque

Message Too Old, No Replies

Why do my Perl scripts sometimes just not work?

         

csdude55

8:14 pm on Dec 24, 2022 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



This has been happening for awhile and it's driving me crazy. I upload a script and it gives me a generic fatal error. Sometimes (but not always) I can delete it and reupload it, and it's fine.

I thought it was a hiccup with my internet service, but now I have a new internet provider and it's still happening so I don't know.

Here's this week's problem child. The file is named "copy_users.cgi". I uploaded it as root (so owner is root:root), changed the permission to 0755, then ran it in the browser. I got a generic error that the file failed.

Deleting it and reuploading it didn't help, even when I did it the next day.

I used chown user:user copy_users.cgi to change the owner, but that didn't help.

I copied the entire script to "test.cgi" (in Windows, using Ctrl+A, Ctrl+C, then Ctrl+V, and then uploaded to my CentOS server) that is also root:root and 0755, and there it works fine.

I modified "copy_users.cgi" to this (copied from "vi" so you can see there aren't any hidden characters):

# vi copy_users.cgi
#!/usr/bin/perl

print "Content-type: text/plain\n\n";
print "Done";
exit;

Literally NOTHING here but printing the word "Done", but I get this in the error log:

(2)No such file or directory: AH01241: exec of '/home/user/www/cgi-bin/copy_users.cgi' failed
End of script output before headers: copy_users.cgi

It can't be an issue of a typo or something, since copy-and-pasting the entire script to another script works fine. And since it's happened with other scripts, I don't think that it's an issue of an Apache file not liking the name "copy_users.cgi".

Any other thoughts?

phranque

4:44 am on Dec 25, 2022 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



is this at all related to your post in the Apache forum [webmasterworld.com]?

csdude55

7:22 am on Dec 25, 2022 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Possibly, but I'm not sure. I never did figure out why it didn't work, either. It was originally uploaded last year, and this time when I uploaded a modification and restarted Apache then I still had the same error. Eventually I deleted it and reuploaded the same script, and it worked.

It's possible that there was a hidden character in there, but it's also possible that it had the same problem that I've had with several Perl scripts. Without being able to duplicate it on demand, and without being able to see why the server is making them throw an error, I'm just flying blind.

But since I at least had some sort of an error message here, I was hoping it might be a little more clear :-)

csdude55

9:12 pm on Dec 26, 2022 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Update:

I have "test.cgi" working, and "copy_users.cgi" throws the generic error. They have identical text and permissions. I build in Notepad++ on Windows 11, then upload to my Linux server using FileZilla.

If I rename "test.cgi" to "copy_users.cgi", then "copy_users.cgi" will work.

But here's what's really weird.

Filesize for "test.cgi" is 2,475 bytes.

Filesize for "copy_users.cgi" is 2,547!

I opened "test.cgi" and copy-pasted the exact text to "copy_users.cgi", and it's still 2,547.

I uploaded a blank "test.txt" file and pasted the exact text from "test.cgi", it's 2,547. I did the same thing with "test.bak", and it's also 2,547.

Then I did the same thing with "test.cgi.bak" (a cache file created by Notepad++). This file is 2,475.

Then I did the same thing with "copy_users.cgi.bak", and it's 2,547.

So it appears that the magic trick here is that it has to contain "test.cgi" in the file name.

I tried uploading them to different directories, including to /home (outside of the user account), and had the same filesize discrepancy. So it doesn't appear to be an issue of directory.

I opened scripts containing both filesizes (2,547 and 2,475) using "vi" with ":set list" and see absolutely no difference between them, but there appear to be 72 bytes worth of hidden characters that are probably causing a problem.

If I use vi to create a new "copy_users.cgi" and then paste the contents to it, then it's 2,475 and works as expected.

All of this makes me think that FileZilla is the problem. I've used it for years (maybe decades), and haven't changed any settings that I can remember. The current stable version is 3.62.2 with a build date of 11/22/2022.

I'm trying to register with their forum (although I haven't gotten the verification email yet), and I'll update if I have any suggestions there.

phranque

9:34 pm on Dec 26, 2022 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



have you tried a diff of the two files?

if that is not informative then I would try creating hexadecimal dumps of both files and then comparing those character by character.

csdude55

10:38 pm on Dec 26, 2022 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Both of those are new commands for me :-)

For this, I have "copy_users.cgi" (78 bytes) and "test.cgi.bak" (74 bytes). I copy-and-pasted "copy_users.cgi" to "test.cgi.bak" so they should be identical, but there's still a 4-byte difference. The files are 4 lines (well, 5 if you count a blank line), so one would think a hidden character at the beginning or end of each line:

#!/usr/bin/perl

print "Content-type: text/plain\n\n";
print 'done';
exit;

Here is the result of diff:

# diff copy_users.cgi test.cgi.bak
1,4c1,4
< #!/usr/bin/perl
<
< print "Content-type: text/plain\n\n";
< print 'done';
---
> #!/usr/bin/perl
>
> print "Content-type: text/plain\n\n";
> print 'done';

If I'm understanding correctly, 1,4c1,4 means that every line is different. Which confirms my theory of a hidden character at the beginning of each line, but that's as far as I'm getting.

So then I did hexdump:

# hexdump -C copy_users.cgi
00000000 23 21 2f 75 73 72 2f 62 69 6e 2f 70 65 72 6c 0d |#!/usr/bin/perl.|
00000010 0a 0d 0a 70 72 69 6e 74 20 22 43 6f 6e 74 65 6e |...print "Conten|
00000020 74 2d 74 79 70 65 3a 20 74 65 78 74 2f 70 6c 61 |t-type: text/pla|
00000030 69 6e 5c 6e 5c 6e 22 3b 0d 0a 70 72 69 6e 74 20 |in\n\n";..print |
00000040 27 64 6f 6e 65 27 3b 0d 0a 65 78 69 74 3b |'done';..exit;|
0000004e

# hexdump -C test.cgi.bak
00000000 23 21 2f 75 73 72 2f 62 69 6e 2f 70 65 72 6c 0a |#!/usr/bin/perl.|
00000010 0a 70 72 69 6e 74 20 22 43 6f 6e 74 65 6e 74 2d |.print "Content-|
00000020 74 79 70 65 3a 20 74 65 78 74 2f 70 6c 61 69 6e |type: text/plain|
00000030 5c 6e 5c 6e 22 3b 0a 70 72 69 6e 74 20 27 64 6f |\n\n";.print 'do|
00000040 6e 65 27 3b 0a 65 78 69 74 3b |ne';.exit;|
0000004a


0a / 0d are line feed / carriage return, and that's the only difference I'm seeing; copy_users.cgi (78 bytes) has both, while test.cgi.bak (74 bytes) only has the line feed.

Do you think that's the source of my problem, or just a coincidence that shouldn't cause this generic error?

phranque

11:45 pm on Dec 26, 2022 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



i was wondering if the difference between 2,475 bytes and 2,547 were the CR/LF sequences being reduced to LFs.

what OS is the desktop and the server?

you might want to try playing with the transfer data type in FileZilla [wiki.filezilla-project.org] for the .cgi files.

Brett_Tabke

12:30 am on Dec 27, 2022 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I thought you would have run into that before Dude. Yes, you need to run unix line enders on unix. Windows Perl is not picky about enders and runs either. They will run from the cmd line, but for some reason apache barfs at it.

Other than that, it could be an ownership problem.

same file:
dos line enders: [pubcon.com...]
unix enders: [pubcon.com...]


#!/usr/bin/perl

push @INC,'.';
use CGI::Carp qw (fatalsToBrowser);


print qq|Content-type: text/html\n\n|;
print "hi";
exit;
1;


Screen shot showing both running from cmd line:
[pubcon.com...]

csdude55

6:05 am on Dec 27, 2022 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



what OS is the desktop and the server?

@phranque, the desktop is running Windows 11 and I'm building in Notepad++, but for the most part I've been editing in Notepad during these tests.

I thought that the server was CentOS, but WHM/cPanel says that it's running Linux 3.10.0.

you might want to try playing with the transfer data type in FileZilla [wiki.filezilla-project.org] for the .cgi files

The only options it gives is Auto, ASCII, and Binary. I have "cgi" set to upload as ASCII, but for the sake of testing I changed it to force all uploads to ASCII. That didn't help, though :-/

I thought you would have run into that before Dude. Yes, you need to run unix line enders on unix. Windows Perl is not picky about enders and runs either. They will run from the cmd line, but for some reason apache barfs at it.

@Brett_Tabke, I feel like I dealt with something similar about 25 years ago :-/ I've always built in Windows and uploaded to Linux or CentOS, though, so somewhere along the way my software was taking care of it for me.

I haven't changed anything in my software lately, though, so I'm not sure why it's a new issue. The server is 2 or 3 years old, and my computer is 1 or 2 years old. Unless an update somewhere along the way caused a problem.

What's weird is that I have another Perl script that I uploaded on 11/22/2022, and it works fine! And, of course, uploading test.cgi seems to work fine. But "copy_users.cgi", "contact.cgi", and "gas.cgi" are throwing this same error.

I changed "copy_users.cgi" to "copy_users.pl" and had the same issue, so the issue isn't restricted to .cgi extensions.

You're right that it works via SSH, though, so the problem is definitely related to Apache:

# perl copy_users.cgi
Content-type: text/plain

done

As for Apache, I'm running:

Server Version: Apache/2.4.54 (cPanel) OpenSSL/1.1.1s mod_bwlimited/1.4 mod_fcgid/2.3.9
Server MPM: event
Server Built: Dec 21 2022 18:16:54

I do have a lot of custom rules set up in the Apache CONF, but they're all restricted to a subdomain. I copied "copy_users.cgi" to the main domain and it had the same error, so it's definitely not related to those CONF files. Besides, the fact that I can upload it to test.cgi and then rename it to copy_users.cgi to make it work makes me think it HAS to be related to the upload, not the file name.

csdude55

6:17 am on Dec 27, 2022 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



After posting that, I decided to download a different FTP client and see if that changed anything. So I went to Google, and while FileZilla was the first result, WinSCP was the second.

And guess what?

I used it to modify copy_users.cgi, I simply added a line break to the end and then removed it and let it re-upload. Now the file size is 74 bytes, and it works with no error!

I'll have to test more tomorrow, but as of 1:15am it looks like the problem is with FileZilla.

Brett_Tabke

11:48 am on Dec 27, 2022 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



What text editor are you using? You should be able to specify Unix enders for everything right in your editor and not have to mess with the ftp conversion. Again, Windows Perl and Windows Apache will not care about the enders, but Unix Apache will.

I think Unix Apache gets confused on the 'shebang' line if the enders are not in Unix format. Apache never actually starts Perl after the call to the windows formatted Perl file.

Meanwhile, Windows Apache does not actually use the shebang line and sends it straight to Perl based on the mime/file type (eg: .cgi or .pl).

Running a Windows formatted file from the Unix cmd line works, because you are calling Perl directly and not using the shebang line and Perl deals with it seamlessly.

[edited by: Brett_Tabke at 7:34 pm (utc) on Dec 27, 2022]

csdude55

6:58 pm on Dec 27, 2022 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



What text editor are you using? You should be able to specify Unix enders for everything right in your editor and not have to mess with the ftp conversion.

Notepad++, and good news! That worked :-)

Under Settings > Preferences > New Document, there's an option to change "Format (Line Ending)" to "Unix (LF)". That only applies to new documents, though, so opening and saving an existing file didn't fix anything. I had to create a new document, copy and paste the contents to it, then save it to overwrite the original. At that point the file size was smaller, and uploading it worked.

I can't find a way to make it modify existing files, so that kinda sucks. The only I found was to replace them by going to "Find in Files", then replace \r\n with \n in *.cgi and *.lib. As far as I can tell, that worked.

I've been using Notepad++ for years (decades, maybe), so when/why that changed? I have no idea. The earliest file that was modified when I replaced \r\n was dated 6/26/2019, though, so I'm guessing it dates back to at least that!

Of course, it could have been a change in FileZilla that stopped it from changing \r\n to \n, too.

WinSCP also converted \r\n to \n, though, so right now it looks like I need to open every Perl script one at a time in WinSCP, make a slight modification then undo it (like, hit Space then Ctrl+Z), then save it and re-upload it. Once it's uploaded, I'll need to download it to overwrite my local copy. Then I can go back to FileZilla if I want.

Brett_Tabke

7:36 pm on Dec 27, 2022 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



[stackoverflow.com...]

From the "Edit" menu, select "EOL Conversion" -> "UNIX/OSX Format".

csdude55

8:30 pm on Dec 27, 2022 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thanks! I tried it and saw that "Unix" was disabled, but I'm pretty sure it's because I've already done a global replace of \r\n to \n.

explorador

3:42 am on May 8, 2023 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I feel your pain, I've been there. Besides what's said here regarding the invisibles and returns, sometimes new servers add their own "cache" that you can't modify or have access to it, so you get an error and while the script has been fixed, the error remains and you have to wait for it to expire. Severs sometimes are (today) layer on top of another layer of "things".