Kirsle.net
This is a re-do of my previous blog post about Perl upload progress bars - my previous approach was completely wrong. By the time $q->upload(); is used, the file has already been received and stored in a temporary location, and so the "progress bar" in this case is really just gauging how fast the server can copy the file from one place to another on its hard drive.
So this post is how to really do a real working file uploader progress bar in Perl.
The basic steps required to do this include:
$q->upload(); and everything like before.The source code needed for this is still amazingly short and concise, compared to the source codes you'll get when you download solutions from elsewhere.
Implementing this doesn't require any special Apache handlers or mod_perl or anything fancy like that.
Sources:
upload.html<!DOCTYPE html>
<html>
<head>
<title>Upload Test</title>
<style type="text/css">
body {
background-color: #FFFFFF;
font-family: Verdana,Arial,sans-serif;
font-size: small;
color: #000000
}
#trough {
border: 1px solid #000000;
height: 16px;
display: block;
background-color: #DDDDDD
}
#bar {
background-color: #0000FF;
background-image: url("blue-clearlooks.png");
border-right: 1px solid #000000;
height: 16px
}
</style>
</head>
<body>
<h1>File Upload Test</h1>
<div id="progress-div" style="display: none; width: 400px; margin: auto">
<fieldset>
<legend>Upload Progress</legend>
<div id="trough">
<div id="bar" style="width: 0%"></div>
</div>
Received <span id="received">0</span>/<span id="total">0</span> (<span id="percent">0</span>%)
</fieldset>
</div>
<div id="upload-form" style="display: block; width: 600px; margin: auto">
<fieldset>
<legend>Upload a File</legend>
<form name="upload" method="post" action="upload.cgi" enctype="multipart/form-data" onSubmit="return startUpload()" id="theform">
<input type="hidden" name="do" value="upload">
<table border="0" cellspacing="0" cellpadding="2">
<tr>
<td align="left" valign="middle">
Session ID<span style="color: #FF0000">*</span>:
</td>
<td align="left" valign="middle">
<input type="text" size="40" name="sessid" id="sessid" readonly="readonly">
</td>
</tr>
<tr>
<td align="left" valign="middle">
File:
</td>
<td align="left" valign="middle">
<input type="file" name="incoming" size="40">
</td>
</tr>
</table><p>
<input type="submit" value="Upload It!"><p>
<small>
<span style="color: #FF0000">*</span> Randomly generated by JavaScript. In practice this would be
randomly generated by server-side script and "hard-coded" into the HTML you see on this page.
</small>
</fieldset>
</div>
<div id="debug"></div>
<script type="text/javascript">
// a jquery-like function, a shortcut to document.getElementById
function $(o) {
return document.getElementById(o);
}
// called on page load to make up a session ID (in real life the session ID
// would be made up via server-side script and "hard-coded" in the HTML received
// by the server, thus it wouldn't require javascript at all)
function init() {
// Make up a session ID.
var hex = [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9",
"A", "B", "C", "D", "E", "F" ];
var ses = "";
for (var i = 0; i < 8; i++) {
var rnd = Math.floor(Math.random()*16);
ses += hex[rnd];
}
$("sessid").value = ses;
// we set the form action to send the sessid in the query string, too.
// this way it's available inside the CGI hook function in a very easy
// way. In real life this would probably be done better.
$("theform").action += "?" + ses;
}
window.onload = init;
// This function is called when submitting the form.
function startUpload() {
// Hide the form.
$("upload-form").style.display = "none";
// Show the progress div.
$("progress-div").style.display = "block";
// Begin making ajax requests.
setTimeout("ping()", 1000);
// Allow the form to continue submitting.
return true;
}
// Make an ajax request to check up on the status of the upload
function ping() {
var ajax = new XMLHttpRequest();
ajax.onreadystatechange = function () {
if (ajax.readyState == 4) {
parse(ajax.responseText);
}
};
ajax.open("GET", "upload.cgi?do=ping&sessid=" + $("sessid").value + "&rand=" + Math.floor(Math.random()*99999), true);
ajax.send(null);
}
// React to the returned value of our ping test
function parse(txt) {
$("debug").innerHTML = "received from server: " + txt;
var parts = txt.split(":");
if (parts.length == 3) {
$("received").innerHTML = parts[0];
$("total").innerHTML = parts[1];
$("percent").innerHTML = parts[2];
$("bar").style.width = parts[2] + "%";
}
// Ping again!
setTimeout("ping()", 1000);
}
</script>
</body>
</html>
upload.cgi#!/usr/bin/perl -w
use strict;
use warnings;
use CGI;
use CGI::Carp "fatalsToBrowser";
# Make a file upload hook.
my $q = new CGI (\&hook);
# This is the file upload hook, where we can update our session
# file with the dirty details of how the upload is going.
sub hook {
my ($filename,$buffer,$bytes_read,$file) = @_;
# Get our sessid from the form submission.
my ($sessid) = $ENV{QUERY_STRING};
$sessid =~ s/[^A-F0-9]//g;
# Calculate the (rough estimation) of the file size. This isn't
# accurate because the CONTENT_LENGTH includes not only the file's
# contents, but also the length of all the other form fields as well,
# so it's bound to be at least a few bytes larger than the file size.
# This obviously doesn't work out well if you want progress bars on
# a per-file basis, if uploading many files. This proof-of-concept only
# supports a single file anyway.
my $length = $ENV{'CONTENT_LENGTH'};
my $percent = 0;
if ($length > 0) { # Don't divide by zero.
$percent = sprintf("%.1f",
(( $bytes_read / $length ) * 100)
);
}
# Write this data to the session file.
open (SES, ">$sessid.session");
print SES "$bytes_read:$length:$percent";
close (SES);
}
# Now the meat of the CGI script.
print "Content-Type: text/html\n\n";
my $action = $q->param("do") || "unknown";
if ($action eq "upload") {
# They are first submitting the file. This code doesn't really run much
# until AFTER the file is completely uploaded.
my $filename = $q->param("incoming");
my $handle = $q->upload("incoming");
my $sessid = $q->param("sessid");
$sessid =~ s/[^A-F0-9]//g;
$filename =~ s/(?:\\|\/)([^\\\/]+)$/$1/g;
# Copy the file to its final location.
open (FILE, ">./files/$filename") or die "Can't create file: $!";
my $buffer;
while (read($handle,$buffer,2048)) {
print FILE $buffer;
}
close (FILE);
# Delete the session file.
unlink("./$sessid.session");
# Done.
print "Thank you for your file. <a href=\"files/$filename\">Here it is again.</a>";
}
elsif ($action eq "ping") {
# Checking up on the status of the upload.
my $sessid = $q->param("sessid");
$sessid =~ s/[^A-F0-9]//g;
# Exists?
if (-f "./$sessid.session") {
# Read it.
open (READ, "./$sessid.session");
my $data = <READ>;
close (READ);
print $data;
}
else {
print "0:0:0:error session $sessid doesn't exist";
}
}
else {
print "0:0:0:error invalid action $action";
}
You can download my full proof-of-concept test below:
Notice: this code is called "proof of concept"; it is NOT production-ready code. You should NOT download this if all you want is a complete plug-and-play solution you can quickly upload to your web server to get file uploading to work. I wrote this code only to show how to make a file uploader in the simplest way possible; this is useful for developers who only needed to know how this is done and who will write the code themselves to develop their production-ready file uploader.
If you want to treat this as a plug-and-play solution, I'm not your tech support about it. The code was never meant to be secure or useful to allow the general public to upload files through it. Session IDs are made up client side for example which is a bad idea in real use case scenarios, etc.
There are 52 comments on this page. Add yours.
Neat trick! I'll have to use this some time.
that a perfect solution for most of developer headache. thank for your honesty to share. i always love simple non-commercial script.
can you help for config my page?
i used <?php session_start(); ?> at top of upload.html and echo session_id() in value of <input id='sessid', but got
received from server: 0:0:0:error session 7647507949712654928 doesn't exist.
what could i do?
thanks alot friend.
In my code the session ID is used in two places. In the JavaScript function init(), it sets the session ID in the text box you see on the page, and also modifies the form's action to add a "?" followed by the session ID.
When you click the upload button and the ajax starts polling it to see the progress, it sends the session ID from the text box. So, if you're replacing the session ID stuff with PHP calls, make sure you put the session ID in both places. The Perl script receiving the upload only knows about the session ID from the form's action, and if the JavaScript is using a different one when polling for the progress the Perl script won't be able to find it.
In any event, the code I have on my site is just a proof of concept, you'll eventually want to handle the session ID stuff in a better way than that.
i think.XUpload is an advanced upload of progress bar indicator for web based file uploads written and Perl and One of my major problems is attempting to have long-running processes initiated via a web browser and all the things that go with it: making sure the process works, not using the many server resources at one time, working within a basically stateless environment
I got the same error,
received from server: 0:0:0:error session CE54B75E doesn't exist
I haven't (yet) setup the script to use PHP session, this may or may not be the root cause.
Issue: When uploading a file, the progress bar will pause randomly, this happens about 75% of the time. It appears that the .session file is not longer being updated.
What could cause this?
Kirsle - you are amazing! That was EXACTLY the script I was searching for. I think uber-uploader is an excellent script for uploading and following the upload progress - but it is somewhat overkill. Especially what i don't like is that you are using a chimere of two languages (PHP and Perl). Your script is truly amazing because it does the job AND it is incredibly small! (which leaves a lot of options for customization) Great job!
Best uploader ever. Still is any way to show the upload speed? Thanks
Calculating upload speed just involves checking how much has been uploaded at two points in time and comparing them.
For example, if 100 KB was uploaded at one point and then the app checks how much was uploaded 5 seconds later, and it's now 200 KB, then it means 100 KB was uploaded in 5 seconds, or 20 KB/s (100 / 5).
The java script checks every 5 seconds? how can i set the time i want. Thanks
Just edit the JavaScript to increase the interval. Hint: look for the setTimeout lines, and change the number from 1000 ms (1 second) to whatever you want.
Ok i will try now.
I tryed but im not good at javascript:( here is the page http://mediacenter.7n.ro/upload.php
maybe you can help me
Hello,
can i use the script with more than one files ?
Krisle please help me always have error: how i can make?
received from server: 0:0:0:error session C6A77578 doesn't exist
your script! http://www.1m65.ru/u/upload/upload.html
My code isn't a complete application ready to be downloaded and installed on your server. Instead it's just a set of proof of concept code, demonstrating the bare minimum that's required to get a file uploader working in Perl. I wrote it only because it's hard to find this information online, and anyone who's ever had to dissect an existing file uploader application would know that the existing solutions are massive and bloated and it's hard to find out how they actually work.
I wish people would stop assuming my code is plug-and-play and ready to go; it's not; all it does is show you how to make a file uploader. It's not very secure at all and the fact that session IDs are generated client side is what many web developers would call "very bad practice" in terms of web security.
Having said all that, if you can't resolve this error on your own, you should probably find an existing plug-and-play solution; mine is useful to developers who want to write their own code, and developers would know how to debug errors like this.
Dear Kirsle!
Thank your for your reply! Im understand what this not plug and play script.
And ask your help me and find mistake.
Now i see Sub hook, making later than sub ping? why?
And on server file $sessid.session making to late.
The Perl CGI module will call the hook sub for each chunk of data received from the user. When the upload first begins, the .session file won't exist yet because the hook sub hasn't created it yet. So you should only see that error message for a couple seconds at most until the file starts being received by the server.
Even when the JavaScript gets that error, it still continues pinging the uploader to check the status again.
Also note that this won't work well at all on Safari and Chrome; on these browsers, all ajax requests get blocked as soon as the page begins trying to load another page. There's a workaround for this by submitting the form into an iframe on the page, via , so that the main page doesn't have to go away so the ajax requests aren't stopped. The result page that loads in the iframe, then, would do a parent.window.location="..." to send the user to the results page.
Im understand about Safari and Chrome. But im used Exporer and Mozilla.
1) No im comment: #unlink("./$sessid.session");
2) And server have: 70A26E90.session
only when file is UPLOADED.
all files. 1444548:1444942:100.0
But when file uploading session file is not making. He not have on server.
3) Im not understand why is sub hook make file only in last time.
It may be due to a special server configuration. My code was tested on a default Apache install on a Fedora server. If the hook sub is only being called once the server has received 100% of the file, then I'd look into the server's configuration; it would seem to be buffering the file somewhere and not invoking the CGI script until it already has the entire file.
Thank Your!
Im test on another serever every is ok, i think is problem with this server.
Excellent script. I'm learning Perl and this went a long ways towards that end.
One issue I found is that this script does not treat binary files very well on a Windows (IIS) host.
I changed this bit of code...
# Copy the file to its final location. open (FILE, ">X:/upload/$filename") or die "Can't create file: $!"; binmode FILE; while (<$handle>) {print FILE;} close FILE;
...and it now works on IIS (tested).
Near as I can tell, this method would be good practice on any system and regardless of file types being dealt with.
Thanks again.
-T.
Indeed. I usually always put binmode on my filehandles, but as this was just a test script and I was only running it on a Linux server (where binmode doesn't matter), I didn't bother to write that extra line of code.
Thanks for pointing it out though; shows your Perl skills are coming along pretty nicely too.
You have done a great job!
Your solution is very compact and does the job perfectly.
Congratulations and THANK YOU!
Fantastic! Thank You.
What is
"&rand=" + Math.floor(Math.random()*99999)
for in function ping()?
The web browser will cache the results of an ajax request if the request URI is the same each time. Adding a random number to each request prevents the caching and forces the browser to hit the server again.
Outstanding!
Hi There,
I sent a file 140MB and the progress stopped at 40MB, the upload was still going and the session file still being updated but no more ajax pings were being generated by the browser (IE8)
I thought maybe the ping was being seen as a TCP flood attack so I turned off TCP flood detection at both ends and set to 3000 ms This time the browser stopped pinging at 70MB, I ran a sniffer and there were no ping requests coming from the client.
I increased to 10000 ms and got the whole file through OK :)
Do you know if ajax has a limit (quantity or time)?
Just a couple more points for anyone doing large uploads.
# Copy the file to its final location.
open (FILE, ">./files/$filename") or die "Can't create file: $!";
my $buffer;
while (read($handle,$buffer,2048)) {
print FILE $buffer;
}
close (FILE);
This is very slow and happens after the upload hits 100% leaving the client wondering what is going on now?
At the start of the script
my $q = new CGI ( \&hook, undef, 0 );
will tell CGI not to bother saving the data
I then add this in sub hook {
# Write file chunk to a bin file.
open (NEW, ">>$sessid.bin");
binmode NEW;
print NEW $buffer;
close (NEW);
so that the file is saved as it arrives
I then and replace the copy code above with
rename ("$sessid.bin", "./files/$filename");
On a Windows system this passes the task to the OS and is instant even if the destination is on a different drive or NAS!
finally
close $handle;
Will ensure that the CGITemp file gets deleted, in my case it's a 0 byte file anyway.
Please ignore my ajax limit post I was sending over ssl and each ping opened a new tcp connection I will try getting the pings unencrypted via a seperate script :)
Hi kirsle, first, Thanks! 2nd i have a problem :S ... that example not work for me on IE and Chrome, and i new in perl, i cant resolv, can you help me? please...
(sorry for my bad english)
On Chrome (and Safari) there's a known problem where ajax requests aren't allowed to run when the page begins trying to load a new page (so, when you hit the submit button, that stops ajax from working).
I posted a workaround in an earlier comment:
"Also note that this won't work well at all on Safari and Chrome; on these browsers, all ajax requests get blocked as soon as the page begins trying to load another page. There's a workaround for this by submitting the form into an iframe on the page, via , so that the main page doesn't have to go away so the ajax requests aren't stopped. The result page that loads in the iframe, then, would do a parent.window.location="..." to send the user to the results page. "
I haven't done extensive testing on IE (might have only tried IE 6)... it worked for me on whichever version of IE it was that I tested, but it might be that IE is following Chrome's lead now on that ajax thing.
Thanks for such a compact and simple to comprehend code.
Just 1 question. I have been searching around to figure out how to write the final line I need. This final line would replace the "Print" line after the file has completed it's download.
I don't know too much about cgi.
I need a line of code that will load a specific PHP file using the POST method.
Can you provide an example that will work with your code? Many Thanks.
Good job! I planned to implement an upload progressbar to my site this weekend. This will help. The CGI module (which is a core module in Perl) appears to be at the heart of most implementations I have seen so far on the web. Apparently PHP is not cut out to do this kind of work at such low level. For those looking for more info on CGI.pm check out the documentation at http://search.cpan.org/dist/CGI/lib/CGI.pm . Thanks for the article, you are right that the examples on the web are not very good.
Great script! I have a question: My hosting provider limits the time that a perl program is allowed to run to 1 minute. So if a user has a slow connection or a large file, the upload fails. Would it be possible to end the cgi script every 30 seconds from the browser, and then restart and append the next 30 seconds of upload to the file on the server?
As far as I know that isn't possible. :(
Thanks for this insightful solution, K. I love how you syntax-color-coded the example files. How did you do this?
I used the vim text editor (http://www.vim.org/). The commands are:
:let html_use_css=1 :TOhtml
And then :wq the file it generated and there ya have it.
Thanks for sharing. The script, even if it's just a proof of concept, seems to work almost well.
The only bug I noticed is that the ajax ping doesn't stop to be executed when the upload is completed even if the file is correctly uploaded and you will see "received from server: 0:0:0:error session NUM OF SESSION doesn't exist" .
I am using an iframe as target of the form but I don't think is the problem. On your original version the ping stop only because the page is reloaded.
Do you have any hint? Using a JSON as output method and intercept the response within the ajax call will do the trick perhaps ?
-Thanks
Pardon my ignorance, but I don't quite understand your explanation of how to use an iframe to get this to work with Safari or Chrome.
Where do you put the iframe? Somewhere on upload.html, I assume?
</p></li> <li><p>What is the src for the iframe? upload.cgi doesn't seem to work. I tried making a third file I called iframe.html and made that the src, but that didn't work either. I just end up with the "0:0:0:error invalid action unknown" message as soon as I load upload.html</p></li> </ol> <p>iframe.html</p> <pre><code> <!DOCTYPE html> <html> <head> </head> <body> iframe<br> <script type="text/javascript"> if (window.location != window.parent.location) window.parent.location = "upload.cgi" </script> </body> </html> </code></pre> <p>A push in the right direction would be much appreciated. Thanks.</p>
@Egon:
What you'd do is, the HTML form would post into the iframe, like this:
<iframe name="my_iframe" src="about:blank" width="1" height="1" style="display: none"></iframe> <form name="upload" action="upload.cgi" target="my_iframe">
So when the form is submitted, it gets posted into the iframe instead of the main window. Then in upload.cgi when the upload is complete, it would print out some JS code that would make the parent window redirect, like,
<script> window.onload = function() { parent.window.location = "/upload-complete.html?id=$whatever"; }; </script>
Great. That makes much more sense and it worked like a charm. Thanks for the quick reply.
Hi, I love the script. I am trying to modify it for use with multiple files. Any ideas on that??
Thank you very much for making such an awesome script! Regards, Tsubasa Kato
Is there another option besides Ajax? Ajax runs off PHP and my company will not allow PHP on its servers.
Is there another option besides Ajax? Ajax runs off PHP and my company will not allow PHP on its servers.
Ajax isn't directly tied to PHP, the back-end could be any kind of codebase.
Anyway, more recently I have found how to get an upload progress bar on the JavaScript end of the transaction, without the server side needing to help this along. On GitHub I have a flask-multi-upload demo I put together when figuring this out. The back-end there is written in Python, but that's not the important bit, here is the JavaScript side of it.
The relevant bit is this XMLHttpRequest hook where the front-end responds to the upload progress to show the progress bar. This example is using jQuery, I'm not sure off hand how you do this in 100% pure JavaScript (there are probably node.js modules available to make it really nice). This StackOverflow answer may help, seems with the raw XMLHttpRequest API it looks something like this:
let xhr = new XMLHttpRequest(); xhr.open('POST', ''); xhr.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); xhr.upload.addEventListener("load", function () { // a "done" handler, do what you need with this }); xhr.upload.addEventListener("progress", function (event) { if (event.lengthComputable) { var complete = (event.loaded / event.total * 100 | 0); document.querySelector('.meter').style.width = complete + '%'; } }); xhr.send(null);
This script works great, thanks!!! I have a question with an error that appears after the upload is complete. I've included it below. My script says: unlink("$sessid.session") if "$sessid.session"; so it should never try to use unlink if its not there. Any ideas? Also, I can never get the image to display in the progress bar, only the blue color. Tried every possible css combination but will not work.
[Fri Apr 1 11:03:11 2022] upload.cgi : unlink0: C:\WINDOWS\TEMP\ERr3Ye5olM is gone already at C:\inetpub\wwwroot\cgi-bin\drivers\apps\upload.cgi line 0.
Hey @Mike Herbstreit
I'm guessing your code snippet unlink("$sessid.session") if "$sessid.session"; had a typo and was missing the -f operator that makes Perl test if the file exists?
unlink("$sessid.session") if -f "$sessid.session"; # or maybe (my Perl is rusty at this point..) if (-f "$sessid.session") { unlink("$sessid.session") }
Although either way I'm not sure why it would say the file wasn't found unless the code was somehow being double-posted to (where the first post unlinked the file, and the second one tried again and failed) - but the -f check should hopefully prevent the error happening in that case.
Also, I can never get the image to display in the progress bar, only the blue color.
The CSS is pointing to the blue-clearlooks.png that was in the zip, relative to the upload.html file, if they're in the same directory on your server it should work but if you moved blue-clearlooks.png somewhere else the CSS should update to point to the new location. You may also check your web browser's Developer Tools on the Network tab and see what URL it was trying to load the image from and verify it's the correct one, or make the necessary adjustment if not!
Thanks Noah! The -f fixed the issue with unlink. Regarding the image, I've tried relative path and URL without success. Perhaps it's the way I nest the code. Thanks for your help!
Hi, Noah :) Thank you for doing the work and on top of that, giving it back to the world upon completion. I know too well how it is searching around for code to build on when you need something you haven't done before, testing it, altering and expanding it, making it serve your particular needs, maybe encountering issues never encountered before. Sometimes it is pretty straightforward and other times it is hair ripping :P Much appreciated <3
Works pretty much like a charm, and even with the small annoyance of the below explained situation, it's a much better solution than having the user staring at the upload form he submitted, not seeing that the page is loading, and wanting more and more to either exit or hitting upload again, and again. Of course we program in double click (or x click) prevention, timeouts between uploads. and duplicate checks, but all that does for the user uploading, is tell them that they did something "wrong", it doesn't tell them why it was wrong necessarily, and that the file IS actually being uploaded,
I was wondering, no matter how low I set the timeout (don't want to set it freakishly low, the server admins might not like a frequency of .05 seconds and such), the bar will not start showing progress until nearer the end of the upload , maybe somewhere between midway and endway, for big files. Is this some Apache setting or something on my particular server, or does the script do the same when you run it?
Hi again :)
I guess you've discovered ages ago what I discovered over the course of trying this out, the cause of the lagging, and have abandoned this solution as I will. It was a beautiful dream, though :)
Hey @malmklang
I don't program in Perl very much anymore, but in the mean time since I wrote this post I had learned more about how to make these kind of features work, and a lot of it can be done purely on the front-end in JavaScript nowadays.
I have a Python example of this on GitHub here: https://github.com/kirsle/flask-multi-upload
Python has a lot of similarities to Perl (interpreted scripting language, single threaded, etc.), the Python back-end of this example is rather simple and straightforward: it reads the uploaded file and stores it, the same as a naive Perl upload script would do (without needing to install any CGI hooks to monitor the upload progress or anything at all fancy).
Instead, the progress bar in this example is handled on the front-end in JavaScript using XMLHttpRequest hooks. The example is using jQuery which is a little dated now, but with a bit of research it should be possible to find the equivalent vanilla JavaScript XMLHttpRequest progress event.
This measures the upload progress from the browser's side, so should be fairly accurate and doesn't need anything fancy on the server: your script can just do a straightforward $cgi->upload() read without the event hook needed.
0.0109s.
