Kirsle.net
In this blog post, I'll recount my experience in web development over the last 20 years and how the technology has evolved over time. From my early days of just writing plain old HTML pages, then adding JavaScript to those pages, before moving on to server-side scripts with Perl (CGI, FastCGI and mod_perl), and finally to modern back-end languages like Python, Go and Node.js.
Note that this primarily focused on the back-end side of things, and won't cover all the craziness that has evolved in the front-end space (leading up to React.js, Vue, Webpack, and so on). Plenty has been said elsewhere on the Internet about the evolution of front-end development.
I first started writing websites when I was twelve years old back around the year 1999. HTML 4.0 was a brand new specification and the common back-end languages at the time were Perl, PHP, Coldfusion and Apache Server Side Includes.
The very first websites I made were actually built using a drag-and-drop WSYIWYG tool called Juno Homestead which, at the time, was hosted by the internet service provider we used, Juno.
But when I started to write my own websites from scratch, I wrote them in plain old HTML -- and not the modern HTML, either.
We had to use tags like
<font face="Arial" size="2" color="blue"> to manage the styling of our pages.
The <body> tag had attributes like link="blue" vlink="purple" alink="red"
to set the color of your hyperlinks, a bgcolor and a text color to set the
page background and default font color. Page layouts consisted of tons of nested
<table> tags, and cool effects like rounded borders were accomplished using
images in the corners of our tables.
The <frameset> tag was also commonly used around this time to put your website
into frames. Anyone remember this? You could have a left-side frame holding your
"navigation.html" and the right frame holds your "home.html" page. The nav frame
would link to pages targeting the right frame, so that you didn't need to
copy-paste your nav bar into every single HTML file on your site! Nowadays all
that remains of framesets is the <iframe> tag for embedding a page inside
another.
Aside: HTML 4.01
HTML 4 was a brand-new standard at the time, not yet fully supported in the modern web browsers. In HTML 4, we would lose the
<font>tag and table-based layouts were deprecated in favor of this new-fangled thing called Cascading Style Sheets. It took some time to get used to.
Before long, I picked up some basic JavaScript skills so I could make my pages do "cool" stuff, like pop up an alert box when you click a button or validate form parameters.
In one of my earliest websites ever, I programmed a really basic Pokémon style
battle system. Each .html page showed the current battle progress and had links
to other pages in response to clicking attacks to use. At first the battles were
pretty deterministic: you were navigating a complex tree of dozens of HTML
pages depending the order you clicked the links.
When I made a more advanced version of this "game," I used a <frameset> and
had a hidden form with text fields to hold the score, and JavaScript would
read/write those fields to keep track of data as you navigated the site.
There used to be two dialects of JavaScript: the Netscape one and the Microsoft one, JScript. They had subtle differences, and web pages often had to detect whether you were running Netscape or Internet Explorer and serve entirely different websites to each browser! The lazy webmasters, though, would just put a little Works best in Netscape badge on their page and not bother.
I actually learned Perl outside of a web dev context: there were some chatbots popping up on AOL Instant Messenger and I wanted one, and found some Perl programs that ran on desktop computers that you could program bots in.
But then I learned Perl can be used on the web for server-side scripting.
See, with JavaScript you could run scripts on your web page that run in the
user's browser. But you can't do certain things with JavaScript, such as let someone
send you an e-mail through a <form> on your page, or to create a Guestbook page
and let users save comments on your site (that could be seen by other users).
With Perl, you could create a .cgi file and place it along with your .html
and .js files on your website. When a web browser visits your Perl (.cgi)
script, the server would execute your script on the server side so you could
do things with the browser's request before finally delivering an HTML response
for it to display.
For example your guestbook.cgi could open a text file on the server holding the
previously-saved comments and then send the browser some HTML code that displays
them. Then a <form> on the page could POST back to the guestbook.cgi and it
could append the new comment into that text file.
Thus I learned about the Common Gateway Interface, or CGI. If you were
programming in PHP around the same time, you were likely also working with CGI.
CGI defined a standard way that the web server software (like Apache) would
interact with an arbitrary program to handle a web request. Apache would set
environment variables like QUERY_STRING and REQUEST_METHOD which the script
could access (to accept form parameters, etc.) and the script in turn would
print out a Content-Type (like "text/html") and print HTML code for the server
to give to the web browser.
My first exposure to writing Perl CGI scripts was on the web hosting company Tripod. They were similar to Geocities and Angelfire for the early web. Tripod would let you upload your own Perl scripts, but they did NOT provide any standard modules to use with them.
So, for a Perl script to process query string parameters or form data, we had
to do it the hard way. The environment variable "QUERY_STRING" held the raw
query string, like ?name=Bob&age=20, minus the leading "?" mark you see in
your web browser's URL bar. We'd have to manually split the parameters at the
& sign and then again at the = sign to make sense of them for our script:
#!/usr/bin/perl
# Parse the query string into a hash map of parameters.
my %params;
my @query_parts = split(/&/, $ENV{"QUERY_STRING"});
foreach my $part (@query_parts) {
my ($name, $value) = split(/=/, $part, 2);
$params{ $name } = $value; # NOTE: you'd also need to URL-escape it,
# turn the %20's back into spaces etc.
}
# Send an HTML response.
print "Content-Type: text/html\n\n";
print "<html><body>Hello, $params{'name'}!</body></html>";
For POST requests from HTML forms, the Perl script would read its standard input like a file and parse the results in a similar way.
If I weren't using Tripod and instead had a proper Perl web host, I would've had
access to the CGI.pm module which would handle this common low-level stuff
for me. Then I could just call param("name") and get that query parameter
without parsing it myself.
In PHP sites, the query and form data is already parsed for you into the $_GET
and $_POST objects respectively. In Coldfusion, into the #Form# variable.
When you're dealing with raw, low level Perl though you had to do it by hand.
CGI scripts were really cool, but they had a slight problem: for each and every web request that came in that ran your CGI script, the server needed to run the script from scratch every time.
Perl and PHP are dynamic languages, so each time Perl executes your script it needs to parse it, compile it in memory, and run it. This only takes a few milliseconds, but for popular websites these milliseconds add up.
My first dozen or so websites I programmed made heavy use of Perl, and eventually I needed to investigate better technology to speed my sites up.
Enter FastCGI. The basic idea behind FastCGI is that your Perl script should only be loaded up and compiled once, and then re-used as many times as possible to serve many web requests before the script exits. For Perl you had to specially adapt your scripts to be able to run in FastCGI mode, but it was worth it.
Modern PHP sites often run in FastCGI mode today, especially when using the nginx web server software.
Real production-ready Perl websites don't use FastCGI, though, but instead commonly used the mod_perl plugin to the Apache web server.
With mod_perl, the Apache server itself runs your Perl script using an embedded Perl interpreter. This lowers some of the overhead of calling out to a separate Perl executable and (allegedly) has even better performance than FastCGI does.
I never used mod_perl personally for my little old websites, but the first few Perl shops I worked at professionally all used this module. mod_php is similar but for PHP sites.
Whether using CGI, FastCGI or mod_perl, your Perl (and PHP) sites sit very close to the web server software, like Apache or nginx.
At the very least, Apache is executing the Perl interpreter to run the script, or at most Apache itself is running it. In both situations, the script tends to be executed as the system user account running the web server and has very close ties to it.
One implication is that, if you wanted to set up a local dev environment on your laptop where you can develop a Perl site without directly working on the production server, you needed to also install Apache and set everything up. No Apache, no Perl site. Also, oftentimes restarting the Apache server would also kill and restart your Perl sites.
After working for a few Perl shops professionally, I broadened my horizons and took a Python job. What a breath of fresh air this was!
Python sites tend to work fundamentally different to Perl, PHP or Coldfusion sites. With Python, I can easily run a local dev environment on my laptop without needing to even install Apache or nginx. The Python script creates its own HTTP server, and I can navigate to http://localhost:5000 and see my Python web app running locally.
See, unlike CGI based back-end sites, Python itself runs a web server.
With an Apache server, when a request comes in for "/index.html" or "/theme/style.css", Apache by default would serve that file from the document root on the hard drive. When the request is for "/guestbook.php" Apache would execute the PHP script using the CGI standard and PHP would do its thing and then return an (HTML) result for the web browser.
In Python, you are the program that processes those web requests. If the browser wants "/favicon.ico" it's up to your Python script to send it the icon file data it wanted, or when the web browser posts to "/guestbook" that's a Python function that runs to handle it. (Note that in real-world websites, a production server usually lets Apache or nginx handle the basic static files like images and icons and Python would only be invoked for the URLs it needs to handle).
In a typical Python application, Apache isn't used as much for server software and instead nginx is more preferred. Since the Python app speaks HTTP itself, nginx simply proxies your request directly to Python. The Python server runs as its own process separate from nginx, and nginx can be rebooted without the Python server being rebooted, and vice versa. You get a very clean separation of concerns, and your Python app isn't closely bundled to the server the way a Perl or PHP site is bundled closely to Apache.
At work we were using Python with nginx, but for my personal sites I still had a lot of legacy Perl code running on Apache, and was making heavy use of Apache specific features such as mod_rewrite (redirecting all web URLs to my index.cgi script, so I could have pretty URLs despite using Perl).
Python is able to run as a FastCGI service in Apache the same as Perl and PHP sites, and this is how I configured it at first. Now I had my Python sites running alongside my Perl ones, but they were closely coupled to Apache now and this I did not like much.
When I said Python apps natively run their own HTTP service earlier, I really meant they use a protocol called WSGI (Web Server Gateway Interface). It's Python-specific and is basically HTTP-like and is how nginx typically talks to Python applications.
Apache has mod_wsgi to enable WSGI support between Apache and Python and I briefly used this to bridge the gap while converting over to nginx.
Later on I discovered the Go language and I love it. My web blog is currently written in Go, after previously being written in Python a couple times and in Perl a dozen times more than that. (I like to keep re-inventing my blog over and over again, okay?)
A web app written in Go works very similarly to Python: the Go program itself runs an HTTP server and handles all web requests by calling various Go functions for each. In a production environment you would typically have nginx in front that proxies requests to the Go app.
Unlike Python though, Go by itself is a very capable server program. Python has a disadvantage of being effectively single-threaded (due to the Global Interpreter Lock). Python apps in production are typically configured to run multiple clones of the program, sometimes on multiple servers, in order to handle the volume of web requests coming in. Python can't easily handle simultaneous requests with just one process, so you need to run many. Go on the other hand can get quite far with just a single process: it has built-in concurrency features that are top notch.
Some people even elect to have their Go app be the web server and remove nginx from the equation all together! I don't do this though because I still run various Python apps that need to share the web ports, so nginx is here to stay for me.
Server-side JavaScript web apps in Node.js also work on a similar principal to Python and Go. They run their own HTTP service and nginx (or Apache) proxies to them. Ever since entering this layer of web development, I could never go back to the Perl way of closely-bundled CGI scripts and heavy Apache-based dev environments. Yuck!
The landscape of (back-end) web development has evolved considerably over the years, from the early days of CGI scripts executed by the web server to the more modern era where your web app stays a safe distance away from the web server. Ever since moving into the realm of apps written in languages like Python, Node and Go I wouldn't wanna go back to the old days of Perl and PHP type sites.
There are 6 comments on this page. Add yours.
Great info, thanks. What languages would you say a new web programmer needs to learn? By learning, say, Go, can one realistically avoid leaning Perl? What are the things that we no longer need to know?
jj
@jj: for back-end web development I'd recommend Python or Node.js, as these are still very popular languages and there's lots of jobs for them. Go is a good one too; the language is picking up momentum in the professional world, but isn't quite as well adopted as Python and Node are.
Perl, as much as I loved it for being my first programming language, isn't necessarily worth learning anymore in 2020 if you don't already know it. I still write a small amount of Perl for shell scripts on my servers, but haven't written a web CGI script in a long time. PHP is still a popular language and a lot of new products are still written in it, but I've never been a fan of that language. See "PHP in contrast to Perl" which mostly gives a good overview of all the weird quirks of PHP language. The page may be out of date by several versions but this was what PHP was like when I had the choice of learning it and decided, "nah!"
You can also consider working on the front-end side of things. Set up a Node.js dev environment to build front-end code using a framework like React, Vue or Angular.
When I started in professional web development, I was a "full stack engineer" doing both front-end and back-end -- but front-end at the time only consisted of basic HTML, JavaScript and jQuery; before all the modern front-end frameworks blew up the scene and before Node.js even became a thing. Nowadays there is a great divide between front-end and back-end, and each side has a whole entire world of knowledge to be learned, so most people specialize in one side or the other. Myself: I mainly prefer back-end development, but I can do front-end (in like Vue.js) -- I'm just not as efficient in front-end as some of my co-workers who specialize in it are.
For my personal projects like this web blog, it's all back-end code rendering HTML responses for the web browser, with very little-to-no JavaScript needed on the front-end. That's the way I like it, personally.
Noah,
Thanks for the response. I asked a question earlier about doing DOS on Linux with USB support, and your response was very helpful. In fact, I printed it out and re-read it several times. This reply is likewise great, especially because it comes from an experienced and talented web developer. I take it that by "back end" you mean server side, e.g., CGI scripts, and by "front end" you mean client or end-user pages. I am not a professional programmer, so I don't know the lingo, and I am probably not as smart as you are, so I am very grateful that you take the time to respond to my questions. Surprised to hear Perl is no longer recommended as first rate.
I note that you wrote a script to generate custom prompts in Linux. Years back I wrote a batch file to generate custom prompts in MS-DOS, and it won the Best of the Batch award and got me published in DOS World magazine, which I was very happy and surprised about, since I was new to computers, and that was the first program that I ever submitted. After reading your response to my question about DOS on Linux, I was able to transfer my old programs onto MS-DOS as a guest with VirtualBox, and that particular script worked fine on MS-DOS, but it didn't work on FreeDOS, even though, of course, I installed ANSI.SYS. Since FreeDOS is an MS-DOS emulator written in C/C++, and they didn't have access to the MS-DOS source code, I would be surprised if all MS-DOS programs worked on FreeDOS, even though that was their goal.
Anyway, thanks again. You've been a lot of help.
jj
I take it that by "back end" you mean server side, e.g., CGI scripts, and by "front end" you mean client or end-user pages.
Yeah, back-end means "code running on the server machine" and front-end is "code running inside the user's web browser." Back-end could be CGI scripts or servers written in Python or Go which render HTML pages for the user, or they could be API servers that send JSON data to a front-end script (the latter is what big sites like Facebook tend to do: they're very front-end heavy with all page transitions happening in JavaScript on the browser, and they get "just the data" from the back-end to update the page with).
My personal flavor of web development is to keep it old-school: the back-end server renders the HTML response to the browser. This way pages work even if you disable JavaScript entirely. A lot of modern front-end heavy sites completely break with scripts disabled and all you'd see is a blank white page, because they need JavaScript to even show the site layout and theme! The best of both worlds is to have the server provide HTML and then have front-end JavaScript take over and enhance the experience (reduce page-load times between clicks on links through your site, etc.) so it's still accessible to bots and users with scripts disabled but is a pleasant "modern" web experience for the average user.
I don't go too crazy on the scripts for kirsle.net though. :)
Noah,
Thanks again, you're an amazing person!
jj
(PS I go so far as to think that old school is madness, but it's madness we need more of. As for me, I admit that I like writing in completely dead languages like QuickBasic, DEBUG scripts, and MS-DOS batch files. What is more mad hatter than that?
Dead languages don't die; they just fade away into ignominious obscurity where spies and snoops and programmatic pilfering agents care not to go. Programs written in dead languages are not broken by new language developments, while cutting edge programs die by a thousand cuts, often changes for the sake of change, and it is often a hefty bit of pocket change, too. Dead language programs can hold and hide hidden treasure, and scoffed at, they can yet sail openly on the high seas unmolested.
As a famous Chinese statesman said, though I forget who, "We must become the empty ship, because an empty ship is not fired upon."
Penguins may not fly with eagles, but they don't get sucked into jet engines, either.
Yesterday, a lecturer I know said, "Anti-worlds? Utter nonsense!" The world of science is dead right. I toss and turn in my sleep. Meanwhile, my cat lies like a radio, tuned into the world with her green eyes.
-- Vozenensky
Vozenensky was right! Philosophy is history. Poetry is just posies. Religion is wacko.
But, "Who would be sane if it weren't for the mad? Where's the oasis without the
sand?" (Vozenensky) "What is madness but nobility of soul at odds with circumstance?"
(Roethke) Hey, what's wrong with posies, anyway?
"All mattterings of mind do not equal one violet." (ee cummings)
It's all Funsville.. Your web site is funsville, too, and I'm learning a lot, thanks. )
thank you so much for this post, it helped me a lot in understanding about the underlying fundamentals of web technologies.
0.0117s.
