A software developer sits at her keyboard writing a series of words. She saves them to disk and then clicks a button and the words she has written operate, they do. There is a strange fascination in the idea of a type of writing where words act directly on a machine, rather than on the brain of a reader.
The question of whether software is art or engineering has been little considered and never answered. Looked at from either angle, software is unique and does not fit easily into established categories; if art, it is art that acts automatically, when completed, through the interpretive medium of a machine, and can act in no other way; if it is engineering, it is the unique engineering discipline where the concept may become the machine without the usual manual phase where the concept is reduced to physical reality.
It is important to remember that, like any other human categories, "art" and engineering are not separate watertight compartments, but overlap each other. Most engineering experiences contain some elements of "artistic" creativity, and some art (particularly those arts expressed in physical materials such as stone) contain a significant engineering element. In either case we tend to assign the resulting work product primarily to one of the categories. The category to which we should assign software is not a foregone conclusion.
Art frequently involves a communication directly from the writer's unconscious to the medium in which the art is expressed: a theme springs to mind and is put down on paper, with nuances, twists and detours which the artist does not "plan" but which close a loop for her, often creating a sense of physical satisfaction and well-being. By contrast, a didactic novel whose intellectual theme is chosen, then planned to include particular incidents illustrating philosophical ideas, may be said to be "engineered". Most such novels fail as too didactic, too much of a lecture when an artistic experience is expected.
Another feature of the artistic experience is that it takes place in a black box, so to speak: it either is completely self-generated by the artist (very rarely is a first novel commissioned from a writer) or, if there is a patron, he does not attempt to control the process. The more independence I have, the more clearly I am following an artistic and not an engineering approach. If you ask Michelangelo to paint the ceiling of your chapel, you are asking him to function as an artist; if you dictate to him the nature, coloration, size and relationship of the objects he will portray there, you are asking him to function as an engineer.
Most software development today is done in the latter environment: I commission you to implement (for example) an electronic commerce system for the auction of rare coins; we work together to define the requirements, set milestones and deliverables, and you undertake to produce the software we have defined by the dates we have agreed. Like most engineers, you still have the opportunity for some independence and artistry, in creative and elegant technical solutions to problems. But most of your time may be occupied with engineering issues: what resources to throw at the problem in what combinations to produce the promised result on time.
Many new categories of computer software begin in an artistic environment: significant types of software used on a personal computer, such the spreadsheet, the database, and the browser, each have an original "artist" associated with it who created the original prototype alone, without deadlines, and investigated how to turn it into a product later. Looked at this way, original artistic works were transformed into engineered products later.
The borderline between art and engineering may be in large part based on whether we begin with a solution to a public problem or to the artist's individual problem. "Create me a web-based rare coin auction system" is a public problem. Finding a neat software implementation of hypertext, viewable across public networks by a thin client easily installed on any PC, sounds more like a solution to a personal urge of an artist: only after he has created it does the investigation begin as to what public problems the product may be used to solve. Sometimes, of course, we wind up with a neat solution which matches no public problem. My favorite example is the product which split the output from one VCR to several televisions, something no-one would ever need to do in their home.
From this point of view, software is most like art when it is done privately, quasi-secretly, on the developer's own time, therefore in a complete black box, or (similarly) in an R&D environment with broad independence, the ability to choose ideas of interest and express them in the artist's own fashion, and little management of the process in terms of deliverables or deadlines. Software is not unique in this sense; many products of physical engineering disciplines have emerged from this kind of environment (famously from the aviation "skunk works" at Lockheed).
The risks of this approach to software are bad products, or brilliant solutions which match no known public problem. This is no different than the phenomenon of commissioning a work of art which is never completed, is inferior or which is brilliant but does not match the patron's expectations.
Software considered uniquely as an engineering discipline raises a completely different set of problems. Much has been written on the phenomenon of project failure: large software projects on which millions of dollars are spent but which do not result in a working product. While in many cases the blame is placed on poor management (inadequate planning, bad resource allocation, lack of a methodology, etc.), the question has not been raised whether significant blame for the failure lies in treating software as an engineering process when it is not.
Physical engineering creates products for a world which operates according to rules that are fairly well-understood. Gravity will affect buttresses in a particular way that never changes. If you imagine building bridges for a world in which the gravitational pull may change in intensity or even reverse itself, you have a better understanding of the inherent problem of engineering software. Software runs in an environment of such complexity that the problems of engineering for it appear to be much greater than in mechanical or electronic engineering. Even very simple issues can be extraordinarily difficult to solve. For example, I get dropped from my ISP two or three times in an email session: is the problem in my CPU, its serial port, my email software, the modem, the phone line, or in the host computer I am accessing (and if it is in the host computer, is it a software problem, a network problem, their modem, etc.) Many software failures involve elements which work fine by themselves but have unintended consequences when used with each other (competing for the same space in memory and locking up the computer, for example.)
Every computer user is used to occasional crashes as a fact of life. The laptop on which I am writing this essay has frozen several times in the past few weeks, twice crashing so irrevocably that a "ctrl-alt-del" would not reboot the machine and it had to be turned off. One of those amusing files which floats around the Net has a title like "if operating systems were cars...." and says that such cars would cost less than $100, be able to carry large numbers of passengers hundred of miles on almost no gasoline, and then crash at least once a month killing everyone aboard. Another memorable joke from a few years ago: Windows 95 would improve on 3.1 by assigning a particular function key to generate a General Protection Fault; you would no longer be required to run an application in order to crash your computer.
Large corporations live by a particular set of rules which involve knowing when things are going to get done and how much money they will cost. Naturally they want to commission software projects, or do them internally, according to this same rulebook. The problem is that software projects done this way assume a level of stability that does not exist: you must plan for accidents presently unknowable and for the shifting of conditions which change across too wide a spectrum to be planned for.
Years ago, Frederick Brooks described the law which was later named after him: the more people you have working on a software project, the longer it takes, due to management overhead and the time it takes them to communicate with each other. I am not certain if Brooks' law can be equally applied to other engineering disciplines; it is possible that other types of engineering are more modular and that the people working on the parts don't need to speak to each other as much. (Object oriented and component based approaches are supposed to make software development modular as well.)
There is another side effect of team-based software development Brooks didn't describe, and which is best summed up in the old saying that "a camel is a horse designed by a committee." There is a fascinating gap between experience and current practice when you compare the creation of Mosaic or Visicalc by individuals working alone or in very small groups with today's received wisdom that no commercial product can be created by a team of less than fifty to one hundred developers.
I think it would be helpful to regard software largely as an art form. New disciplines often start under the radar and are perceived only later to be art. The artist of the Lascaux cave paintings may have believed he was performing a sort of engineering (forcing a desired result from the universe or the gods by portraying a succesful hunting expedition). During Shakespeare's lifetime, plays were considered to be commerce. Even when a discipline is recognized as an art form, it frequently struggles to free itself from an "engineering" approach: Movies today are an attempt to implement someone else's creative impulse via committee, with predictable results.
The French New Wave introduced the "auteur" theory of film-making, and perhaps what we need now is an "auteur" theory of computer software. This implies finding and fostering the "artists"--the creative talent who will be at the center of a project, if not working solo on it. It would also require some other changes in philosophy in the corporate world: less concentration on the budget and more on the best approaches for obtaining a quality result; and the counter-intuitive idea that software can be done better with fewer people involved.