Originally published in the September 1996 issue of:

---

Sometimes You Break the Code,
Sometimes the Code Breaks You

By Barry McKinnon

Just a generation ago, a sizable chunk of the professional sound electronic products on the market had the cheerful glow of tubes to warm the hearts and the equipment rooms of the users. Digital was a word that people would associate with fingers before they would think of computers. The word "software" would have been more likely to conjure an image of a down-filled quilt or pillow than a cantakerous collection of 1's and 0's loosely fixed to a disc (floppy or otherwise). Now, 30 years later, you're hard pressed to find a piece of professional audio, video and control equipment that isn't directly or indirectly associated with some type of software and some form of digital control or signal processing. Systems Contractor News took a brief look at this phenomenon in Wade McGregor's article, "Who owns the Code," in our NSCA Daily issue in May, but we felt it was worth a little deeper look into the implications this has for the manufacturers, consultants, and contractors in the systems contracting market, as well as the impact on facility owners.

There has been a great deal of material written about technology convergence and what this means in terms of training and the needed knowledge base to stay with the front runners, but the biggest change we're seeing, as an industry, is where the value is being added in value-added services. A generation ago, the contractor added value by taking a truck full of components, installing them, wiring them together, and ensuring that the customer was happy so they could receive the payment for this added value. As more of the products used in systems contracting have software based components, more of the value is being added as knowledge or intellectual property in the form of programming to make the system function. The infrastructure being used to add that knowledge is the software provided by the equipment manufacturer(s).

Still doesn't sound all that complicated? Here's a few questions to ponder. When the contractor's truck, emptied of components, leaves the owner's facility, precisely who owns the intellectual property consisting of the programming that makes the system work; the owner or the contractor? Who is ultimately responsible for the warranty on software and programming; is it part of the product manufacturer's warranty? Part of the contractor's system warranty? What if the programming problem is caused by a fundamental software bug? What if a software bug destroys hardware unrelated to the software/equipment manufacturer, such as an amplifier control system causing a condition that destroys loudspeakers? Does the owner of the system have a license to make use of, and benefit from, the software and the intellectual property that has been added? Is it a limited time renewable license or a one time license? Can the owner duplicate the software and programming for use in another location, on another identical system installed by a different contractor? Is the owner entitled to software upgrades as part of the system purchase, or are they on their own once the check clears? When the owner's MIS department wants to know where the documentation for the control system software is before you get your check, will you have an answer for them? Who resolves unresolvable conflicts between multiple pieces of system related software running on the system control computer, manufacturer A or manufacturer B? Who pays for that, the contractor, out of their slim profit margin, or the manufacturers out of their slim profit margin? How do you budget for the time it takes to sort out a system with multiple software components required to run it? How do you specify the ownership or licensing of software as part of a system specification? Which industry should we be looking to for a model to base our approach on? How do you specify the testing of complex software? Think of baggage handling at Denver airport. Do you accept the 1 in 100 chance catastrophic glitch as a hazard of using computers in the system? If your car failed as often as your office computer software, you'd have traded it in years ago, or had it listed in the Lemon Guide.

This is just the tip of the software issues iceberg. With some time and effort I'll bet you can add a few of your own. Some of these questions look like they should be easy to answer, but the more you chase the answers around the more they elude you. When you fully analyze the implications of some of these situations, it becomes obvious that these questions are not going to be at all straight-forward to answer, because those answers are going to cost somebody some money. Let's take a look at how this affects each of the players.

The manufacturers invest a sizable amount of capital into the research and development of hardware, and the related software to make the hardware run. They have market pressures on them to release a product early enough that the market knows they are competing, and hopefully late enough that any software or hardware problems will not produce major failures. In the computer software world, it appears that every piece of software you buy is a beta test or development copy (release v1.11645a) , as it is so difficult to test every feature under every operating condition to ensure that no bugs exist. Smaller markets make it harder to justify the investment in software, and the systems contracting industry has a small installed test base, and less capital available to work with than the computer software giants, so either the software has to remain fairly simple, or the complex software may not always be fully tested or developed by the time it ships. The manufacturers are also continually responding to market pressure to add features to match or beat their competitors, as well as working on requests or responses from their installed user base. Every new feature has the potential to add a bug in a stable section of code. In computer programming, perfection is actually defined as some sort of statistical likelihood of being perfect in the majority of circumstances and operating conditions.

The manufacturers' goal is to ensure that their hardware stays competitive with other hardware, and that some return on investment in both hardware and software can be seen before the code has to be completely redeveloped to keep up with the competition. The code will likely have a shorter life than the hardware. The software has to be simple enough that the contractors can program it themselves with minimal tech support, or the manufacturer can take the view that the software is part of the product setup cost and will charge for the programming. The value is being added in two different places in those two scenarios. Having software that is contractor or user programmable will cost more to develop, as it takes more documentation, or more interface development. Having software that is programmed for specific projects at the manufacturer's facility takes the value-added service out of the contractor's hands, and tends to limit the universal appeal and application of the product. In both cases, the owner of the equipment is able to make use of the value-added programming, but the question of ownership of the programming is still unresolved. Where the contractor does the programming, the manufacturer usually retains ownership of the machine code, or high level language that the software is built on, but gives away the value-added programming to the contractor, and the owner may, or may not, expect to own the value-added programming for the system. Where the manufacturer does the programming, both the contractor and owner may think they have purchased the intellectual property of the value-added programming, but they may both be wrong.

It is in this commercial market place environment that the manufacturer juggles the question of responsibility for software warranty, software support and intellectual property. The bottom line for them is one of profit, how much investment can they make in software support once the hardware associated with it has been sold and installed. This is the kind of problem that the Big Blue computer company faced before the rest of the market place ran away with the hardware and software profits.

The consultant's goal is to provide the most effective system design, using the most appropriate technology, ensuring that the maximum reliability is being provided, and to look out for the client's interests throughout the process. This includes the resolution of all ownership, function, performance and documentation issues through the writing of a detailed and comprehensive specification. At some point in the project, everyone has to agree that the system is finished and signed off, clearing the way for payment. As more software is added to the mix, it means consultants have to address new issues in specifications, and find a way to resolve the ownership or licensing of software, even when there may be several manufacturers involved, each with their own approach to software ownership. As the consultant is acting on behalf of the owner, it is important that the consultant know in advance what the status of the software and value-added property will be at the completion of the project, so that the consultant can inform the owner when all material has been delivered as required (and which items they should not be expecting). This definition of ownership or right-of-use has to be built in to the specification in some form.

The consultant must also provide an adequately detailed description of the software requirements to ensure that the contractors bidding on the project can estimate the effort required to program it, and for the owner's benefit, ensure that the final system configuration will be suitable. There are few compression algorithms that work well in writing specifications, so the amount of detail required in the specification to fully define and describe software operation would rival the amount of information in the software itself to implement the required functions. Describing software operation in extreme detail usually makes it harder to allow for alternate platforms or products in the specification.

The issue of performance testing of software is a relatively new one for systems consultants. It is one thing to test the system being run by software and have it demonstrated that it meets the audio or video specifications as appropriate. It is quite another thing to test the software itself and to ensure that it is fully compatible with the operating system and any other applications running at the same time. Software that is obviously buggy is not hard to spot, but it's the subtle bugs that can be problematic. The ones that are tough are where some peculiar operating condition of the control computer will flip a soft switch and leave it there till the computer is rebooted, locking out some specific function, or locking all out but one. These may not show up till after sign-off of the system, and in fact, may not show up often enough through the warranty period to be able to have it corrected before the system warranty has expired. If it were a hardware problem, the system would not have been accepted until an intermittent were fixed or replaced.

The consultant has to balance their knowledge of how complex these systems are to install and troubleshoot with their responsibility to their client, the owner, and attempt to produce a system design and specification that is achievable and will serve the client's best interests.

The systems contractor's goal is to cope with software that may be in a perpetual development state (we just got release v1.11645b), use it to program the control of a hardware system that is likely fairly straight-forward by comparison to the software, deliver and install it all on time, ensure that it is working correctly when the consultant and owner are present at substantial completion. In the process, they want to try to keep some of the money that has been flowing through their hands. In an optimum situation, there will be only one control system protocol to handle. But in the situations in the real world, it is likely that you may have a complete computer controlled DSP based product at the front end of the system, and it may have to talk to a different manufacturer's amplifier control system, and the user interface may actually have a third component layer in the form of an integrated media control package. All of these may have to talk to one or more pieces of audio or video equipment that have an RS-232 or similar connection that varies from the control protocol of other gear used in the system. The complexity of the software and programming increases much more quickly than the hardware interconnection complexity, as the number of variables you can pack into a given space is much higher.

In a complex system, the software debugging can take much longer than the hardware testing and setup. Fine tuning software can literally go on for months after the project is complete, either as bugs are found, or as the owner refines and redefines their uses and requirements as they use the system. The time it will take to complete an adequately complex system of hardware and software can be very difficult to predict. It is a real world example of the universal Turing computational machine, where you can't predict in advance how long it will take for the machine to complete the running of the program, you just have to run it till it finishes, and then you know exactly how long it will take. This makes it very difficult to bid on projects that require a fixed price for the system programming (which from a consultant's and owner's point of view is the only way to control the cost of the installation). Unfortunately, some systems seem to be non-deterministic (chaotic) where the installation is never quite finished and the service calls never really stop.

It is in this quirky computational environment that the systems contractor is faced with questions regarding responsibility for software warranty, and the ownership of the value-added programming. If the software is a bit buggy (now release v1.11645c), who pays for the additional labor to make the system work? This value-added intellectual property may be worth many times what the owner paid as part of the systems installation contract. Does the owner own the intellectual property, and is it their's to do with what they may? Does that include giving it to another contractor to install in another venue? From both the consultant's and contractor's point of view, it is important that the job have a high enough profit margin that the contractor is truly prepared to take ownership of the project and the cultivate the owner as a long term client. If the project drags on well past the point of profitability the contractor's interest certainly becomes focussed on the bottom line, and less so on the project.

From the user's perspective, they want a system that does what they need and does not cost a fortune to maintain or upgrade. In this respect, it is not far different from the purchase of a large computer system, where a combination of hardware and software are used to deliver both function and performance suitable to their needs. If you've been reading any of the recent stats that point to the high costs of maintaining a network of PC's versus a mainframe computer system, the analogy looks even better. The biggest difference between the computer industry and the systems contracting industry is that you don't have the option of switching from amplifier manufacturer A's control system to manufacturer B's control system because it has a better and more bullet-proof interface. To change software you have to change hardware. This was another problem that the Big Blue computer company had before they lost market share. With the new ubiquitous integration of software components into what was a primarily hardware-based systems contracting industry, there will be increased maintenance costs, and the owner is ultimately the one who will be paying the extra costs for those answers I alluded to earlier. (On the way, the other three players will absorb their fair share of increased costs).

The fact is that any adequately complex system is going to have additional maintenance costs associated with it. The problem we have as an industry is that we don't know what the magnitude of those costs are going to be for any of the four parties described here. The manufacturers don't really know how much money it will take to stay in the ring developing hardware and software, or how much return they will see before some young gun company takes a shot at them. The consultants don't really know what the long term costs of software based systems will be. The contractors don't know what the real costs of installing complex software based systems will be till they are done. The users don't know what the long term costs of owning and operating complex software based systems will be anymore than the companies did when they invested in computer networks. Once you add in the question of ownership of value-added programming, and the responsibilities for maintenance of that code and programming, you are talking about the most volatile part of the profit margin equation for all parties. With that in mind you might want to have another look at the questions I asked earlier, and see how easy they are to answer now.