In the past few weeks I have started work on a major software development effort. It is a very complex bit of software for capturing images from high-performance cameras and reducing them mathematically to functions used in high-end digital rendering. The process is very complicated mathematically and computationally intense, in some cases requiring as much as eight hours of processing.

One of the first questions I wrestled with was which language to use for the development. I was surprised at the answer.

I started my career in IT more than 30 years ago developing mathematical code for scientific applications, and as I sat down to make the decision I felt like I was in a time warp. When I was growing up, savings accounts were common. You made small deposits which were recorded in ink in a small paper book about the size of passport. Over time the pass book took on a comfortable worn look that was a source of pride, reminding you of your thrift, as my worn passport reminds me fondly of my travels. Once a quarter the bank totaled your balance and added a bit of interest. The rates were terrible, — 2 or 3 percent at best, but were made worse because of the infrequent compounding. One reason banks didn’t compound interest often was that the market didn’t demand it, but another reason was that it wasn’t easy to do. The rudimentary computing equipment and software of the time meant the interest was literally added to an account with an “interest??? program that took hours to run for all of the bank’s many accounts.

The biggest bank in town was innovative and moved quickly to automate with Burroughs banking equipment, and with their added computing power began to offer some impressive new services. The early investment in high-end computers paid off, and the bank eventually grew into a major service provider to other banks, and provided me with my first professional IT job. The immediate application of the technology, though, was to provide current (daily) account information to multiple branches in the form of huge, green-bar reports, to print deposits in the bank books rather than writing by hand (wow, did that look high-tech) and, my favorite part, to compound interest every month.

They advertised this widely, with a cheesy jingle typical of the era. My father was good friends with the executives of the competing bank (which was literally across the street) and they took the escalation seriously. They responded by buying the latest and greatest from IBM (a 1620) and after a few intense months, proudly announced weekly compounding. The war escalated, and Burroughs crowd announced daily compounding. The IBM camp then launched the nuclear option – continuous compounding. The executives from the Burroughs bank stepped up and said “Oh, yeah, we can do that too???.

Oops – there’s a problem. Continuous compounding is done in a completely different way from periodic compounding. In the old days you ran an interest program – once a quarter, month, or whatever. As the interest rate war progressed the banks just bought newer, bigger, faster computers so that they could run the program more quickly. With continuous compounding, however, you only calculated the interest when there was an event – a deposit, withdrawal or statement, and then you had to do exponentiation. Unfortunately, the Burroughs COBOL of the day did not have a built-in function for exponentiation.

That omission got me my first professional gig. The bank came to the local college and asked for someone who could write code to implement this one function. I got the task and dispatched it in short order. I wrote it in Burroughs Algorithmic Language (which hasn’t been seen in the wild since 1977 and is presumed extinct).

So how have things changed? My recent experience suggests that things might not have changed as much as I thought. First, in recruiting for coders, I found that there were a huge number of people who were skilled at building web and data applications, but the folks who know how to make machines do math fast are rare birds, indeed, just as it was 35 years ago.

This didn’t surprise me too much, as the market for business applications is vastly larger than the market for scientific applications. What did surprise me, was the way the old languages are still competitive. The test I devised was to write an equation solver (using the Newton Raphson method) in several languages and trying several different compilers. I constructed a problem where the root was exactly pi, so I could check the answer against published values to very high precision.

I tested Fortran, C, C#, and J# . The results were completely clear – the object oriented languages were crushed. The simple old procedural languages really are faster for the mathematical heavy lifting. It was close between the two C compliers I tried and the three Fortran’s, but the Intel Fortran compiler seems to be king of the hill. It was fast, and the results were spot on to 32 decimal places.

It may not be sexy or look very modern, but if you have to do serious math on 16 mega pixel images, plain old Fortran is still the way to go. Certainly the vast bulk of the new company’s code is going to be written in something more current – their web site, e-commerce site, and internal system, for example, but the hard core, the nut that will make or break the company is going to be in Fortran. Now I’m back where my bank employers were in 1970 – looking for some math stud at the local college.

Technorati Tags: software security, banking, Fortran

In the course of my career I have been involved in a fair number of startups. I’ve had pretty good luck, and most of them have been successful. One, however, was a complete failure. I refer to that experience as my DIY MBA. You can learn more from failure than you can from success. It is very difficult to determine what made something succeed (apart of course from our genius, hard work, and moral virtue), but if we look at something long enough and with whatever objectivity we can muster, we can usually find a root cause for failure. If we’re smart, we won’t that make that particular mistake again.

One of my favorite books is Engineers of Dreams: Great Bridge Builders and the Spanning of America by Henry Petroski, the wonderful writer on engineering. It is a book about extraordinary success – the construction of the great bridges of America, but it is as much about failure as success. Bridges fall down throughout the book, but each failure shines light on another aspect of bridge design and the limits of the materials. The heroes (the great engineers) learn from their experience and continually build better and longer bridges. What I took from this book, beyond an appreciation of the bridge builder’s art, science, and mastery of the political (all big projects involve politics) is that that failure is something to be treasured once you get past the pain.

My one entrepreneurial failure was in a company that did desktop publishing around 1980. My partners and I bought two NBI 3000’s, very early, very expensive word processors. Businesses in and around Boulder, CO, some students and professors came to us with hand written drafts which we turned into beautiful printed documents. The machines were complicated, temperamental, and difficult to use, though they defined the state-of-the art at the time. Only trained operators could manage them. Word Processing was a task for pros.

Of course the company failed. Things went well for a year or so, but then the first practical PCs and word processing software came out. People began to do their own word processing, even on crude, small, expensive machines. When the IBM PC came out and legitimized PCs, everyone started writing on computers and doing their own desktop publishing. The stream of customers dried up and we closed the doors.

My first thought about Document Control (the company) was that our technology was simply made obsolete — that we were selling slide rules in the age of calculators. I think though, that we were really swept aside by a more fundamental imperative – the human urge to do things themselves. In the Pharaohs days literacy was regarded as something for specialists, and the leading classes hired scribes to perform that difficult task. When the telephone was invented we needed operators; when cars were invented they were often driven by mechanics. Over time in each case the difficult became simple and the rare became commonplace. For the things that count, people would rather do something themselves than have it done for them if it is within their ability and comfort zone.

This principle pertains as much in IT as in other arenas. IT was once entirely the province of the professionals operating in glass houses, who accepted data on cards from the acolytes and returned them manna in the form of green bar. It was inconceivable in those days that computing would become the province of the everyman but, of course, it has.

I am not referring to people using personal computers to access the Internet, using email, and writing newsletters for the PTA. That is really too obvious for comment. What I am referring to is the continuing trend in all sorts of organizations to empower the individual. Individuals prefer simple hosted applications like Saleforce.com to complex CRM from central IT and they use desktop tools like the Microsoft Office Suite to build very complex applications. The things they build are not tightly integrated like the applications built by professionals. Processes may still involve many manual steps that a pro could program in a few hours (after a week of committee and budgeting meetings, another week of design review, and two weeks or so of testing). We pros can do it better – complete automation, all sorts of bells and whistles the rubes would never think of, audit trails, better security, and all that good stuff, but the users prefer the stuff they build (or discover) themselves. It does what they want and they understand it. More than that, it empowers them. They feel in control. In the end, DIY always wins.

Going into word processing was not a mistake. The machines were great and they did things that simply weren’t previously possible. The mistake was in persisting even after the first personal computer showed up. The was on the wall, and it read that in the end, DIY always wins.

In building our enterprise applications we must be cognizant of this same imperative towards DIY. There will always be central IT applications for things like basic accounting – accounts payable, accounts receivable, etc., but organizations are dynamic – buying businesses and being bought in turn, reorganizing, opening and closing business lines, launching new products and dropping others, doing studies and projects. The central IT department is always running behind but the DIY community filling the gap with their jury-rigged lash ups. Giving the pros (i.e. us) a break, though, real IT is hard.

I know that this informal IT – the IT that takes place outside the purview of the IT organization – is already and important part of every business. I suspect that in many organizations these home grown applications may actually be as or more important than the official stuff. My wife who works in the accounting business, for example, is tracking the company’s backlog and progress in an Excel spreadsheet as the tax deadline approaches even though her company uses the best accounting software in the business.

As IT tools have improved, everyone has become a practitioner to some degree, just as our ancestors learned to drive cars and make calls and our ancient ancestors learned the arcane art of reading. A challenge for us as professionals is to give the non-professionals the tools they want and need so that they can define how they do their work, and then work with them, not against them, to improve these informal processes and bring them up to “professional grade.” DIY is a powerful imperative and much of the IT that we now regard as the realm of the professional will inevitably move into the hands of users. We should not resist this, but enable it. I would love to see what users could do if they could have a real-time access, with Excel as a front end, to a company’s core data in real time. What would they build? I know that they would build better tools for their personal work, but they might just go beyond that to provide new insights into the way the company operates and models for how the company’s processes can be improved.

Technorati Tags: diy, it

Did you ever catch a box turtle when you were a kid? They are beautify, affable and interesting little fellows. If you see one, catching them is no problem. You just walk over and pick it up. Even though it has a razor sharp jaw, its instinct is not to bite, but to pull in it head and legs and hope for the best. In a little while, after it realized you are not some bad actor, it comes out of its shell and begins the slow, deliberate process of escape. By my experience, turtles are pretty patient. If there is a tasty bit of lettuce along the route, they are content to stop and eat if, even if you sitting a foot away. Still, when the lettuce is done, they will continue their escape AND THEY WILL SUCCEED.

Their astounding rate of success has always amazed me. Without fail, I as a child, and each of my children in their day would put a turtle down somewhere in the yard and then go about some game or chore. We would glance back at the turtle every few minutes noting its slow progress to the fence or bushes. When the turtle got close, we would haul it back to the center. Then, patiently, it would resume its long walk. We would catch them a few times, repeating the cycle, but finally we would turn around the turtle was gone.

There are two possible explanations for this. The one I would like to believe is that turtles are wonderfully canny creatures and fleet afoot, who consciously lull us into a false sense of security before they dash to the perimeter with lightning quickness. I would like to believe that because I rather like turtles and would quite enjoy the thought that they are clever as well a cute. The more likely explanation is that it is simply impossible for us to pay sufficient attention to events that unfold so slowly. Turtles almost live in an alternate universe, with a wholly different time constant.

I like writing about turtles, but this is an IT blog, so let me extend the parable to something related to computing. My discourse goes through San Francisco. San Francisco has a great rail system – Bay Area Rapid Transit or BART which is highly automated. Back in the later 1970s they had an interesting accident. BART trains are largely computer controlled, with human operators monitoring for safety. Trip after trip, day after day, year after year, the computers would sense that the train was reaching the end of the route, slow the train to a stop, and start off on the return trip. The operator would walk to the opposite end of the train to once again be at the front. The system worked very reliably, so the operator grew comfortable relying on the computer, and would start his trek through the train before it had actually come to stop. You can see where this is going – one day the train missed the end of line single and didn’t stop, and the operator, who could have saved the day, was not at the controls.

There is a common lesson in the cases of the turtle and the BART train. Human beings cannot catch rare events. We simply cannot maintain the concentration. It is bad design to build a system where the human backs up the computer. The human WILL be day dreaming when things go wrong. It is better design to keep the human active and in the loop, literally running things or at least sharing responsibility, even when the computer can probably do it better. When the human fails (and he will) the computer will be there to back him up. A human and computer together can provide system redundancy, but only if the human is at the forefront and the computer as the backup.

Clearly we shouldn’t get carried away with this principle. We don’t want an accounting system where the math is done by hand with the computer checking the addition. In design of real time systems, though, this is a very important principle. It surfaced last week in a trading system I was reviewing. The system is used for large block trading (BIG deals). In my tests I found that you can enter an asking price that makes no sense (transposed digits or a shifted decimal point, for example). As I said, these are big deals, so you have to confirm the price before the deal closes, but you confirm not by entering the price twice, but by simply clicking “confirm.” In the heat of an active trading day, the traders who are very confident individuals will certainly hit confirm automatically, without actually reading the data. Ninety-nine percent of the time, this is not a problem.

Lest you think this sort of design problem is rare, you can trace the wreck of the Exxon Valdez and the navigational error that led to the shooting down of Korean Air Lines Flight 007 over the Kamchatka Peninsula to precisely the same human factors problem – using people to back up computers, not the other way around. The key message here is that when we design and test a system, we must focus on the whole system performs people included, not just the software.

I was part of a panel at a university recently speaking to prospective computer science students. The panel members were from industry — a few of the biggest companies and few smaller ones. We each had ten minutes to speak before QA, so I stripped my talk down to two simple points. The first point was that IT is fun. We truly build amazing things. I’ve been in the business 35 years, and I have never been bored. Each day offers a new challenge. The other point was that there will be plenty of jobs for decades to come just trying to get things right. Software sucks. No product on the market has more defects than software and no projects have a failure rate close to that of IT development work.

The next generation can live off the argument that they can certainly do better than the fogies doing the work now. The challenge for them, as it was for us we made the same claim, is living up to the bold promises. I have been through five distinct new approaches to building software, each designed to deliver better products and better delivery management. Each has moved the ball forward a bit, but we still have a LONG way to go.

Just how bad are we? There are many studies of failure in IT, but the results are all over the place. The most optimistic estimate I can find says that about 30% of projects fail. When I mention that number, a fair number of people tell me it is too low, but let’s grant ourselves the benefit of the doubt. We fail 30% of the time.

How does that compare to other industries? Before we take on the masters of quality lets start with an easier opponent – how about used car salesmen? Every year in United States, about 45 million used cars are sold. The Feds estimate that the odometers of about 450,000 of these have been tampered with, and that in another 500,000 cases the seller fails to disclose a serious known defect. That works out to a failure rate around 2%.

We in the IT business are 15 times worse than used-car salesmen. Yet, the demand for our products and services is tremendous. IT related professions dominate the top-20 positions on the US Department of Labor’s list of high-growth jobs. Why is this so, if our products are so bad?

The answer is that software, bugs and all, delivers real value. We would love our programs to work without error, to never crash, to never need patching, but even with these sorts of problems, software does amazing things. Once in awhile we all encounter a program that makes us go wow. Remember the first time you used online mapping? That was a wow, but we quickly get bored. The amazing becomes commonplace — part of the wallpaper. That’s sad. I wish it was possible to flip a switch, and show ourselves and the world juts how far we’ve come. It would be a cruder, poorer, and much more boring world without us.

That said, software is still crap – amazing crap so customers will continue buying, but crap nonetheless and we owe them better. The challenge of making software better is a difficult one. All of the Justice League blog entries are, at root, about this one subject. Sometimes we write about broad issues other times about the very specific, but our objective is always to make software better. Better functionality, more reliable, and more secure.

This blog, rather than being about solutions, is about the intriguing, challenging, and sometimes frustrating dichotomy of software, it is both amazing, with a capacity to dazzle and inspire while it is also mundane – just a bit of bug ridden code. The Roman God Janus has two faces – one facing forward, one backward.

Building software is much the same. We constantly envision the future and it drives us to imagine and to realize that imagination we build ever more complicated systems. There is the rub. The other face of Janus looks back to the past. In IT that is the hard job of not making the software, but making it good. Janus was the god of doorways, a wonderful image for IT. The doorway to the future in IT rests on equal measure on vision and quality, looking forward to what can be done and backwards at what we have. We must aspire to be Janus.

A final observation that will set the stage for a future blog entry. Our troubles lie in the imbalance between the forward looking face and the backward. There is so much potential in software that we are constantly reaching farther; farther in some cases than we can deliver. It is the reaching that gets us unto trouble, but it also the reaching that keeps us energized and fresh, and once in while makes our customers say “Wow.”

Cigital is in the business of making software secure, often by telling our clients precisely how and why their software is not secure. There are an almost infinite number of ways to be vulnerable so it should be no surprise that we rarely find the perfect system. I’m tempted to say never, but I’d have to check around on that. We have a saying in the office that the only truly secure system is one that is buried underground with no wires in or out and no users.

Some of our clients, though, are shocked and more than a little annoyed when our assessment is critical. A top level manager may express gratitude when we confirm his gut feeling, but the middle manager with direct responsibility for the system is often embarrassed and defensive, and the guys in the trenches are downright pissed. That’s too bad. We aren’t top level managers. We are trench warriors ourselves. We truly appreciate how very difficult it is to write excellent code—code that does what it is supposed to do, code that works fast, code that works reliably in a production environment, code that is maintainable and code that is secure.

I once lead a development team that produced some code that managed some highly visible and regulated financial transactions. The code had to be 100% free of errors as there were many critics with deep pockets, political connections, and dozens of lawyers who wanted it to fail. The client wisely hired an independent (end excellent) team to perform extensive IV&V at the test level and the code level. The team showed up in my office where I gave them all the documentation, all of the test results and test code, and unfettered access to my pre-production staging area. I figured they were just code shmoes like me, just doing their job, so I tried to be nice. Their attitude though was that of an IRS auditor on the trail of a bootlegger. One day I walked in and gave them a couple of boxes of Girl Scout cookies from my troop (Thin Mints, I believe). They turned down the cookies and told me that it was completely inappropriate of me to offer them anything of that sort. I lost it. I screamed “They’re not bribes, they’re goddamned cookies. Eat the ——- cookies!” They were so shocked they all instantly grabbed one and ate it very quickly. It was hilarious, and we eventually had a laugh over it.

At Cigital we don’t want to be like those fellows. We love smart coders—they are our kind of guys. Let the guys in sales play golf with the bosses. We would rather drink beer (or Mountain Dew) with the folks on the midnight pizza shift. At root though, we do review other peoples’ work, and sometimes the auditor mentality takes over. One of the best guys at Cigital recently took on an audit gig. He told his manager that he wanted to keep the report secret until the end of the project, so he could produce a bug-rich audit report. I suppose the idea was to show how smart he is and we are. His sage manager told him to cut it out and share the findings as they emerged. The result was that we produced a long bug list, but noted in the final report that almost everything had already been addressed. That’s a good outcome for our client and a good one for us—we got another gig.

Writing good code is real hard, and the smart guys who do it are the heroes of our business. We love you guys, but we sometimes do have tell your that your baby is ugly and we’ll go beyond that to describe every deformity, wart and blemish in lurid detail. Understand, though, that we, like you, want that ugly baby to grown up into a runway model.

Technorati Tags: governance, regulation, software security

I am strong advocate of service oriented architectures. I have seen them work—not in theory, not in demonstration, but in real, deployed, mission critical applications. In sitting down to write this blog entry I was going to do yet another “SOA - Hype vs. Reality” essay. I’ve written a lot of these over the past four or five years, and, frankly, theses pieces are getting boring to write and boring to read. Its just not a burning issue — the next generation architecture debate is over, and SOA won.

Sure, most of what you see when you walk into a big IT shop is still monolithic applications, but those are echoes of IT history—dead men walking. SOA won because it is a natural evolution and logical extension of the way we have been building systems since Grace Hopper showed us the light. Our first programs looked monolithic, but only on the surface. We learned that good practice requires us to decompose our programs into procedures, subroutines, (or whatnot depending on the language) that had limited functionality with well-defined interfaces—nothing too large to fit in the space between our ears. Code reuse as the Holy Grail.

SOA is just the extension of good programming practice except that high bandwidth, reliable, standards-based networks and emergent standards for things like message transport, service registration and service invocation mean that our systems need not operate in a single memory space. The acronym SOA may not survive. Software sales dudes always come up with new names to sell a polished up version of the same-old, same-old, but service-oriented architecture and, more importantly, service-oriented architecting is here to stay; its centrality in future systems is as inevitable as the sun rising in the east.

With that rant out of the way, what I want to say here is that there are unique security issues related to security. I plan to discuss these in forthcoming blog entries, but I have a meeting today with the Ettienne Reinecke, CTO of Dimension Data, one of the best networking companies in the business. I have been thinking hard about how the guys like him play in SOA beyond giving us reliable pipes. Let me share my thoughts.

Networks work really well—software not so well. Why? I suggest that the success of the networks lies in three factors—componentization, standardization and that management. Networks are not monolithic. They are composed of lots of individual pieces, each with a well-defined function (sound like SOA?). The performance and interfaces of each component type are very well-defined by standards. This means that different components can be integrated with moderate ease and some reasonable expectation that they will work together. It also means that the people who develop the components can continue to refine their designs—to polish them until they are sparkle, making each generation, faster, better and cheaper—as long as they keep the interface constant.

Finally, the network guys assume that the network isn’t going to work. They assume that components will break, so they instrument the network to monitor its performance and they watch it carefully. Good network engineers also design for graceful degradation—the ability to lose performance in increments rather than catastrophically (think escalators vs. elevators). We on the software side can learn from this. The presumption of fallibility and the capacity for graceful degradation are alien concepts in the software world. They shouldn’t be.

In moving to SOA, we are componentizing. In adopting web-service standards we are standardizing, though our standards are much less mature than those for networks. Good for us—we have to two of the lessons down—componentization and standardization.

Let’s take the final step, and build our SOAs with the assumption that they will fail (a safe assumption) and engineer-in monitoring and the capacity for graceful degradation from the beginning. We can “ping” network components to see if they are alive, query their state, test them and change their state during operation. We should be doing this for SOA services. Ultimately, system operators should be able to look at panels like those now used for network operations, showing in color codes the status of each service. They should be able to detect problems and fix them on the fly.

Essentially, as we move towards SOA we are shifting from thinking of our systems as discrete entities to thinking of a computing ecosystem that runs 24 hours a day, 365.2564 days per year (more or less). This model is completely unnecessary for a simple program that does one thing, and only runs once in a while, but these are becoming an endangered species.

The trend in IT has always been towards greater integration. The data from one program feeds into the next, and the next…and the next. We can think of a data store as the DMZ between two applications or we can think of it as a cache in a larger system. The boundaries between systems are often arbitrary and are often simple reflections of org-chart boundaries rather than computational or business logic. At the level of the enterprise, we are evolving towards a more comprehensive view, beginning with simple shared services like database and directory, but moving, inexorably and inevitably towards shared functionality.

The network engineers have already made this transition. We should lean from them and relentlessly componentize, standardize and manage.

Technorati Tags: software security