Sin-Yaw @ Juniper

I found myself preaching. Maybe I do it all the time, but rarely I was self-aware of that fact. I must have delivered the same sermon enough times already.

What separate professional software development and amateur? The ratio of effort devoted on quality. Amateurs make it work. Professional show evidences. The amount of efforts in professional software houses devoted to quality and verification easily dwarfs standard definition of development effort.

The verification takes place in phases. First is the effort to make sure the software works as designed. This is most commonly done by the same person who designs the software, but more effective if by someone who sits close to him/her and watches over his/her shoulders as he/she codes. The goal is to make sure the software behaves well for all ranges of inputs.

Next is the effort to verify that software meets the requirements. Most likely, someone who understands the requirements will come up with methods to verify such, without giving too much concerns on how the software was designed.

After these two steps, the developer confidently declares his/her software works. He/she now must prove the new does not break the old. This is simple enough. Just run all those previously developed tests in the first 2 steps.

Imagine many of such teams following the same process. When they all come together, they do not work together. There could be misunderstanding of the interfaces or inadvertent changes that affect other parties. This is why we need integration tests.

At the end, companies frequently will “try run” on their best customers’ environments. Sometimes, the best effort “in the labs” does not discover problems “in the real.” This is the attempt to be as “real” as possible. It also ensure the satisfaction of the best customers.

Some requirements are hard to verify. Specifically, performance requirements demand benchmarking, reliability needs stress testing, security needs attacks. None of these come “naturally,” and software houses frequently use simulators. They create un-natural but concentrated faults to put the software under duress. The most common simulator is for new hardware platforms that are still in development.

This whole system is under the supervision of a general test harness that automates everything, supplemented with as little manual tasks as possible, for maximum execution speed. This harness handles test sequencing, failure management, metrics of progress, report generation, and even provisioning of resources.

Trackback URI | Comments RSS