The general saying around open source quality has traditionally been to quore open source guru Eric Raymond ⁽²⁾: “Given enough eyeballs, all bugs are shallow”. The principle behind this is that publicly available code for popular and mature open source products tends to be reviewed by hundreds of developers, far more than most commercial software companies could afford to test their own products. Let’s face it, it makes perfect sense. But can you make procurement decisions based on the gut-feel of industry big-wigs? Of course not. So where is the evidence to prove it? Handily, there are reams of academic research projects that draw that very same conclusion. We take a look at some of those results here.

Open source quality is as good as, if not better than, comparable commercial software

Open source development generally eschews many of the procedures that we normally take as software development best practice without software quality suffering. For example, in open source projects you will be unlikely to find detailed planning and scheduling efforts, deadline-driven milestones and documented risk assessments, but as a study of 200 OSS projects by Elbaum and Zhao ⁽³⁾ shows, what you will find is a rapid, iterative release process that results in continual improvement by large numbers of developers contributing iterations, enhancements and corrections.  Another study of 100 open source applications by Angelis et al ⁽⁴⁾ found that structural code quality was higher than expected and comparable with commercially developed software, and the more well-known projects like Apache and the Linux kernel actually showed a substantially lower defect density than in comparable commercial products.

How is OSS quality achieved and how can you measure it?

So we have good data to show that OSS is of high quality, and in some cases of even higher quality than comparable commercial software. But how does this happen and what are the keys to success? Or more to the point for many of our readers, how can you tell if the OSS you are considering reaches these same standards? Published research currently points to the following key areas that result in high quality open source software, we will look at each of these in turn.

Sustainable communities

Open source product communities commonly consist of core developers, peripheral developers, bug reporters and end users. This model, featuring successive layers of user types in ever-increasing numbers, is called the ‘onion’ model. A small core team maintains control and modularity and will do about 80% of the core coding according to Fielding et al ⁽⁶⁾. A much larger number of peripheral developers will add new functionality, fix bugs and peer review code, leading to fast innovation. Volunteer testers take on the bug reporting role and have a major effect in reducing bugs. Through a process of meritocracy users tend to move through the community towards the core. A large and sustainable community will develop code rapidly and debug code effectively, and should be an OSS project’s key objective. Angelis et al ⁽⁴⁾, Raymond ⁽²⁾, O’Reilly ⁽⁷⁾ and Nakakoji et al ⁽⁸⁾ all provide good reading and research on this topic.

Well documented and modular code

Code modularity supported by good documentation allows programmers to extend the product by working on separate modules, without needing to change or understand the core system, or interfere with each other’s progress. This reduces the risk of new bugs being introduced by product extensions and hastens product innovation. Research on a wide range of open source projects including those by Fielding et al ⁽⁹⁾, Cole and Lee ⁽¹⁰⁾, Moon et al ⁽¹¹⁾ and Bollinger et al ⁽¹²⁾ has established clear correlations between high code modularity and low defect density.

QA process

In commercial software development, procedural QA rigour is crucial in achieving high quality. In open source development, a variety of techniques are used to achieve the same. Peer review is commonly cited as the key reason for open source quality. Clearly it’s just one part of the picture, but an important part indeed. Effective peer review is facilitated by the rapid release cycles that typify open source development. Raymond ⁽²⁾ suggests that implementing and responding quickly to peer reviewers’ comments and code keeps them involved and interested, resulting in a product that grows and extends rapidly and reaches high quality quickly. As Gacek and Lawrie ⁽¹³⁾ note, it’s not an efficient model should a commercial organisation take the same approach with similar numbers of reviewers, but it works very well as a continuous inspection process in conjunction with rapid release cycles, in the absence of commercial pressures and interests. Dinkelacker and Fuggetta are also worthy reads on this topic and support these conclusions.

Elbaum and Zhao ⁽³⁾ found that test plans, testing tools and structured QA methods are not widely used in open source projects. Indeed, Fielding et al ⁽⁶⁾ found that the community performs the bulk of the testing, on a much wider range of test environments that any commercial software company could afford. Automated build and regression testing processes are becoming more widespread also, but the methodology used largely depends on the available expertise and resources. The user base is often the only choice.

Code reuse

It also seems that code re-use plays an important part in the lower defect density found in open source products. Conradi et al ⁽¹⁴⁾ found that code originated from reuse had 50% less defects than code written from scratch. Many open source products rely heavily on pre-existing open source components and code libraries rather than write features and functions from scratch. These pre-existing components have already been debugged and are proven in real-world use, and it is the most popular of these that get reused regularly. As Kit Plummer from Accenture notes, it’s a “Darwinian cycle where the best implementations rise to the top and weak simply don’t survive”.

So what next?

The above information should give you peace of mind and an idea of what to look for in an open source project you’re thinking of implementing. There are a few evaluation models you can also use to help provide an evaluation framework.

Navica’s Open Source Maturity Model is a formal process, published under an open license, that assesses the maturity of the key elements. The Business Readiness Rating is a proposed evaluation model under development by a group of major organizations that seeks to expand on both previous OSMM models to develop an open standard for scientific OSS modeling and rating.

Further reading

1. M. Aberdour, "Achieving quality in open source software", IEEE Software, vol. 24, no. 1, 2007, pp. 58–64.

2. E. Raymond, The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary, O’Reilly, 2001.

3. L. Zhao and S. Elbaum, “A Survey on Quality Related Activities in Open Source ”, Software Eng. Notes, vol. 25, no. 3, 2000, pp. 54–57.

4. L. Angelis et al., “Code Quality Analysis in Open Source Software Development ”, Information Systems J., vol. 12, no. 1, 2002, pp. 43–60.

5. Succi, Paulson and Eberlein, "An Empirical Study of Open-Source and Closed-Source Software Products ", IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, V.30/4, april 2004

6. Mockus, R. Fielding, and J. Herbsleb, “A Case Study of Open Source Software Development: The Apache Server”, Proc. 22nd Int’l Conf. Software Eng. (ICSE 00), IEEE CS Press, 2000, pp. 263–272.

7. T. O’Reilly, “Lessons from Open-Source Software Development”, Comm. ACM, vol. 42, no. 4, 1999, pp. 32–37.

8. K. Nakakoji et al., “Evolution Patterns of Open-Source Software Systems and Communities” Proc. Int’l Workshop Principles of Software Evolution, ACM Press, 2002, pp. 76–85.

9. Mockus, R. Fielding, and J. Herbsleb, “Two Case Studies of Open Source Software Development: Apache and Mozilla”, ACM Trans. Software Eng. and Methodology (TOSEM), vol. 11, no. 3, 2002, pp. 309–346.

10. G. Lee and R. Cole, “The Linux Kernel Development as a Model of Open Source Knowledge Creation”, Haas School of Business, Univ. of California, Berkeley, 2000.

11. J.Y. Moon et al., “Essence of Distributed Work: The Case of the Linux Kernel”, First Monday, vol. 5, no. 11.

12. T. Bollinger et al., “Open Source Methods: Peering through the Clutter”, IEEE Software, vol. 16, no. 4, 1999, pp. 8–11.

13. Gacek and T. Lawrie, “Issues of Dependability in Open Source Software Development ”, ACM SIGSOFT  Software Eng. Notes, vol. 27, no. 3, 2002, pp. 34–37.

14. Mohagheghi, Conradi, Killi and Schwarz, “An Empirical Study of Software Reuse vs. Defect-Density and Stability”, Proceedings of the 26th International Conference on Software Engineering, 2004, pp. 282 - 292