Telecom Signaling Industry News

TMCNet:  Updated NIST Software Uses Combination Testing to Catch Bugs Fast and Easy

[November 10, 2010]

Updated NIST Software Uses Combination Testing to Catch Bugs Fast and Easy

(Targeted News Service Via Acquire Media NewsEdge) WASHINGTON, Nov. 9 -- The U.S. Department of Commerce's National Institute of Standards and Technology issued the following news release: Researchers at the National Institute of Standards and Technology (NIST) have released an updated version of a computer system testing tool that can cut costs by more efficiently finding flaws. A tutorial on using the tool accompanies the new release.

Catching software "bugs" before a program is released enhances computer security because hackers often exploit these flaws to introduce malware, including viruses, to disrupt or take control of computer systems. But it's difficult. A widely cited 2002 study prepared for NIST reported that even though 50 percent of software development budgets go to testing, flaws in software still cost the U.S. economy $59.5 billion annually.

Exhaustive checking of all possible combinations of input actions that could cause software failure is not practical, explained NIST's Raghu Kacker, because of the huge number of possibilities, but it's also not necessary. Based on studies of software crashes in applications, including medical devices and Web browsers, NIST's Rick Kuhn and other researchers determined that between 70 and 95 percent of software failures are triggered by only two variables interacting and practically 100 percent of software failures are triggered by no more than six. "Testing every combination up to six variables can be as good as exhaustive testing," said Kacker.

Working with researcher Jeff Yu Lei and his students from the University of Texas at Arlington, NIST designed Advanced Combinatorial Testing System (ACTS), a freely distributed software tool to generate plans for efficiently testing combinations of two to six interacting variables. The method goes beyond the commonly used "pairwise" approach to software testing, which tests combinations of two variables, so it can detect more obscure flaws. (See "'Combinatorial' Approach Squashes Software Bugs Faster, Cheaper" in NIST Tech Beat, Dec. 12, 2007, at Kuhn describes the process "as packing as many combinations into a set of tests as efficiently as we know how." For example, testing all possible interactions for a product with 34 on and off switches would require 17 billion tests. Using ACTS, all three-way interactions can be evaluated using only 33 tests and all six-way combinations with just 522 tests, instead of 17 billion.

The first version of ACTS was released in 2008. Since then, it has been distributed at no cost to 465 organizations and individuals in industry, academia and government. "About half of our users are in IT, but other heavy users are in the financial, defense and telecommunications sectors," said Kuhn. In August, NIST and Lockheed Martin initiated a Cooperative Research and Development Agreement to study the application of ACTS in the company's large and complex software applications. The two groups will jointly publish the results.

NIST released the latest update of ACTS in October. The new version includes an improved user interface and a better method of specifying relationships between parameters for testing. This can eliminate the problem, for example, of spending time on tests for invalid combinations, such as using Internet Explorer on a Linux system. Information for requesting ACTS is available at

Just released is a new tutorial, Practical Combinatorial Testing, that introduces key concepts and methods along with explaining the use of software tools for generating combinatorial tests. Cost and other practical considerations are addressed. The tutorial is designed to be accessible to undergraduate students in computer science or engineering and includes extensive references. NIST Special Publication 800-142 (link to can be downloaded at

TNS PA-PA 101110-3099524 61PoojaAnand (c) 2010 Targeted News Service

[ Back To Telecom Signaling's Homepage ]

Featured Events

5G NA Signaling Day

November 14-16, 2016
Sheraton Dallas, Dallas, TX

Featured Webcasts

Securing The Signaling Interconnect: Oracle's Perspective On Recent Security Events

Today's telecommunications landscape is changing rapidly, and the "institution" of trust between networks that has been assumed over the decades is no longer relevant. We are now hearing of vulnerabilities in CSP interconnection points that have been known ...

Applications of Wi-Fi Calling

With all of the discussion about faster data access and richer content, it could be assumed that voice services no longer play an important role in the business model of the communications service provider. Join ReThink Technologies Research and Oracle Communications as we explore ...

Featured Whitepapers

Oracle Communications Diameter Signaling Router Main Differentiators

Diameter is the protocol used by network elements in LTE and 3G networks to enable and monetize services, such as voice, video and data. Diameter enables revenue-generating data services; including tiered data plans, loyalty programs, application specific QoS, content provider and Internet of Things (IoT) solutions ...

Multi-Layer Security Protection for Signaling Networks

The sanctity of mobile operators' networks and brands will depend greatly on their ability to deliver QoS guarantees to roaming and interconnect partners, while simultaneously protecting increasingly multimedia-savvy and socially connected ...

Featured Datasheets

Oracle Communications Diameter Signaling Router

Centralizing Diameter routing with cloud deployable Oracle Communications Diameter Signaling Router creates a secure signaling architecture that reduces the cost and complexity of the core network and enables elastic growth, interoperability and rapid introduction ...

Oracle Communications Mobile Security Gateway

The Oracle Communications Mobile Security Gateway is a high performance gateway that allows the Communications Service Provider (CSP) to cost effectively expand network coverage and increase capacity by incorporating Heterogeneous ...

Oracle Communications Evolved Communications Application Server

As service providers drive their networks toward an all-IP and virtualized state, they require the means to design and deliver compelling high definition voice, video and multimedia offers via Voice over LTE (VoLTE) and Voice over WiFi (VoWiFi) ...

Featured Infographic