Soup to Nuts Testing

Quick Summary: Product testing encompasses a broad spectrum of activities that need to be carefully considered.

Abstract:

Testing of the capability, reliability, and performance of every new product is a fundamental requirement that everyone accepts. However, what tests need to be performed, by whom and when can be hotly debated. Naturally, depending on the product, its complexity, and its indented use, testing requirements can vary widely. Open discussions on the required test, held early during the new product definition phases, need to be conducted to ensure all parties’ needs are met.

The expression “Soup to Nuts” implies everything from the beginning to the end. It is being used in the context of new product development to attempt to show the bounds of what testing could encompass. Everyone at every level, internally or externally, will agree that testing a product is critical to a product’s acceptance and success. Unfortunately, defining “testing” presents a dilemma as captured by the ancient Indian, subcontinent story of seven blind men each touching a different part of an elephant. Each person drew a different conclusion about the size and shape of the elephant based on their personal, limited exposure. Each person may define testing and the associated successful results differently.

The table below lists several different types of tests that might be applicable to certain products. At first glance, the list will appear to be overwhelming, even to the point of being total ignored. The number and type of tests will vary considerably based on the product, its complexity, and its intended use. For example, the rigors applied to an artificial heart product will vary considerably when compared to a smartphone app targeted at preschoolers. Even though the list is long and may look complete, the details for every entry can vary significantly. A Google search for any of the tests listed will result in many articles discussing each one in detail with many variances and interpretations.

The purpose of the list is to allow the conscious debate regarding which tests to omit and which tests to include and their relative prioritization and timing. The list is ordered in probable time sequence in accordance with typical product development activities. However, based on the product and its fundamental requirements, the testing order could verify significantly. For example, if the product’s most important feature is sub-second transaction processing, that feature, which will be dramatically impacted by the product’s fundamental architecture, should be tested very early in the process.

The company should be cautious in reaching the conclusion that every possible test should be run. All tests take time and resources. Those elements are always in short supply. The inclusion of each test should be based on the risk associated with not performing the test as well as the consequence of a defect or abnormality slipping through, only to be discovered later.

The time to determine which tests should be included, when they should be run, and the resources that would be required, needs to be considered in the early product planning stages. Only then can accurate, overall project planning be completed.

Some may argue that the new agile or fast iteration development techniques negate the need for many of these tests and certainly the time required to sequentially perform them. Rather than argue about the effectiveness of the development methodology used, the focus should be on ensuring that the results of the tests are acceptable, independent of when they are run and by whom.

No.	Test/Trial Phase	Location	H/W, S/W, Both	Physical Description	By Whom	For Whom	Capability
1	Proof of Concept	Development Lab	H/W	Breadboard working in the lab	Developers	Management, Product Management	Basic operation of major capability, "good path" only
2	Lab Demo	Development Lab	Both	Mockup working	Developers	Management, Product Management, Marketing, Sales	Basic operation of major capability, "good path" only
3	Developer Testing	Development Lab	Both	Unit, Integration, and System Testing	Developers	Developers	General operation and requirements verification
4	Alpha Test	Development Lab	Both	H/W in mostly final form, S/W "good path" only	Developers	Developers, Product Management	Functional elements work but may be fragile
5	Use Cases	QA Lab	S/W	Validate "good path"	QA Team	Development, Product Management	Verify requirements
6	Performance Test	QA Lab	Both	Load testing	QA Team	Development, Product Management	Verify capabilities
7	Robustness & Recovery	QA Lab	Both	Test during anomalous environment	QA Team	Development, Product Management	Verify capabilities
8	Mis-use Cases	QA Lab	Both	Test unorthodox uses	QA Team	Development, Product Management	Verify capabilities
9	SQA Testing	QA Lab	Both	H/W: soft tool, final form, S/W final	QA Team	Development, Product Management	Meets all functional requirements
10	Beta Test	Customer Lab	Both	Controlled customer environment	Customer w/developer support	Development, Product Management, QA, Customer	Usability and robustness in customer controlled setting
11	Pre-Production	Manufactur-ing	Both	Determine manufacture-ability	Manufactur- ing	Development, Product Management	Buildability
12	Highly Accelerated Life Testing (HALT)	External Lab	H/W	Verify operation over extreme conditions	External Lab	Development	Determine ultimate failure points
13	System Security	QA Lab or Contractor	Both	Verify hard and soft intrusion resistance	QA Team or Outside Consultants	Development, Product Management	Meets customer security requirements
14	SQA Validation	QA Lab	Both	Testing of final product built by production	QA Team	Development, Product Management	Scalability and all features and recovery tested
15	Documentation Verification	Customer Service	Both	Validate that the documentation meets the needs of all users	Customer Service	Company and Customers	Product can be supported during and after deployment
16	Customer Service Validation	Customer Service	Both	Test in mock customer application	Customer Service	Company and Customers	Setup, diagnostics, trouble shooting, resolution testing
17	Final Quality Test	QA Lab	Both	Final product, documentation	Product Management	Development, Customer Service	Top ten fitness for use capabilities review
18	Customer Service Installation Validation	Customer Service	Both	Simulated customer environment	Customer Service	Development, Product Management	Validate documentation, setup, and troubleshooting
19	Customer Field Trial	Customer in Field	Both	Customer Location	Customer w/ Customer Service observation	Development, Product Management, Customer Service, Customer	Setup and operation by customer with customer service help
20	Customer Trial Deployment	Customer in Field	Both	Customer Location	Customer	Company and Customers	Customer setup and operation and Acceptance Testing
21	Customer Initial Deployment	First Customer Rollout	Both	Customer Location	Customer	Company and Customers	Customer unassisted initial deployment
22	Customer Fulfillment	Manufactur-ing	Both	Customer Location	All company elements	Customer	Order placement, thru manufacturing, delivery, and customer support testing
23	Customer Rollout	General Availability	Both	Customer Location	Customer	Customer	Customer Delighted

Article Number : 6.020605

A Handy Reference Guide for Executives and Managers at All Levels.

9 Volumes 42 Chapters ~700 Articles

Browse Select Read Download

The weight of your world does not have to be on your shoulders.

The articles in this site will help to lift that weight from your shoulders.

Pick an article similar to how you pick a route on a page of an atlas.

There is no need to look at other articles, just as you ignore other pages in an atlas.

It is easy to start a business but it is hard to run. Bumps and unexpected sharp turns in the road are always present.

Others have traveled the road before you; learn from them. This site may help.