After a trial is complete, there is a simple, three-question final exam that will be the most important test to determine the success of the trial. Even if all of the features and functions worked flawlessly as envisioned and specified, the three final exam questions will be the primary determining factors to consider before moving forward or, perhaps, retrenching.
As described in some of the articles in the Development chapter of this collection, there is a rule-of-thumb that states the “good path,” in terms of a product or service operating properly when every condition encountered is as expected, represents about 10% of the overall product development effort. This does not imply that all of the factors are perfect. It assumes that all factors are within their normal bonds. However, the “bad path,” which occurs when some of the conditions vary from the “standard” expectation, represents about 90% of the effort in the development of a product that can be used and supported in actual real world conditions. The “bad path” implies that something varied from the expected conditions. It could be user error, a related system not being available, or a component being out of tolerance. It is not uncommon that companies are so anxious to begin a trail that they start long before the “90% Bad Path” items are properly addressed. In those cases, when “bad things” occur during the trial, they most likely will require intervention that is beyond the capability of the customer.
To be fair, a large part of any trial is to discover situations that may occur in “real world” deployments with customers that are not obvious or were not considered in the product’s development. Therefore, attempting to address all “90%” of the possible anomalies is not practical. However, when discovered, some of these anomalies could have fundamental impacts on the product that could require significant re-trenching to address their occurrence in the future. Others, perhaps, can be addressed by minor changes in labeling or user documentation. Since multiple support personnel may be involved during the trial process, it is critical to document and discuss each unexpected condition or involvement to understand its root cause and the long term corrective actions required to address the underlying issue.
In some situations, a product or service offering may work exactly as intended and meet all of the customer’s expectations, but may still not be a viable alternative. We all experience this situation when we walk down the aisles of virtually any retail store. We see “new and improved” items that are more attractive than the item we currently have and would like to have the new item, but simply cannot justify replacing what we have. Products in this category fall into the “nice to have” instead of the “must have” category. For personal items, the barrier between “nice to have” and “must have” may be passed quickly and based on emotion instead of sound financial analysis. Not so, in commercial settings where the new purchases may be scrutinized by others or the change impacts are substantial. With the company’s laser focus on their new offering, it is quite common for them to miss a customer’s reluctance to change, no matter how “nice” the new product is.
Did the trial uncover, or were certain aspects of the product identified, that cannot be easily scaled prior to the product’s commercial launch?
It is easy to fall into the trap of focusing a trial on the steady state operational aspects of a product, assuming that it was initially installed properly and the user was trained. Many, if not all of us, have experienced the situation that once a product was working, it worked! However, the effort to get it to initially work was traumatic. The difficulty often stems from the incredible variations in the product’s environment. Anyone who has attempted to load new software on a PC has probably experienced that situation. In the past few years, software developers have made great strides in simplifying this task, but calls to help desks are still quite common.
As another example, does the product require custom configuration before it is shipped to a customer? For a limited trial, this approach may be acceptable, but for high volume shipments, perhaps through a stocking distributor, may be totally impractical. Finally, if the product requires connectivity to some back office service, can that service support high volume, simultaneous users? A more general case of considering the impacts of scalability is discussed in the article, “Process: When to Start.” In that article, the concept of “adding a zero” was discussed. The exercise consists of considering what would happen if it was required to ship or support ten sales instead of one, or one hundred instead of ten. After the impacts of adding one zero are understood, repeat the exercise by adding two zeros; ship one hundred instead of one or one thousand instead of ten. This exercise will quickly identify the first breaking point long before it occurs and allow you to consider methods to avoid it.
Candidly taking the trial final exam and understanding the results can dramatically impact your next steps after a trial. The advantage of taking the trial final exam is that you can grade it internally, long before prospects and customers take the exam with you and judge the results by themselves.