By Frank Puranik – Senior Technical Specialist
If you’re anything like a lot of customers whose software testing departments I’ve spoken to you’ve just rolled your eyes at the title: “We haven’t even got our functional test fully sorted out. How can we possibly start on non-functional testing”.
The answer is: “You have to. If you don’t, it will cost you big in remedial costs, lost revenue, reputational damage and star ratings”
Stay with me in this article and I’ll explain why…
Firstly, let me come clean: I do have an “axe to grind”. I come from a non-functional test background: “Testing in the Network” (not to be confused at all with Network Testing – see boxout). Because of that I’m going to use Network-related examples to make my point here.
Notice I don’t say “Network Testing” – that’s a totally different thing and is, in general, only the province of network engineers. Network Testing is obviously important; networks are the “roads” through which our applications communicate and they need to work as well as possible. But, by the time they reach application engineers and testers, the networks simply “Are what they are”: We can’t really improve cellular (mobile) network quality, we can’t make a busy WAN less busy, we can’t make the Internet run faster, we can’t make satellite networks less “laggy”. It is what it is! And we need to work with it
Why do we care about Testing in the Network?
You can do all the functional testing you like to ensure:
- Every button works
- Every calculation is correct
😀
Still, when your customer uses your application in a Real-World Network they get something like:
They’re going to think you didn’t test it at all.
It’s worse than that, though, they might even think their account has been hacked (note the unauthorized – 401 message), when actually what happened was that the network gave up at the wrong point in the transaction.
So here’s a “non-functional” problem that has created a critical application issue:
- The app has issued a very user unfriendly error message (401 – really?!)
- The app has misled the user about the nature of the problem – it looks like a security issue
- The app provides no information on what to do now
Risk Analysis
How, then, can we justify that we must complete all functional testing when there are gaping holes in the behaviour of the application for non-functional (environmental in our example) reasons which can’t easily be dismissed.
Good practice says that during application design, development and project management of the complete development lifecycle we need to look at the risks of issues with our application and spend our time (and dollars) accordingly, not simply blindly say we must focus all our efforts on completing “all” the functional testing.
Our example above shows a relatively high risk event, given the propensity of mobile networks to deliver extremely variable connections, dependent on location, other users, weather conditions and time of day. If you’re not developing for the mobile environment you might say, that won’t affect us, but there are other issues:
-
The application may run unacceptably slowly in the real-world network
-
The application may display interminable spinning circles of waiting, hour glasses or just not respond in the real-world environment
- The application may exhibit timeouts and resultant poor error messages in the real-world environment
Neither, despite initial appearances, are these issues performance related: we’re not talking about a little, or even a lot, slower than expected, we’re talking of issues that make the user give up completely.
These are, therefore, all risks which the project management of the application cannot ignore or rule as out of scope.
Why does this happen?
It’s very simple really:
- The designers may use network communication approaches that cannot work in real world networks at all
- Prototypes are developed and demonstrated in perfect LAN networks
- Developers will build and unit test their code in perfect LAN networks
- Most separate Testing is functional and this will also use perfect LAN networks
So the first time most applications see real-world networks, with all their foibles, is:
- For internal applications: pre-production
- For external applications: beta testers or customers
Either way, by now, you’ve spent a lot of time and money developing your product down a particular direction which there may be no easy or cheap way out of. Its fundamental communication design may be incompatible with the network environment it must operate in.
In general terms then, our non-functional problem is an environmental one.
It’s akin to a motor car manufacturer building a car for off road use while, at all initial stages (prototyping etc), using only smooth paved roads, and somehow being surprised that it didn’t work well off-road. It wouldn’t happen – in this process everybody understands the “off road risks” and factors them into the design accordingly.
The Solution – Shift Left
Well firstly we need to understand that there is a real and very tangible risk here. You can’t ignore the risk unless your application will never use the network at all, or only be used in a local (LAN) network.
We need to shift “Testing in the Network” to the left, starting with real-world network testing of the prototypes (so that we don’t go too far down a design blind alley) and continuing this through all the development, unit testing etc.
Some organizations (by which I mean the whole company when applications are being developed for external users, or the development department, if applications are being created for internal use) do recognize that, but still take the risk, leaving network testing very (or too) late because it is so hard to get a real-world network into the development and test environment.
There are answers to that. While it is true that doing development and testing in a real network is difficult, iTrinegy produces Virtual Test Network products that, delivered as hardware or virtual appliances, let you create any style and quality of network you need.
That makes it easy to have your test network available at all application development stages and effectively make this test network form part of every non-functional test you do. In this way, you can fully shift “Testing in the Network” to the left.
Your prize(s):
- Eliminate Remedial costs due to poor customer networks
- Increase your customer revenues and therefore yours – end users more productive
- Increase customer satisfaction:
– Build reputation
– Better customer to prospective customer recommendations
– Get Better Star Ratings