Quality is free, but only to those who are willing to pay heavily for it.
Peopleware: Productive Projects and Teams (1987)
At HE Inventions, we make products for the packaging industry. We call them Digital Inspection Tables, and they do what you would expect. They help you inspect products, digitally.
Digital in this respect is a combination of projection and paperless reporting. Sounds relatively simple on the outside: select a job, the image projects, the operator says yay or nay. However, as with most products, developing and testing is wrought with danger.
We currently have a wide ranging tech stack as we need to: control hardware, do camera and projector calibration, and also present a fast interface. Additionally, these are hosted in more than one computer in the product. We have systems based on C++, C#, Python, and Typescript. All of our systems have extensive unit tests (automated by CI) which they have to pass before they can go through to build.
However, there’s always an unforeseen issue when it comes together into the chassis and again when coming into contact with the customer.
When we deploy our product, we can’t just spin up another instance on a cloud service. We need to buy in hardware and deploy onto that. Hardware is a constantly moving target. Products are moved into end of life and we need to use equivalent machines. 99% of the time this is fine, but every so often you hit a curve ball such as a USB bus being incompatible with your depth camera. This means we need to do a lot of integration and manual testing for each new hardware configuration to be confident that the machine will be considered “turn-key” from the customer perspective. There are not many alternatives other than “pressing buttons” testing, whether that’s manual or automatic.
One of our product’s key selling points is integration with a customers network – which is to say there are a near infinite number of weird setups we have to deal with. Setups which perfectly work for the end customer, but don’t work very well for us. For example, Samba shares which auto disconnect without file activity, or incredibly restrictive firewall rules. The only way of dealing with these is either on the day, or to build up enough template solutions you can modify to the new situation. Neither can be done in advance very effectively.
The golden thread running through this is that testing for industrial products, on an instance level, takes time. A lot of it. And if you missed something, you are not getting on a plane to fix the machine you’ve just warrantied for 12 months – because not all customers have internet connections or allow external connections inwards.
The automated testing that comes with CI/CD systems (such as Gitlab) is phenomenal and has saved us a ton of time. However, it still has not saved us from having to do intensive manual testing on each of our machines. Every sale is critical and with Quality products, reputation is worth significantly more than a few hours of testing per machine.
Discussion Points
- Do you have any testing stories which guide your product?
- How have you solved testing hardware products?
- What am I missing?