Software Publishing: The Testing Of Your Solution

From Dwayne Wright - Certified FileMaker 9 Developer
TWITTER: dwaynewright

This is part of my ongoing series of articles about setting up your own software publishing business (cause FileMaker is very good option for doing that). This was originally published by me in a guide called FileMaker Software Publishing back in 2002. .

If done at all, most FileMaker testing is done is very small steps, usually in the ScriptMaker environment. Seldom is a FileMaker solution put into a professional testing phase before implementation. Testing takes time, time is money … so … the common math is that less testing equals more money saved.

The problem is that without proper testing, all database users become test subjects and they generally didn't know it. These test subjects are trying to be productive team members and you violate their trust as a designer by implementing an untested software solution. If you are working as a consultant and the client gives permission for implementation without testing, that is their business decision to live with. It’s a different matter to sell a FileMaker solution that has not been tested.

Before releasing a robust FileMaker based solution, we recommend 4 phases of testing.

- Internal Designer Testing
- Testing During Production Of User Documentation
- Hire An Outside Tester To Test Solution and Produce User Documentation
- Controlled Beta Testing To Test Solution and Produce User Documentation

Internal testing includes the developers project design testing. This covers the "in the moment" tests of calculations, scripts and layout design. Normally this includes the last thing you do before marking a design action item as complete. This testing is good except it isn't evaluated on a whole. It is always possible that one change in a database can have an adverse reaction to another part of the database.

Testing during documentation is one of my favorites. Probably, because I like the writing process but I find it an excellent testing tool as well. When you write a tutorial for example, you might find yourself thinking " this isn’t quite the way I want" and you redo it. I’ve had occasions were I’m writing a tutorial where a script breaks, a tab order is out of kilter, a field has a value list when it should not, etc…

Hiring an outside tester can be a little expensive but can also be a valuable asset. Normally, it’s a per project type of event. You look to hire a professional experienced in the vertical market for your solution. It’s best if they are unfamiliar to FileMaker but not a requirement. You should have a written list of feedback you are wanting to get from the tester. Don’t simply ask them to look it over and tell you what they think.

I suggest a list that corresponds with your written tutorial. Have each testing action item correspond to an action that a user would normally perform. I would ask them how important an item was to the solution, how well did it perform and what comments do they have.

After all of this first round of testing is complete and changes have been made, you are now ready to enter the final round of testing. This is the popular event known as beta testing. You may want to have a mixed set of FileMaker savvy individuals and a set of vertical market specialist of which the solution is intended. You can gather folks by promoting the beta test program on email lists, via press releases and on your web site. Of course, you can also extend the beta test promotion to any current customers and prospects. Remember that any promotion for beta testing for your solution is the same as advertising for your solution.

The beta test process is used primarily as a bug hunt and for feedback for the next version. Usually, it’s too late at this stage to make a significant amount of new feature implementations to the product. Each beta tester needs to be given a unique identification number and their comments recorded. Being a FileMaker designer, it shouldn’t be a problem for you to build a beta tester database. As with the outside tester, you need to have an organized plan in acquiring your testing results.

Be sure that each beta tester realizes that this version of the software is not for production use but only for testing. You should have each user sign an agreement to be eligible for the testing process. This agreement ensures they will not discuss the software with those outside the beta test process ( non disclosure), they realize it’s not to be used in a production environment and whatever else you deem necessary to go into the agreement.

A beta tester that does not provide feedback on the product is not worth having. Stay in touch with your testers and make sure they realize how important they are in the design process.

When you think you are ready to roll your solution out, you will want to do some internal and external testing. Internal testing is done at your site. This is one reason why I usually purchase one new computer a year and don’t get rid of my old ones. This is a good way to test a solution on different machines, different operating systems and different screen sizes.

If possible, test your solution on clean machines after fresh operating system installations. If you are distributing a runtime version of your solution, it is not a bad idea to test systems that do not have FileMaker installed previously and ones that do.

If possible to use different partitions on a machine and different operating systems to test your solution. You can even use utilities such as Virtual PC, which emulates a Windows operating system on a Macintosh computer. Any level of testing you can do internally is a good thing.

If your solution is not a runtime version, you will probably want to test your product in a networked environment. You may want to test it under both FileMaker Server and hosted configurations.

You should also test your solution under the different password levels you have in the system. This would include the productive user of the system on each password as well as attempts to access functions a password should not have access to.

More info about the author and FileMaker in general, contact me at

© 2008 - Dwayne Wright -

The material on this document is offered AS IS. There is NO REPRESENTATION OR WARRANTY, expressed or implied, nor does any other contributor to this document. WARRANTIES OF MERCHANT ABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE EXPRESSLY DISCLAIMED. Consequential and incidental damages are expressly excluded. FileMaker Pro is the registered trademark of FileMaker Inc.