There are many different methods of software testing and it is important for developers to know both the advantages and disadvantages of choosing one over the other. A few of the most common methods of software testing are black box testing, white box testing, and grey box testing. Each of these options can be beneficial and provide great feedback for the developer but choosing the method of choice will greatly depend on the application being tested and the personal preference of the developer.
White box testing, also called glass testing or open box testing, differs greatly from black box testing because the tester will focus on the code of the application (“Black Box vs. White Box” 2008). When using the white box testing method the tester will need to have extensive knowledge of how the code in the application is working. This can dramatically increase the overall cost of the testing process (“Software Testing Methods” 2013). On the other hand, having a tester that is code savvy can help with code optimization and can lead to problems in the overall user experience being solved in a more time efficient manner (“Black Box vs. White Box” 2008).
Another form of testing is Grey box testing, which as the name implies, is a mix of black and white box testing methods. Grey box testing requires the tester to have access to the design documents and the database the application is using. This makes preparing the test data and scenarios much easier. The tester will still be focusing on the user interface and not the code as in black box testing, but the tester will have a better understanding of what exactly the application is doing behind the scenes than a black box tester would (“Software Testing Methods” 2013).
Black-box software testing is a technique that tests apps to see how the apps function to a non-technical user. The tester’s goal is to accomplish tasks set out by a testing plan. The tester knows what he has to accomplish, yet is not aware of how the code functions. The type of user and the type of app must be taken into consideration in order to gain the most useful information from the black box test. For example, a technical app would be easier to use for someone that knows a lot about technology and understands programming. Where as a non-technical user would feel overwhelmed and frustrated by the app. This is an oversimplification of software testing theory, the metrics required for a successful test have to be documented in a software test plan.
According to Laurie Williams in collaboration with NCSU.edu, “Black box testing (also called functional testing) is testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions.” (Williams, 2006) In black-box testing the tester does NOT have access to the code. Relying only on the external output generated by their direct actions that are inserted into the application. The software test plan should define what outputs are expected when executing certain actions.
Black-box testing focuses on the functionality of the user interface, it also focuses on the users and how they interact with the application. Testers should be seen as internal users that give valuable feedback for your application. Although, most black box testers don’t require technical skills. Some black-box testers have scripting skills in order to automate testing. In order to understand how the testing is accomplished, let’s take a look as some of the roles and responsibilities the tester has. The tester needs to know that every input returns an appropriate output. Aside from looking for functional defects the black-box tester should also look for usability issues. When testing it is important to identify known usability issues found either through testing or through user bug-reporting systems. It is critical to understand what quality is from the user’s perspective and to make sure the tester works with the development team to meet these goals.
The IEEE definition of test case is “Documentation specifying inputs, predicted results, and a set of execution conditions for a test item.” (Gumaste) Writing a test case procedure to black box test an app primarily focuses on functionality without looking at the source code. The main focus is to look at the application and interact with the UI as a user would. In order to write a test case you have to have an abundant knowledge of the software you are writing it for. The test is essentially set up to have non-programmers ‘evaluate’ the app in normal use cases. The tester only has to be aware of how the app should function and verify that it is working properly, and does not need to have any knowledge of the source code.
As Robert Japenga alludes to in his paper on How to Write a Great Software Test Plan, you will NEVER be able to test enough (Japenga). There is never a way to test every single aspect of a piece of software. He goes on to say that if this fact makes you a little uneasy, than you are starting in the direction of a great test plan. To write a good test procedure you first have to know why you are writing it. The whole purpose is to find errors. If you know anything about the code aspect of developing software you know how complex even the most simple program can be, so you have to be prepared from the start of development to implementing testing procedures along the design path. With this in mind you should prepare a plan as soon as the software is defined. Then you should be focusing on the software specifications. The software specifications state what the core functionality of the app will be, and this is what you want to check for failures. These are the most import aspects of the app from a user’s standpoint. When writing test procedures it is important to have a timeline as to when the app is to progress through the process of development and launch. You then have to identify who will verify all the information that is to be tested, and be sure to have all the hardware needed for testing (Japenga).
According to Robert Japenga there are 5 items that need to be addressed to create a successful test plan. The first is “Definition and Objectives of the Phases of Testing” which focuses on module testing, integration testing, systems/acceptance testing, and best testing. The second is “Schedules” this part defines the phases of the testing plan by answering the questions similar to, what are the dependencies of each phase, and how long will each phase last? The third item that needs to be defined is “responsibilities,” responsibilities addresses who will be creating and executing test cases as well as who will be fixing them. The next item is “Target Equipment Required” this item identifies what equipment will be required for testing. An example would be a physical development server, or a pre-production server. Finally, the fifth area, “Test Equipment Required,” identifies EVERY test equipment required to make the test plan succeed, including what equipment needs to be purchased or rented (Japenga).
Japenga goes on to explain that there is a second part to creating an efficient test plan and that the goal is to define exactly what should be tested. You start off with a Software Requirements Specification document and create test cases from it. Next we create testing scenarios outside the SRS document that encompass the system as a whole. Finally, you use off-design test scenarios that are also outside and not defined in the SRS document. This allows us to see how the software works and makes certain the application is not executing faulty processes (Japenga).
According to Jeff Sandberg, developer of TrustedAd, In order to create a Black-Box Developed application, there are four stages that should be followed. The first stage is to write the documentation first, which allows the application to be defined before you start to program. The second step is to write the actual code, while using the documentation that was created in stage one. After you have an idea of the functionality you want to create and have the code, the next step is to write your tests and test the application. The code along with the documentation should match the tests. The final step is to refactor your documentation AND code and follow through the cycle as many times as necessary to reduces errors and bugs with the application (Sandberg, 2012).
Developers will deduce that Black-box testing is created to minimize the amount of bugs in the user interface, while White-Box testing is used to reduce the amount of errors in programming logic. The combination of the two, Grey-Box testing, allows the tester to tackle both types of testing procedures and test the overall efficiency of the code, yet, Grey-Box testings also allows the tester to test the user experience of the human-machine interface.
Photo by thierry ehrmann.
Black-box vs. white-box testing: choosing the right approach to deliver quality applications.
(2008). Retrieved from http://www.cs.unh.edu/~it666/reading_list/Defense/blackbox_vs_whitebox_testing.pdf
Gumaste, V. (n.d.). Test case writing (creation) 101. Retrieved from
Japenga, R. (n.d.). How to write a great software test plan. Retrieved from
Sandberg, J. (2012, November). [Web log message]. Retrieved from
Software testing methods. (2013). Retrieved from
Williams, L. (2006). Testing overview and black-box testing techniques. Retrieved from