AI in Software Testing: All You Need To Know

Table of Contents

AI-driven testing makes the software development lifecycle much easier. There is a strong relationship between testing and the quality of software development. Yet, there might be some pitfalls in AI in software testing. Below, you will find out the principles of how to use AI in software testing.

AI in Software Testing

The topic of artificial intelligence took my interest more than 10 years ago. At that time, I was working on my dissertation and on the research in the field of application of artificial intelligence methods for image processing and recognition. The images were photomasks (special elements used in the manufacturing of microelectronic products). My goal was to get an image of the template, to recognize the distinctive marks on it — fiducial marks, to combine the resulting image with the reference one, and to identify defects.

To recognize fiducial marks, it was necessary to use a system which would combine two images of photomasks according to certain artificial intelligence methods: a reference and a working one, obtained in real conditions of interference. The alignment was carried out according to the fiducial marks, which are located at the corners of the photomasks and can have a certain shape. The methods used for fiducial recognition are based on the use of neural networks.

The architecture of the neural network was a multilayer perceptron, whereas the approach to training the neural network was based on the modification of gradient methods and the application of the hyperbolic wavelet transform proposed in the thesis. This made it possible to receive a system that provided high reliability of recognition of fiducial marks and determined defects in images with reduced requirements for resource consumption of hardware. The result is the following system:

Fig. 1. Artificial intelligence system for identifying defects in photomask images

Fig. 2 (a) Result of recognition of fiducial marks Fig. 2 (b) Revealed defects of the photomask

My Experience with QA: How It Was

The successful defense of my dissertation brought me not only a Ph.D. degree in technical sciences. I received knowledge which I decided to use in a different direction in the long run – artificial intelligence in software testing.

I started my career in QA and combined my work in an IT company with teaching at the university. My university program included the Software System Testing course, due to my recommendation. Without a doubt, such a course is demanding at the moment.

Testing is one of the most established ways to ensure the quality of software development. The test implementation process began to include the planning, design, creation, maintenance, and execution of tests.

At the same time, there was a transition from direct testing to quality assurance, covering the software development cycle as a whole. While gaining practical experience in the field of QA, I tried to analyze the main problems that I had to face in my work. Practical experience in the field of QA and the knowledge I gained in the course of work on the dissertation made me more interested in the factual application of artificial intelligence for software testing.

Main Pitfalls

I have distinguished the following pitfalls in software testing using artificial intelligence:

Defects

One of the main areas of work of a QA engineer is the search for defects and their classification. For example, determination of the degree of severity of such a defect and its impact on the developed functionality. The correct classification of defects based on their severity significantly affects their further life cycle, as well as the entire development/testing process.

The solution to this problem requires the knowledge, logical reasoning, and experience of a software quality control engineer. The classification of defects is possible on the basis of knowledge about their types and characteristics for each of these types. I recommend using Fuzzy Logic to classify defects. The main advantage of fuzzy classification is the ability to formulate reliable classification conclusions based on incomplete and not entirely reliable input premises.

What has been done to date: it was proposed to apply the Mamdani algorithm for the classification of defects. Mamdani’s algorithm is prominent because it relies on the “black box” principle. Quantitative values are received at the input, whereas at the output we also obtain values in quantitative terms. At the intermediate stages, Fuzzy Logic and the theory of fuzzy sets are used. This is the best application of fuzzy systems. It is possible to manipulate the usual numerical data, but at the same time, it is necessary to use the flexibility that fuzzy inference systems provide.

In practice, the algorithm works like this: a rule base is formed, in which the features of each defect class and class membership are set, the stages of fuzzification, aggregation, and defuzzification are implemented. Signs of a defect that characterize each class are received at the input, whereas at the output we receive the severity of the defect (Fig. 3)

Fig. 3 A system for determining the severity of defects based on fuzzy logic

Limited Time for Writing Test Documentation

A relevant problem in testing is the limited time for writing test documentation. It is difficult to check all possible cases/options/values. Yet, test design techniques might help. They are equivalence classes, boundary values, and causal analysis. Analyzing this problem, I came up with an idea — to use Bayesian networks to select effective tests that should be performed. Bayesian network is a graph probabilistic model, which is a set of variables and their probabilistic dependencies according to Bayes.

For example, a Bayesian network can be used to determine the disease of a patient by analyzing the presence or absence of symptoms, based on data on the relationship between symptoms and diseases. It is proposed to build a Bayesian network based on equivalence classes and use it to calculate the probability of a defect of a certain type. Then, select the most effective tests on the basis of causal relationships. This area is under study and there are interesting developments found.

Neural Networks

Having become acquainted with the capabilities of neural networks (ANNs), particularly their ability to learn and solve forecasting problems, I came up with an idea of using ANNs for software testing (Fig. 4)

Fig. 4 An example of the structure of a neural network

The training sample can consist of a verified site which would provide training on the neural network (this would refer to the reference version). I recommend limiting the training sample in relation to possible elements on the site. For example, buttons, selects, or checkboxes. After learning the neural network, you can submit a site that contains bugs to the entrance. As a result, we will receive information about discrepancies between different versions of the site. By using different neural network architectures and teaching methods, you can experiment and choose the option that gives the best result.

Results

The aspect of AI in software testing will certainly develop. The use of artificial intelligence methods in test automation, choosing the most effective tests, predicting the number of defects in releases — all these are relevant and interesting objects for research and development. Understanding this direction requires knowledge of linear algebra, mathematical analysis, probability theory, and testing.

Do you need help with Quality Assurance?

Keenethics can assist you in solving any software issues. Get in touch with us and together we will find the best solution for your business.