Process of formal software testing

Ultimately, formal methods will acquire some form of acceptance, but compromises will be made in both directions: formal methods will become simpler and formal methods training will become more common. Formal methods are distinguished from other specification systems by their emphasis on correctness and proof, which is ultimately another measure of system integrity.

Proof is a complement, not a substitute, for testing. Testing is an important part of guaranteeing any system's fitness, but it is finite. Testing cannot demonstrate that a system operates properly; it can only demonstrate that the system works for the tested cases.

Because testing cannot demonstrate that the system should work outside the tested cases, formal proof is necessary. Formally proving computer systems is not a new idea.

Knuth and Dijkstra have written extensively on the topic, although their methods of proof are based on the traditional mathematical methods. In pure sciences, proofs are verified through extensive peer review before publication. Such techniques are time-intensive and less than perfect; it isn't unusual for a published proof to contain a flaw. Given the cost and time requirements of systems engineering, traditional proving techniques are not really applicable.

Because of the costs of hand verification, most formal methods use automated theorem proving systems to verify their designs. Automated theorem provers are best described as mathematical CAD tools: they can prove simple propositions and automatically and provide assistance for verifying more complex theorems.

Formal methods offer additional benefits outside of provability, and these benefits do deserve some mention. However, most of these benefits are available from other systems, and usually without the steep learning curve that formal methods require. The discipline involved in formal specification has proved useful even on already existing systems.

This is also helpful in tracking the testing progress. Each team has a role to play in the delivery of a good quality software product.

Ensure that each team is assigned clear cut guidelines and areas, avoiding a conflict of interest. Establish a clear and precise communication plan along with SPOC Single point of contact list and escalation matrix which will contain a list people or contacts to be used in case of any escalations or in cases where information is needed immediately. It is always advisable to have the team members test a different module each time.

The reason is twofold. For one the tester get a better understanding of the complete project instead of a single module and also new bugs can be found if the module is tested by fresh testers each time. Always be prepared for changes and contingencies this also includes the risks.

Planning such situations help you avoid panic and face the scenario more gracefully too. Always maintain a risk register and review it frequently. You can also have a time slot set aside with all the stakeholders to review this risk register say weekly.

Also, make sure that all risks and mitigation plans are captured accurately. Maintain and update simple metrics as defect found, defects fixed and percentage completion, quality level etc. This will have a much greater impact and visibility than you can imagine. Give it a try. Monitor your production app. And yes for the unfortunate bugs that do find their way to production, make sure they are included as part of the regression test suite with a high priority.

Overall Issues and Reporting All the information regarding bugs and product development should be maintained in a common Share Point or repository for everyone to access and derive reports when needed.

Always have a fresh and open mind when testing software. Have no assumptions. And always refer to the requirements document for doubts and clarifications.

The entire product module should be divided into smaller parts to ensure each and every part is tested thoroughly Always ensure to have included the maximum details in the bug report which should include the user credentials, navigations or step to reproduce, expected and actual results, environment, version, logs and screenshots of the error. Always give clear instructions to reproduce an issue. Do not assume that the reader will know the application.

Mention each object button, link, edit box etc. Discuss and reach a consensus before the starting of the testing on the number of high priority and medium priority bugs can remain open or deferred at the time of movement to production Metrics are important criteria proving the effectiveness of any testing process.

Choose appropriate metrics based on the project you are working on. Verbal communications should be topped up by documenting the same in email and shared with relevant stakeholders. This not only creates better visibility but is also helpful in having a chronological record of communications.

TDD — Test Driven Development is a newer form of development which takes the base from the test cases. This can also be employed for a better quality of code development. Have a UAT testing at least once before the product is released into production. This will be useful in identifying issues which are related to actual client usage. UAT testing is best performed by the users themselves. Always keep a check on the time taken by the development to fix the bugs.

This data is part of an important metrics which is used to calculate the overall efficiency of the development team. Last but not least, trust your instincts. Yes, the QA people sense issues. So just follow your instincts, if you feel a certain area can have issues, mark my words there more chances that issues will be found in that part of the code. As important as testing and quality assurance is, we should give it the time and resources it needs.

This pays off in the long run. These tips will be helpful to you for implementing a good test strategy as well as a quality product. Test Cost Calculator. Try Now! Services We Offer. Once the document cleared the entry check, which part of the document needs to be tested is decided by the author and moderator. Kick-Off: In this stage, a brief introduction about the goal and objective of the review meeting or document is given to everyone at the meeting.

This step is optional but helps in giving understandability of the relationship between the document under review and other documents if there are an ample number of documents related to it. Preparation: In this stage, the reviewer also known as Inspectors reviews the document individually based on related documents, checklists, rules and procedures provided.

While reviewing the document each reviewer identifies the defects, questions, and comments according to the understanding of the document. After that, all issues are recorded with the help of Scribe a separate person to do the logging of the bugs found during the review process. Moderator is responsible for the scheduling of the review i. Moderator performs entry checks and defines the exit criteria. Once the entry check for a particular document is passed, both moderator and author of the document decides which part of the document needs to review.

The formal review team is consists of members. So that each reviewer can focus on a particular type of defect during checking.

This process not only saves time but also reduces the chances of different reviewers finding the same defect. The next phase of the formal review process is an optional kick-off meeting. The main goal of this phase is to get everybody on the same wavelength regarding the document under review and to commit to the time that will be spent on checking.

In this meeting, reviewers receive a short introduction to the objectives of the review and its document. Role assignment, pages to be checked, check rate, and other things that need to be carried out for the review are discussed in this meeting. Distribution of the review documents, source documents, and other relatable documents are also shared during the kick-off meeting. A kick-off meeting is highly recommended as it motivates the reviewers.

In this phase, using related documents, rules, checklist, and procedures each team member work individually on the documents. These team members individually identify bugs, comments, questions according to their understanding of the document, and role.