Generative AI continues to dominate the innovation conversation, and ITX is determined to be a part of it. Because of our commitment to being a leader in the AI space, ITX is constantly experimenting with the tools available today.
Recently, an ITX team wanted to see how incorporating AI into our agile development process could affect turnarounds for a key aspect of software development: testing. When doing manual testing, for each story, a tester defines the test case based on a standard template. They must then complete the steps and verify the results. Naturally, they are prone to human error, so why not see if AI can eliminate or reduce that error?
Our Hypotheses
1. Generative AI can help us reduce the time and effort of defining manual test cases, given a defined acceptance criteria (AC).
2. Generative AI can help us identify new conditions or scenarios to be tested that may not be captured by the acceptance criteria.
To test these hypotheses, the team designed an experiment to compare the work of human testers to an AI. The first test used Microsoft Copilot. Both would be assigned user stories from our backlog; we chose complex stories that had well-defined acceptance criteria and required some business knowledge to complete.
The Experiment
The human testers would then define the manual test cases for each user story, using a standard template. They also gave an estimation of the time and effort required to define the acceptance criteria and the manual test cases.
Copilot, given the same stories to complete, would also define manual test cases, using a standard template and a natural language prompt. The prompt provided the context, role, output format, examples, and constraints of the task.
Finally, a tester not involved in the experiment evaluated the accuracy, completeness, defined testing data, error/alternative path inclusion, and formatting of the test cases, giving them a score of 1-5.
Afterwards, the acceptance criteria were refined and the previous steps repeated multiple times with other AI tools: GitHub Copilot and Jira AI
The Results
The human tester consistently outperformed the AI in the scoring of test cases. Rarely, the AI would have a score close to our human, but never higher. In terms of time invested, however, the AI did come out on top: the time invested in the manual test cases was 58% lower than the time a human would take. That figure includes time spent reviewing and correcting AI mistakes and hallucinations, which points to a real reduction in testing time.
However, no new scenarios were created by the AI. The AI only focused on what was explicitly mentioned in the acceptance criteria.
When comparing the different AI tools, we found that GitHub Copilot and Jira AI both outperformed Microsoft Copilot. We believe these tools deserve further study, though we need review confidentiality concerns for these tools.
There were mistakes from the AI, of course. There were several reoccurring errors as well as restrictions our prompters found during the experiment:
1. Even when requested not to do it, MS Copilot showed some hallucinations, so it requires manual revision and adjustments of the results in all cases.
2. Copilot may not work well with content containing SmartArt, tables, or charts. It does not respect the formatting of test cases using templates attached as referenced files. It almost never defined test cases using tables of bullet list structure with different levels.
3. There are many errors during the requests that prevent creating any content and require re execution.
4. Copilot doesn’t seem to consider images on the referenced files.
So, let’s check in on our hypotheses.
Does Generative AI help us reduce the time and effort of defining manual test cases, given a defined acceptance criteria? Yes, definitely. A 58% reduction in time spent on testing is significant.
Can Generative AI help us identify new conditions or scenarios to be tested that may not be captured by the acceptance criteria? Doesn’t seem like it. None of the tools we tested created new scenarios.
Conclusion
Our team recommends further research on Jira AI and GitHub Copilot, and we’re excited to continue it. To read more about our experiments, check out our Research
Carolina Pernaut is the Manager of Quality at ITX, where she has defined QA processes and provided oversight on many projects. As a key contributor to ITX’s Software Development Lifecycle, she is committed to preventing costly post-release corrections, maintaining client reputations, and delivering reliable value to end users. Her expertise spans multiple industries, including financial services, ecommerce, and insurance brokerage.
Andrew Tyrell-Smith is a UX Writer at ITX. He graduated from the University of San Diego with a BA in English, and has spoken every word he has ever written out loud.
