ITX’s QA Practice Achieves 60% Time Savings on Key Tasks with GitHub Copilot and Jira AI
ITX has helped clients produce thousands of custom software builds over the past 2+ decades, all of which have enjoyed rigorous quality assurance (QA) review. Through this extensive experience, we have learned that defining test cases typically requires 25-35% of the total QA time billed in each Agile sprint. To accelerate the client’s time to market, ITX deployed a team of QA specialists to work in parallel with their colleagues across four client projects. They were assigned to replicate the task of manual test case creation using four different Gen AI Tools: ChatGPT, GitHub Copilot, Microsoft Copilot, and Jira AI.
After investigating all the AI tools and developing best practices, the QA subteam achieved 60% time savings and improved efficiency in their workflows. This allowed for the safe delivery of high-quality products and a boost in speed, efficiency, and client confidence.
CHALLENGE
At ITX, defining manual test cases is an essential task in the quality assurance (QA) of digital software. But it often accounts for 25-35% of the time billed by QA professionals in every Agile sprint.
ACTION
Using 4 GenAI tools – ChatGPT, GitHub Copilot, Microsoft Copilot, and Jira AI – ITX’s QA Practice leadership set out to reduce the client investment, speed the process of creating test cases, and boost client confidence in the outputs.
RESULT
While working in parallel across 4 different client projects, ITX achieved 60% time savings on manual test case creation – a reliable, repeatable QA process that will help future clients ship high-quality products even faster.
Thanks to the way ITX is really thinking through AI integration, we were pretty confident we could perform this task faster using AI. We experimented with 4 different tools and developed a strong set of best practices in prompting and review. That approach helped to show clear results. I’m super-excited for the opportunity to help clients get releases into market faster.
Challenge.
Test cases represent the blueprint for digital quality assurance validation efforts at ITX. By providing a structured, repeatable approach, we are able to ensure test cases uphold the integrity of the software.
As vital as test case definition is to protecting your software’s security, performance, and functionality, performing the task manually can be a significant draw on resources, comprising as much as one-third of the total QA time budgeted for each sprint in the project. Determined to find a way to mitigate the impact and accelerate the process, the leader of the QA Practice within ITX explored a handful of GenAI tools to see which would deliver the best result.
Action.
After identifying and carefully vetting four different GenAI tools –ChatGPT, GitHub Copilot, Microsoft Copilot, and Jira AI – we assembled a team of QA professionals and deployed them as an additional but separate team working across 4 different client projects. Their task was to duplicate the primary team’s test case creation effort, track the performance of each AI tool, and report their findings.
Certain to redact client-specific information before proceeding, the team created test case definitions using each GenAI tool. They then scored the GenAI outputs and compared them to those produced manually, without AI interaction, by the QA specialists assigned to all 4 projects. Time was tracked carefully for both the added team and the QA teams assigned to the client work. This time included the effort required to review, polish, and finalize AI-supported test case definitions.
The Result
By integrating GenAI tools into our QA process, ITX demonstrated our ability to deliver higher-quality software with far greater efficiency. Specifically, the AI-assisted team:
- Saved significant time. We achieved 60% time savings on test case creation through the new AI-assisted workflow. This represents between 15-20% of the total time QA Analysts spend on a given sprint through the duration of the project.
- Identified top tools. We discovered that GitHub Copilot and Jira AI performed better in objective scoring than ChatGPT and Microsoft Copilot.
- Prepared to re-invest time saved. Moving forward, ITX clients have the opportunity to permit analysts within the ITX QA Practice to use these new best practices. These analysts can then use the regained time to drive overall product quality; they’ll conduct exploratory testing, which is the kind of testing that humans are best at.
