Software Testing Unleashed · 2026-05-21 · 25 min
Key moments - from our scoring
Substance score
56 / 100
Five dimensions, 20 points each
Mitko Mitev brings three decades of QA experience to challenge the recurring fear that AI will eliminate testing roles. Drawing on his work with ISTQB, he demonstrates how large language models and AI agents can dramatically accelerate labor-intensive tasks like test case generation, test data creation, and reporting - potentially reducing effort by 20-40% - while leaving strategic decisions and validation firmly in human hands. The conversation spans practical applications across the testing lifecycle: using AI for test planning based on risk assessment, leveraging techniques like equivalence partitioning within trained models, employing agents for exploratory testing, and automating defect clustering and root cause analysis. Mitev addresses the scripting debate directly: even as AI generates test scripts from natural language, testers must understand what's being generated to verify correctness. He positions the shift not as replacement but as role transformation, with testers moving toward more creative, decision-making work while mechanical execution moves to AI. He recommends ISTQB's new syllabi - Testing AI (how to test AI products) and Generative AI in Testing (how to use AI in daily work) - as essential starting points, noting that teams must begin learning now or risk falling behind as software generation accelerates.
No; AI will shift the scope of testing work toward more creative and decision-making tasks while automating mechanical activities. Throughout the past 20 years, similar predictions about test automation, Agile, and DevOps replacing testers proved unfounded, and the pattern repeats with AI.
AI excels at labor-intensive, mechanical tasks like generating large volumes of test cases, creating test data, clustering similar defects, and automating reporting and summarization - potentially saving 20-40% of effort on these activities.
AI currently struggles with context, business logic understanding, and predicting real user behavior and actions - all areas that require human oversight and control to ensure test effectiveness.
Yes, someone must verify that AI-generated scripts are correct and accomplish the intended job; knowing basic scripting helps testers understand and validate what the AI produces.
ISTQB offers two syllabi: Testing AI (how to test AI products) and Generative AI in Testing (how to use AI in daily work), with the latter updating every 3-6 months to keep pace with rapid changes.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode covers practical applications of AI in testing phases (planning, design, execution, reporting) with some concrete use cases like test data generation and defect clustering. However, much of the content recycles familiar narratives (role evolution, not replacement) and lacks depth on how these tools actually work, trade-offs, or failure modes. The 20-30-40% time savings claim is stated without evidence.
you can probably do it in a week. So performance wise You know, it's it's helping a lot.
you can use it uh to cluster uh similar defects then put on uh the the uh projectors you know the uh importance of the different areas of the software that generate more defects
The core thesis - AI won't replace testers, just shift their work - is a standard industry take by 2024. The comparison to previous waves (test automation, agile, DevOps) is reasonable framing but well-worn. Little contrarian thinking or first-principles analysis. The suggestion that business analysts can now write test cases via natural language is presented as novel but is not.
every let's say five years there is some buzzwords coming on the market. Like first was test automation, it will replace testosterone. Then it was what was it after that? Uh agile.
the role of the testers will shift Towards more creative tasks, more thinking and decision making
Mitko Mitev has 30+ years in QA and is deeply involved with ISTQB standards-setting, a legitimate authority. However, the transcript reveals limited hands-on current operational experience; he speaks mainly as a standards body representative and conference speaker rather than as an operator running a test team at scale today. His insights are more curator-level than practitioner-level.
With more than 30 years of experience in the software quality assurance industry, he's one of the leading software test experts.
I know him from the International Software Testing Qualification Board, ISTQB, where he is very active
Few concrete examples or metrics. The 20-30-40% effort savings claim is vague and unsourced. Test data generation and defect clustering are mentioned but not demonstrated with real numbers, tool names (beyond generic 'MCP', 'agents'), or timelines. No specific companies, projects, or measurable outcomes provided. Heavy on what *could* be done rather than what *has been* done.
You can save probably th twenty, thirty, forty percent of the effort and time.
if you have to generate 4,000 test cases, for example, for some complicated software. Um probably a human will need a a month. And with AI support you can probably do it in a week.
The host asks reasonable setup questions but rarely probes deeper, challenges claims, or requests evidence. When Mitko makes assertions (e.g., time savings, AI's limitations on context), the host moves on rather than asking for examples or pushback. The conversation is collegial but surface-level, more aligned chat than investigative interview. Few sharp follow-ups that would pressure the guest to be more specific.
Yeah, yeah. And is the AI nowadays in in charge of of using our test techniques like the accuration partitioning or and this stuff
And uh well I can imagine what comes to my mind is that we we have all this by reporting to do a more uh specific to the different audiences
Computed from the transcript - who did the talking, and the words that came up most.
From test planning to defect clustering: where AI already saves you 30% effort Free e-book: The 7 success factors of software testing. 25 years of project experience in one 33-page workbook, now also in English Get it for free "People should stop asking on interviews what's the difference between class and object. You should probably ask: What is MCP?" - Mitko Mitev This time I talk to Mitko Mitev, about how AI is reshaping our work as testers, without replacing us. Mitko shows exactly where AI tools save real time across test planning, test case generation, and exploratory testing, and why human expertise remains non-negotiable for context, business logic, and validation. We go into the shift from writing scripts to instructing agents in plain language, how ISTQB's new AI syllabi prepare testers for what's coming, and why waiting another year to explore AI might already be too late. With over 30 years in software quality assurance and more than 20 years as a Project and Test Manager, Mitko Mitev is recognized as one of South East Europe’s leading software testing experts.
Transcribed and scored by The B2B Podcast Index.
Every let's say five years there is some buzzwords coming on the market. Like first was test automation, it will replace testosterone. Then it was what was it after that? Uh agile.
Uh then it was uh DevOps. Uh now it's AI. Everybody's you know saying that uh we will get rid of QAs and testers but uh we're still here You know, AI has at this stage of development of AI has a bit of a um challenge with uh context and business logic and also b behaving as a user, you know, predicting what the user actions will be. that's not that easy for uh artificial intelligence.
This has to be still in control of humans. People are thinking maybe w in the future we will not need um that kind of skills like knowing TypeScript or Python, you know, maybe that's not uh important anymore. Um I still think that's uh not going to happen because um even if it is possible for AI from playing language to generate scripts. Somebody has to make sure they're correct and that they're doing the the you know the job that they have to do.
AI tools have better English than mine so welcome to Software Testing Unleashed. Podcast for testers, developers and test automation engineers and all the people in the software development process who want to create great amazing quality software. My today's episode was recorded at TestWarez. A great Testing conference in Poland.
The days were full-packed with great quality content and I had a great time discussing and chatting with the Polish and the international software test community. If you haven't been here, you should definitely go to one of the Next Test Warriors conferences. You find the link to the conference in the show notes. My guest today is Mitko Mitev.
With more than 30 years of experience in the software quality assurance industry, he's one of the leading software test experts. In Southeast Europe. I know him from the International Software Testing Qualification Board, ISTQB, where he is very active, participating in the efforts for setting, improving, and popularizing international standards. We talked about how How can AI enhance our testing business?
Where can we use AI in the different activities of our test process? What are the benefits and what are the pitfalls? I'm happy that Mitko shared his experiences here in the podcast and now enjoy. Hi Mitko, great to have you here on the show.
Thank you for inviting me. Uh thank you very much. Yeah, it's it's a pleasure for me. We know each other so long from the ISTQB stuff and now We are here in uh Wisla at the Test Warriors and uh meeting at this conference and now we have time to make the podcast episode.
I I read your abstract in the for the in the agenda from the conference and uh you're dealing with a very popular uh topic nowadays, uh how AI impacts our testing uh Yes, um it's a hot topic right now. You know everybody's talking about uh artificial intelligence and uh um as you see in the last three, four years it's going into all directions in life people start using it for different purposes. Um in different uh professions and business areas And of course it's uh getting more and more integrated in the IT uh industry.
Um my talk was um focused focused on uh how it can enhance testing especially because we are testing people you know and uh and I I wanted to address uh address it uh how we can use AI tools in different uh uh phases of software testing process and you know I'm uh I person so um I try to structure my um talk uh around the phases that we have in uh STQB foundation level let's say you know test planning test uh design analysis test design execution reporting um so I wanted to uh tell the people how uh easy is to use different tools in different phases.
and um there are a lot of AI tools right now on the market and I think there will be even more in the future. Uh some of them are mature, some of them are not so uh very well developed yet but uh Things are changing fast. And um uh if you start using AI, you can um make your life very not very easy but easier. You can save probably th twenty, thirty, forty percent of the effort and time.
Some uh labor-intensive tasks can be done quickly. And uh of course focuses on test automation because a lot of people are doing test automation as well. I can help there in that area. as well and uh nowadays uh with the coming uh agents on the market it's becoming even more interesting because uh they're more or less uh becoming uh autonomous.
Mm-hmm. And they can you can just tell them what to do and they uh they're going and do it. Of course, um you cannot over you cannot over rely on the AI tools, you have to have people to control it, train it and and verify the results in the end. But indeed the result can be uh satisfactory Yeah , uh it's still it's still uh crucial that we have our basic testing knowledge uh to to verify the the output.
Yes, absolutely. I don't people are afraid that they're gonna replace us in all these tools, but I don't believe that. I think uh the scope of the work of the testers of the QAs will change, but um we will stay in control. Yeah and we can say that the process will be AI assisted but not completely replaced by AI.
I think uh most of the uh mechanical tasks and out out uh automation tasks will be probably done by AI, but you know AI has at this stage of development of AI has a bit of a um challenge with uh context and business logic and also behaving as a user you know, predicting what the user actions will be, that's not that easy for uh artificial intelligence. This has to be still in control of in in real life uh we c uh we can now use it uh and to to support us. Um yeah I give some examples about uh let's say how to create a test plan.
But My intention was to focus more on um on labor intensive tasks. Like in most cases these um areas where creating test cases for example and test data. You know, especially for the test data. It can generate a lot of test data very quickly by just simple explanation what exactly you need.
For example, you can say give me two hundred different users with bank accounts and uh limits on transactions and all this stuff and then you can generate data very quickly. Also for the test cases, the responsibility, if you have documentation and requirements or use cases you can connect your confluence or some other tools and tell uh LLM uh can you generate test cases based on the requirements that I have already. So it's becoming really interesting time for uh people who like uh new technology, uh early adopters as I say.
Um so those kind of heavy tasks, let's say if you have to generate 4,000 test cases, for example, for some complicated software. Um probably a human will need a a month. And with AI support you can probably do it in a week. So performance wise You know, it's it's helping a lot.
Yeah, yeah. And is the AI nowadays in in charge of of using our test techniques like the accuration partitioning or and this stuff, or is it not that useful in this context? Uh it depends on the context of the model. Um usually um you know When you have to use um some kind of uh large uh language model, for example, you train it in advance with uh data and uh in your business domain and uh a lot of uh companies have already prepared uh AI uh AI models with uh test data and testing and of course if you use those kind of techniques that you're mentioning Um it's not a problem to use the techniques and you can just say do this and do that and uh use this uh test technique and that's okay.
Of course you can also say choose y you choose uh which techniques to use and uh you know STQB is a standard, uh de facto a standard in the world. So even if you don't put a lot of context uh behind in in the model. Even uh by public information that is available, techniques are also very broadly uh known and people can use them. Usually no problem.
Yeah, yeah. Especially with the exploratory testing, there this in this area AI can also help a lot. Um of course, you know, um people uh uh believe that uh with exploratory testing um mostly senior people should be uh should be uh busy but um You can instruct your AI to do uh to go different directions in unknown sequences of the process so he can simulate different activities that even people cannot imagine. Yeah, yeah, yeah.
It c it can be a good combination to our critical thinking from the testing with the input from the from the AI stuff. Yes, uh that's the way I see it in the future. I s I s I see it like a combination of AI and uh people. Yeah.
And uh of course as I said the role of the testers will shift Towards more creative tasks, more thinking and decision making, validation, verification and training of the model, because AI is all also has to be controlled trained and controlled and verified results um and more um Mechanical tasks, automated tasks will be shifted towards AI. You also mentioned before that uh the topic of Test automation uh and and uh what what do you think where where are we now in the AI context with test automation and what what what is the direction this automation stuff will go in the next years?
Um So at the moment there are a lot of possibilities and um you know popular um tools for test automation uh integrating also AI uh support and and you know um there are popular concepts like MCP right now that you can connect uh LLM with a test automation tool and basically automate the whole process. Also with the coming on uh agents on the market, you can instruct um the AI to generate your uh native language s instructions like test cases in native language. So some people think the future of the scripting, like writing scripts is probably gone because if you can actually from normal language go directly to test execution without going through the scripting, that's an interesting concept.
Of course it will go through some scripting, but it will be automatically generated. So people are thinking maybe in the future we will not need um that kind of skills like knowing TypeScript or Python, you know, maybe that's not uh important anymore. Um I still think that's uh not going to happen because um Even if it is possible for AI from plain language to generate scripts, somebody has to make sure the correct and that they're doing the the you know the job that they have to do.
So um I still think people have to have some serious knowledge about uh But may maybe it can be a good uh a good interface for the business people to uh what we try with the local tools nowadays that they have and like in like in jet uh uh plain text they can give into an and tool and and then get a a draft for a for a script which can be used by the the test automation engineer or something. Exactly. That's that's actually the added value. of uh using AI because it widens the the uh group of people who can actually do this kind of uh activities.
Um before um only testers and test automation specialists could do this part and now with the introduction uh introducing the plain language uh uh uh test combinations and instructions, business people can be included in this group. So this will definitely add value. Uh the projects will benefit from from this opportunity because with the plain language it can we can instruct the two what to do, that's really uh a beautiful opportunity. Yeah.
Do you think uh that it's There's also a way to use it more in the in the test management stuff or in the reporting stuff uh future? Um yes, the test management part is interesting because um Um as I said you can actually plan. There are agents called planners, so you can actually do a good planning and based on a risk assessment uh which areas we have to test more and which areas we have to test Less also um AI has an influence in the defect analysis, then you can use it uh to cluster uh similar defects then put on uh the the uh projectors you know the uh importance of the different areas of the software that generate more defects for example and then focus the testing in that area too So with clustering of defects and risk analysis and root cause analysis, also with AIs you can analyze the local files and this can give you a lot of information for the reasoning, the the cause of the um the effects and root cause will be easier if you use uh AI.
And at the end with the reporting Uh it is uh if you use AI then you can easily generate a report and summarize it and make it easy for C level managers to read it and say, okay, this is what happened, this is why it happened, and this is what uh our recommendation is if we're going live or not. So the process of management the whole testing uh activities will be easier, I think. Yeah. And it uh well I can imagine what comes to my mind is that we we have all this by reporting to do a more uh specific to the different audiences we have in in in our company so the eye can help us to to go uh into the direction of the different stakeholders too.
Yes absolutely you can You can specify different kontext avarising and reporting the information um intended for business people, intended for technical people, for DevOps, for operational people. So yes, this is really Jusfå and also as I'm not native speaker English um I can say uh AI tools have better English than mine. So for sure the reporting will and the planning will be uh in better English than than some of us. Yeah, yeah.
So a huge support uh possibility for us in the testing industry in the in all the different phases, but not replacing us. very crucial to know because a lot of people I think have nowadays are afraid that their testing job is is gone but as you as I understand it's changing. It's not You know, I've been in uh long years in the industry and every let's say five years there is some buzzwords coming on the market. Like first was test automation, it will replace testosterone.
Agile. Uh then it was uh DevOps, uh now it's AI, uh all the low-code stuff too. You know, s saying that uh we will get rid of KAs and Tesla's uh but uh we're still here. You know.
And I don't believe it will happen again. It will shift the um skill set and the focus because Um as a friend of mine said, um people should stop asking on interviews what's the difference between class and object. When you hire uh testers, you should probably ask what is MCP? Do you know how LLM works?
Do you have an idea how agents work? So really the skill set and the focus will be shifted but we will be needed. Um For me the future is uh AI assisted but not replaced. Yeah, yeah, yeah.
And and maybe it's even more work for us to do in the next years when we think of the wipe coding stuff and all the stuff we're just generating software and code with unknown quality. Yeah. Somebody has to test it. Well AIF affects also software development, not only testing.
I mean nowadays i it is possible for, you know uh software development teams to generate software very quickly and you can't really cope if if If the development is going that direction, you can't really cope with uh human testing only. You really need uh AI support in this uh activities as well. So um I see it as a combination and the whole teams are moving in that direction. And um You're right.
The speed of generated software producing software is enormous and we have to deal with that somehow. Um some people say I will uh study AI but next year. I would recommend everybody to start now. Yeah.
Yeah. Yeah. So, you know, if you uh Wait for three months you may be late already. Yeah, that's true.
It's it's a a thing we have to deal now with. Yes, yeah. Yeah, you don't you don't have to postpone it. You have to have to start now.
I'm a big believer, you know. Yeah, yeah. And you are also a a very uh integrated, respected uh foundation in the in this TQB area. So you're dealing with I ISTQB stuff over decades, I would say, now Uh more than two decades.
Yeah. Yeah, yeah, and and you are worr one one uh is very very deep inside uh this this stuff. And uh how when I want to learn testing in the AI area now, what what is the IST QB bringing? there to support me as a tester.
Right. So I will recommend people to follow the process that ITQB has described. We do have two syllabi, different syllabi in the area of AI. One is testing AI , which is on the market probably.
around two years. And the other one is brand new since July. It's called Gene AI In Testing. Which is very modern.
Both of them are very interesting. One of them is how to test AI products. The other one is how to use AI products in the day-to-day work. Däfter you first have to have fundation.
Obviously, because in för oss this is really the basis. of the software testing knowledge. Um I do believe uh the interest in this two syllabus in the in the next months and years will be enormous because it's really a key uh key area of knowledge that we need to support. And for the GNA especially because it's it's changing so fast.
We um intending to ch to update it every three to six months to catch the new trends and and tendencies. So I would recommend to everybody um to take a look on the syllabi AI syllabi of ISTQB. Okay, I think it would be a good starting point to get absolutely and it we know the ISTQBS as a standardized body is uh uh doing a good stuff there is all it's solid work for the foundation body of knowledge is uh it's a good place um I said already we are de facto the standard in the industry.
So and um I know the syllabi they're very well structured. uh knowledge is very well structured. So it's a good uh starting point for people to get into the testing with AI and yeah and the Mitko, thank you very much for being here in the show that you uh shared your insights, your experiences with the AI stuff. I think it's the a topic we have to deal as a tester and QA people for the next now and for the next months and years.
Yes, it is it is a key uh key area that we have to put some efforts. Yes, yes, that's true. Thank you very much for Very happy that you were here and enjoy the rest of the conference.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.
225: Every Website Already Has An Agent Experience And Most Are Bad With Netlify CEO Matt BiilmannNo Hacks · on MCP (Model Context Protocol)95 / 100
Continual Improvement, Nonconformities, and Corrective Actions | Interview with Carlos CruzSecure & Simple · on Root cause analysis85 / 100
Mozilla CTO: Why Most Enterprises Don't Control Their AICXOTalk · on MCP (Model Context Protocol)83 / 100
Continuous Quality for AI Coding: ChecksumSourceForge Podcast · on Test automation81 / 100
EP 117: 70% of Teams Are Using AI With Zero Oversight. Here’s What HappenedEmbracing Marketing Mistakes · on MCP (Model Context Protocol)80 / 100
Systems Over Signs: How James Ferrell is Engineering Out Workplace HazardsThe Canary Report: Safety & Risk Management · on Root cause analysis79 / 100
ChatGPT Use Cut Student Cognitive Capacity, Study Finds - Graziela Tonin57 / 100
Post-Agile: What Organizations Actually Need Now - Michael Mahlberg69 / 100
Critical Thinking: The Skill AI Cannot Replace in Testing - Tara Walton49 / 100
Strategy First: How AI Enters Regulated Medical Labs - Alexis Savkin54 / 100
Why COBOL Developers Prefer Writing Tests in Java - Szymon Wałachowski, Bartosz Filipek73 / 100