Long before ChatGPT burst onto the scene, ITX shared a blog post that outlined The Seven Perspectives of Digital Quality – among them we included app and system security. How secure is your software in terms of data integrity, coding standards, and infrastructure? How well does it stand up to access breaches, threats, and other vulnerabilities?

AI is helping product teams do research, design, and build software. But faster development doesn’t reduce the need for security or sound engineering judgment. In this latest Topics in Product Series (TiPS) episode, Jonathan Coupal, ITX Vice President and Director of Security, and Besong Tabenyang, ITX Developer and Data Engineer, bring hands-on security and AI development experience to this critical conversation. Perhaps Besong said it best: “AI hasn’t removed the need for software discipline. Instead, it increased the penalty for overlooking it.”

Here are a few tips for how your team can use AI to accelerate product development while building secure, trustworthy products.

Proceed with Caution To Protect Sensitive Information

The first step in secure AI development is understanding what happens to the information we feed into the models we use, Besong says. For example, product teams should never assume an AI tool protects sensitive business or customer data. Instead, they need to understand its privacy posture, data retention, isolation, and administrative controls.

Enterprise-grade tools can provide protections that public AI tools may not. As Besong puts it, “If you put confidential business information in a public AI tool, don’t assume it stays private.” Jonathan adds, “AI is kind of dumb, so you have to be really clear about how you state your requirements as you’re going into the design process. Telling the AI to include security features as part of your build cannot be left as an assumption, it has to be declared.”

Shift Security Left To Save Cost, Deliver Faster

Maybe you recall Product Momentum episode 131 / Shift Left: Integrating a Security Mindset Early in the SDLC – in which security expert Paul Connaghan urged product builders to include Security as part of the product’s design, well before QA or post-release. In this conversation, Jonathan points to an article referencing IBM research that shows how addressing security during design can significantly reduce both cost and delivery time.

AI can also help product teams identify potential threats during discovery and design, he adds. “There’s two types of security problems in a product,” Jonathan continues. “One is engineering problems. We consider those bugs, but security design problems can’t be fixed as bugs because they’re not bugs. Sometimes, it’s a consequence of how someone could misuse a feature.”

Apply Human Judgment To Preserve Engineering Discipline

AI is really good at generating code, tests, documentation, and architectural recommendations – with remarkable speed. But it’s precisely that speed that makes it easier to skip the human reasoning that supports good engineering decisions. Teams spend less time identifying edge cases, validating assumptions, and evaluating trade-offs.

“AI increases code output,” Besong says, “but if our review discipline doesn’t increase with it, we get faster production of unexamined work. AI didn’t remove the need for software discipline. Instead, it increased the penalty for skipping it.” Jonathan places the responsibility firmly on the shoulders of human decisionmakers: “Every time an AI recommends a particular design choice, it’s up to human judgment; and that’s where people with your experience still matter is that bringing that judgment to bear.”

• [04:13] AI is kind of dumb, so you have to be clear about declaring your security requirements as your work through the design process. Telling the AI to include security features as part of your build can’t be left as an assumption, it has to be declared. (Jonathan Coupal)
• [08:23] Public AI solutions are built mainly for convenience and broad adoption. But you have to be mindful of what type of data you’re putting in there. (Besong Tabenyang)
• [11:02] Security information about vulnerabilities is so black and white that the models have been trained on this. The level of expertise that you’re granted by using AI to do security work in the design phase is so good that there’s almost no reason not to use it. It gives you a superpower that you don’t get from a lot of other types of ideation assistance; the quality is so high. (Jonathan Coupal)
• [13:45] Security isn’t just a governance layer. It materially improves how you build because it enforces clear system boundaries, better failure modes, and more trustworthy products. (Besong Tabenyang)
• [17:43] It’s pretty easy to declare essential security requirements. It’s even as simple as take a security framework or the compliance rules of the company, feed them into the model as additional inputs as you’re doing the build and say, ‘in addition to taking into account these compliance or security constraints, let’s talk about how to build out this feature.’ So we don’t even have to say it out loud. (Jonathan Coupal)
• [18:48] AI has increased the volume of code output, but if our review discipline doesn’t increase with it, then all we get is faster production of unexamined work. AI didn’t remove the need for software discipline. It, instead it increased the penalty for skipping it. (Besong Tabenyang)

Stay tuned for our next TiPS episode, as Sean and Dan invite ITX experts to explore the security implications that unfold when AI is integrated into the product itself as a feature or agent, granted rights and privileges as an actor in the system.

Podcast: Play in new window | Download

Dan Sharp [00:00:08] As a product person, we’re often told, ‘hey, go and use AI.’ But maybe before you start typing all of your innermost thoughts in, pause and make sure the tool you’re using is actually acceptable to your company. We have all these great ideas, we’re feeding them into AI, right? What’s happening when that data goes in? Consumer grade material, what’s going on behind the scenes and why should I be concerned or paying attention to that?

Besong Tabenyang [00:00:33] Yeah, great question. Well, if you put confidential business information in a public AI tool, don’t assume it stays private. It’s likely that that data is being used to improve the model, and it’s leaving the control of your organization. It doesn’t necessarily mean the model is bad. It just means that the usage is here. The data handling and the privacy posture is not suitable for sensitive business information.

Dan Sharp [00:01:06] Why does factoring in security early actually make us faster in the long run and not slow us down?

Jonathan Coupal [00:01:12] Well, because the data supports it, not only does it demonstrate mastery to be able to open discussions about non-functional requirements, like security and performance, good architecture, that type of thing, but the work has actually been done to demonstrate that this is true. IBM did a study a number of years ago, but they found that each stage of shifting security leftward in the product development process. Um in improves the quality in ways that decreases the price and increases the speed.

Sean Murray [00:01:48] I’m thinking back to the threat modeling training that you’ve done for the product team, John, and so we need to be thinking about that and like the initiation phase and then the discovery phase before we get to develop and deploy, before we actually start building something.

Jonathan Coupal [00:02:00] It turns. Out like security information about vulnerabilities and stuff like that. It’s so black and white that the models have been trained on this. What happens is the level of expertise that you’re granted by using AI to do security work in the design phase is so good. That there’s no reason, there’s almost no reason not to use it for you guys. Like it gives you a superpower. The other thing to think about too is there’s two types of security problems in a product, right? One are engineering problems. We consider those bugs, but security design problems can’t be fixed as bugs because they’re not bugs, right. They’re just part of the feature and the name AI model might bring that as unidentified as an identified problem with the way the products are being built or or an unhappy path, right? We call it abuse, an abuse story.

Sean Murray [00:03:11] Product Momentum is excited to host the latest installment of our Topics in Product Series, where we bring ITX’s in-house experts on the podcast to tackle tough subjects and give a little insight into how ITX delivers user delight and promotes product advocacy. Today, with our Vice President and Director of Security, Jonathan Coupal, and Developer and Data Engineer, Besong Tabenyang, we will be addressing this rain cloud that’s kind of like hovering over this new age of product builders and UX engineers. So now that. Product and UX roles are being asked to deliver more with AI tools? What are we missing? What new risks are we taking on without the time or training to handle properly? What assumptions have we made that we are like consciously or not ignoring? We’re grateful to have Jonathan and Besong here to help us navigate this topic and we’re going to dive right in. So Jonathan, when a product team brings AI into their build process, what’s an assumption about that AI tool that they don’t realize is putting them at risk?

Jonathan Coupal [00:04:13] Yeah, that’s a great question. Um, our clients make a lot of assumptions about the tooling we start using. And they figure that we’re the experts. So they’re going to figure that. Are kind of doing the right thing. There’s a couple of things out there that is important to consider. One is hygiene, right? The client is assuming that we are using an AI tool in a way that’s still protecting their intellectual property. I E it’s not in training mode and it’s important for us. To make sure that we do the diligence to check, to make that any AI tool that’s integrated into our existing tooling has that option activated or even available, right? Sometimes the vendor says, oh, we’ve got AI in the product now, and they may not necessarily go to the extra extent of actually letting us know how they’ve done the build or how they done the integration. And since we’re actually developing a new product to bring to market, we wanna maintain our client’s competitive advantage by making sure that we cross those T’s and dot those I’s. I think the other one that’s important is they’re going to assume that security is baked into the tooling, right? In the same way that when you buy a house, you assume that the fire department is gonna come when it’s on fire, which may or may not be a reasonable assumption. They want to assume that we’re doing the security thing. And that may or may not be the case. I know AI is kind of dumb, and you have to be really clear about how you declare your requirements as you’re going into the design process. And telling the AI to include security features as part of your bill, this can’t be left as an assumption, it has to be declared.

Dan Sharp [00:06:03] I think, yeah, you, you’re saying something important there. And right there, I think there’s another piece here, right? I guess as a, as a product person, right. We’re, we’re often told, Hey, go and use AI, but maybe before you start typing all of your innermost thoughts in pause and make sure the tool you’re using is actually acceptable to your company, right, like maybe don’t go use free ChatGPT and enter all of you innermost secrets, right to that end, you have the song, right you have developer in your title. You are doing this on a day to day basis. Yeah. I love, I love to hear, you know, concrete example of product team, right? We are, we have all these great ideas or feeding them into AI, right. We got our competitor analysis from a data standpoint, right, what’s happening when that data goes in, yeah, consumer grade material kind of what’s going on behind the scenes and why should I be concerned or paying attention to that?

Besong Tabenyang [00:06:54] Yeah, great question. Um, well, if you put confidential business information in a public AI tool, don’t assume it stays private. It’s likely that that data is being used to improve the model and it’s leaving the control of your organization. It doesn’t necessarily mean the model is bad. It just means that the usage tier, uh, the data handling and the privacy posture is not suitable for, um, business sense, sensitive business information. You shouldn’t be using an enterprise-grade AI tool, which offers business-grade privacy guarantees, such as data retention and data isolation, admin controls and logging, and it doesn’t train on the company’s customers’ data. And it’s built to handle business-sensitive information.

Sean Murray [00:07:46] I remember having early conversations with Jonathan about like, what free AI tools can we use? And the answer was a very clear, none of them. And if you are, or if any of your team members are, make it stop now, right? So I like to, you called that like enterprise grade, you know, enterprise grade like licenses and models, right.

Dan Sharp [00:08:09] Yeah, I think there’s like an analogy here. It’s like, you know, you’re, youre trusting information as someone. Like, is this someone that you think is going to go blab to everybody else, your secrets, right? Like, think of it that way. It’s, like, do you actually trust who you’re talking to?

Besong Tabenyang [00:08:23] Yeah, and public AI solutions are, you know, mainly for convenience and, and broad adoption, but you have to be mindful of what type of data that you’re putting in there.

Dan Sharp [00:08:37] Absolutely. Right, so Jonathan, I have one for you. I think, proud of people. Again, you’re always told to go fast and, yeah, having those conversations with compliance and security. Ah, you are slowing us down. You’re slowing down delivery. But hey, but also, use that. Go fast. Why does factoring in security early actually make us faster in the long run and not slow us down?

Jonathan Coupal [00:09:01] Well, because the data supports it. Not only does it demonstrate mastery to be able to open discussions about non-functional requirements, like security and performance, good architecture, that type of thing, but the, the work has actually been done to demonstrate that this is true. IBM did a study a number of years ago. Um, we could probably include a reference to that study in our, or in our show notes, if you want to do that, but they found that each stage of shifting. Security leftward in the product development process improves the quality in ways that decreases the price and increases the speed. Their estimate was that by addressing a security concern in design makes that particular feature that security feature about 30 times cheaper than fixing it in production. Right, so Every delay that you introduce to wait to deal with security, right? Like if you push security off into QA, it’s probably about 20 times more expensive than simply addressing it in this night. And then once you put it into production, it even more expensive because then you actually have to take it out of production again. You either have to roll it back or you have additional costs that are introduced to the client or the client’s customers at the end point. So, so for the customer experience, yeah, there’s a There’s an increase in time and design, but it pays back with interest.

Sean Murray [00:10:32] Back to the threat modeling training that you’ve done for the product team Jonathan and you’ve always said that these are the things need to happen earlier in the process so really even though we’re using AI now nothing’s changed but we’re just reinforcing that priority right like we need to be thinking about threats you said shift left so we need to be think about that and like the initiation phase and then the discovery phase before we get to develop and deploy before we actually start building something have we thought about the actual threats that we’re Introducing

Jonathan Coupal [00:11:02] Right? Yeah, and the cool thing is like the training data is so clean for security stuff Like there’s not a lot bullshit out there on security information about software products, right like there is with like Information about food or nutrition, right? They’re if eyes are trained on what’s in the world about nutrition You’re gonna get rando stuff right because that’s what they’re ingesting. It turns out like security information about vulnerabilities and stuff like that. It’s so black and white that the models have been trained on this. What happens is the level of expertise that you’re granted by using AI to, um, to do security work in the design phase is so good that there’s no reason. There’s almost no reason not to use it for you guys. Like it gives you a superpower. That you don’t get from a lot of other types of, um, ideation assistance because just the quality is so high.

Dan Sharp [00:12:05] And I think, you know, if we’re talking about, you know, moving the security analysis forward, yeah, it comes to something that Sean and I have talked with in other episodes, the idea of the pre-mortem, right? And I think as product people, sometimes we’re so focused in a pre-mortem on how is a thing going to fail from a usability standpoint or from a business value standpoint, but we should absolutely be baking in how is it, could it fail from security standpoint, right? Cause it might be usable, business values would go down if you’ve got a huge security vulnerability baked into it that you’re You know, a hundred X cost.

Jonathan Coupal [00:12:37] The other thing to think about too is there’s two types of security problems in a product, right? One are engineering problems. We consider those bugs. But security design problems can’t be fixed as bugs because they’re not bugs, right. They’re just part of the feature. Like, if you are building a social networking system, you need to think of the possibility of somebody stalking their ex in the product. That’s not a bug. Right, but it’s still a part of how you secure the product to make it safer for your users, right? And the name AI model might bring that up as an identified problem with the way the products are being built or an unhappy path, right? We call it an abuse story or a misuse story.

Sean Murray [00:13:28] Besong, you are actually building AI tools, right? So you are using AI to build tools and you are building AI, you actually building a AI products right now. From your seat, what changes in how you actually use AI day to day, once security is a part of your process instead of an afterthought?

Besong Tabenyang [00:13:45] Yeah. So when security becomes part of AI from the beginning, the work changes at the level of everyday engineering judgment. We stop asking only does this work and start asking, is this safe? Is it bounded? Is it audible? Is it appropriate for the data and actions that are involved? So that changes everything from the data that we expose to the model. Um, whether we use a hosted or self-managed, um, system, how we protect against prompt injection, how much autonomy we give to agents, you know, what we log and, you, know, where we introduce human approvals in the process. So security isn’t just a governance layer. It’s, it materially improves how you build because Um, it enforces clear system boundaries, better failure modes and, uh, more trustworthy products. More trustworthy.

Sean Murray [00:14:43] Products is what we are actually I think part of like the underlying theme of this episode is more trustworthy products right because we’ve been given these tools and everybody’s very excited about the speed that we can deliver with right I think we you know we’re in town halls with clients or we are you know I’m on the road I’m in the road meeting new clients and like eventually you get back to like hey actually the real goal here is delivery speed right we want you to use these tools because we want to deliver faster. Not because like We’re looking for new and innovative ways to create customer delight, right? Or to like find like new markets to enter anything like that. It’s just like ship faster, right. So we want to accomplish that, but we want to still create trustworthy products. All right, Jonathan. I kind of talked about, I just kind of went into, you know, when we are even working with brand new clients. They’re expecting us to use AI to build products. So we’re saying, we can essentially say that using AI to building products is a table of stakes now. Whether it’s actually writing your code for you or whether it’s doing some of your initial research or whether its just doing your documentation, it is a part of your process. Everybody’s doing it. So from a security standpoint, what’s a best practice that you think is getting skipped as teams adopt AI really quickly?

Jonathan Coupal [00:16:05] Yeah, that’s the tough one. I think the use of the AI might lead us to not be skeptical of the happy path. Because we’re assuming that the AI is taking that into consideration, right? I mentioned the abuse or misuse cases as a formal way of declaring that, but, but we’re, assuming that, the AI’s is taking into account. I remember seeing another study. They suggested that. By just adding the word securely to the end of a prompt, improved the quality of the code, like immensely in terms of adding some additional secure code when somebody was actually doing the code build. It’s the same thing here with design, right? Like how would we build this feature? No, how would build it securely and perform it, right. So just taking the time to. Remember that we need to call out those parts of the product that are important that nobody thinks about bringing up because they assume that the AI is doing that work for us and it’s not, I mean it’s, it’s an idiot savant, right? Like it’s only able to do well what we tell it to do.

Dan Sharp [00:17:20] And all of that, I think you hit upon something that’s really important across the board in product development. And it’s like a knowing knowledge bias, right? And if you’re assuming that something’s being secure and you’re not explicitly saying it, then like that’s going to cascade down. So I think that’s great advice for folks listening, you know, prompting, make sure you explicitly say, do it securely. You’re going to get a much better output out of it.

Jonathan Coupal [00:17:43] Yeah, even as simple as take, you know, take a security framework or take the compliance rules of the company, feed them in as additional inputs as you’re doing the build and say, in addition to taking into account these compliance or security constraints, let’s talk about how to build out this feature, right? So we don’t even have to say it out loud. We can actually document it as like an incoming markup language file that we use when we build everything for the customer. And then it becomes an implicit part of the way we do that work for that client.

Dan Sharp [00:18:18] Right. So in your prompt, right, mention securely, include your company policies as context and also maybe line and don’t get me fired as part of this.

Jonathan Coupal [00:18:28] Thank you very much.

Dan Sharp [00:18:33] I’m curious, you know, from your standpoint, I know you’re in there day-to-day of talk about how you’re building products. What’s something that you’ve noticed that teams are no longer doing with their own processes now that AI is part of kind of their day-to-day evolution.

Besong Tabenyang [00:18:48] From my perspective, what I’ve seen reduce is the work that was done before code arrives. AI makes it so easy to develop complete code, unit tests, and architectural suggestions, and even documentation that teams can go straight from idea to artifact, and without spending time on core software fundamentals such as Identifying edge cases, validating assumptions, and trade-off analysis between components. So it’s not that the AI output is bad, it’s teams can quietly skip these steps that are used to create engineering judgment. So, and another thing I’ve seen is deeper code reviews reduce. So like code reviews now are mainly like this little plausible because the code is there, it’s complete, it has the test, it hasn’t documentation, but we’re not really asking ourselves, you know, does this fit our system? Does this fit or, um, maintenance standards in our security model? And, you, know, that’s the bottleneck shift, like the volume of code that AI, AI can increase AI has increased the volume of code output, but if our review discipline doesn’t increase with it, then we get faster production of unexamined work. Yeah, so I’ll put it this way. AI didn’t reduce the need or didn’t remove the need for software discipline. Instead, it increased the penalty for skipping them and the bottleneck has shifted from code production to actually verifying the reasoning of the output of the code, verifying, the reasoning, of the model. And teams who do that well are teams who kept the software disciplines while accelerating execution with AI.

Dan Sharp [00:20:58] I think, yeah, you hit upon something that, that we’ll kind of keeps me up at night from an, from the AI world standpoint is how easy it is to have AI outputs that look amazing, right? Everybody is asked to go a million miles an hour and it’s so easy to be like, yeah, right. That output’s great. Let’s roll. Right. Yeah. So you mentioned it from the code standpoint, you know, we need to be more diligent as the outputs are getting better from a product standpoint. Right. Like we talk about how, Hey, Yeah. AI might be giving you decision recommendations. It’s still your job, right. To make the decision. Yeah. Jonathan Bassanio for either of you, you know, how, how as lowercase ‘P’ product developers, can we fight kind of back against that pull to let the AI do the thinking for us?

Jonathan Coupal [00:21:47] Well, I mean, one of the simplest things to do is remember that just because you can do something doesn’t mean you should, right? Um, so every time an AI recommends a particular design choice, it’s up to you and your judgment, right. And your own experience to decide whether or not that’s a good design choice or whether or you need to reject it and ask it to try again. Because I’ve seen a number of times where an AI will provide an Uh, something that’s properly engineered, but a bad way to do something, right? Like it would work, but why would you do it that way? Right. So I think human judgment and that’s where, you know, people still are important and people still matter. And people with your experience still matter is that bringing that judgment to bear on the work that this, you know, intelligent intern, so to speak, as is bringing to the table.

Sean Murray [00:22:46] Yeah, I said I’m gonna read more Besong, you mentioned something kind of like my last thought before I want to get some takeaways. You mentioned edge cases in your last answer. And I had the privilege of working with like the king of finding edge cases at ITX. And that’s John Roets, right? If anybody could find an edge case, it was him. But the reason why he’s able to do those things is not just because of his technical aptitude and his product skill, but it’s because he took the time to know the product so well and the customers. Right. And in this game, you know, whoever knows their customers better wins. Right. And it’s also able to readily identify edge cases. And I don’t think that we can trust AI just to be able to figure out everything that could possibly go wrong. That’s still got to be on us. Right. Yeah. All right. So this was an amazing episode. Uh, thank you so much for being here. I’ve got a couple of takeaways. Uh, so. One was from one of the original answers, just kind of like know the guardrails that are actually around your AI integrations. Never make assumptions about security. Free tools are not for work, right? If you are using free tools for work because they work better than maybe the tool that’s been provided or you’re not able to have or you haven’t been authorized to use AI tools yet, it’s now time to have that uncomfortable conversation and use your product skill to actually make the case. For why you want to use these tools, right? I kind of had this thought while we were talking, this conversation isn’t really to like inspire fear, it’s really to use these new superpowers responsibly. Especially like in our game, people that build custom software and in our consultancy, like our reputation is our product, right. So we have to use these tools responsibly, otherwise our business doesn’t survive. So Jonathan, this is based on a quote that you provided. I liked the idea of thinking about security issues as a feature that you introduced. Like you are directly responsible for introducing a security flaw as a capability to a bad actor. So I thought that was a really cool way to think about it. And then finding a way to regularly introduce security language into your prompts. And then lastly, the last takeaway was you are still accountable for all product and design decisions, regardless of who built the code or who actually introduced them gear product. So this was an amazing episode. Thank you so much for being here I had a really good time and I’m looking forward to we’re kind of we’re gonna we’ve got a couple of episodes in this series planned sort of around Using AI to build products. So we’re excited to dig deeper into this topic area. Thanks so much being here today.