AI Compliance and Innovation: How Regulations Shape the Future

In cybersecurity, you have to trust on the behavior of people. You set out some certain technical guidelines, certain things you can capture and reduce the risk. But at the end it's all about behavior of people. And to my understanding, it's the same with AI. You can set out as many rules as you want, but there's only very limited technical possibilities to actually control it or to limit it. So it's the same here. So you need to teach people how to properly apply it or how to properly develop it. Foreign we're kicking off another brand new limited series around artificial intelligence, but this time with a particular focus on generative AI. Over the next few episodes you get to listen in on real unfiltered conversations between a star AI expert and an industry leader as they talk through both the opportunities and challenges of integrating AI across different sectors. In the first episode today we are joined by Antonina Berlodchenko, our head of regulatory and compliance, and Martin Martin Fix, our technical director and AI guru. They will dive into the ever changing world of AI compliance, covering everything from setting up a solid compliance management system to making sure how AI solutions are ethical and transparent moving forward without compromising on scale. We hope you enjoy the conversation. Hi Martin, it's great to have a chance to discuss with you AI regulations and integrations with AI solutions. So what is your observation when it comes to companies understanding of the importance of AI regulation? Hi Vina, nice to meet you. Honestly I believe most companies don't have a clue about this. What I can observe is that AI most likely is just applied and they don't think about too much about privacy or even legal regulatory needs. Compliance things to my impression, one of the most pressing topics for these companies is they're being afraid that using or diving into this regulatory compliance topics will will stop the speed of innovation and the speed of adjusting to this AI. And common sense seems to be in the business that if you're not really fast and just get some AI by tomorrow, you're out of business. So my take that it's actually considered pretty limited. One reason might be definitely that they just don't understand what it is and that it's most of the time maybe not that critical to get the first steps into compliance. What would be your take in terms of how complicated is it really and how afraid do you need to be about regulatory needs for AI? First of all, I completely agree to your comment. I also tend to see similar approaches but overall when it comes to the complexity of the regulations of AI as of now I would say it is now rapidly developing and AI act was published recently in Europe. But with that said, the fact that it was published does not mean necessarily that it is fully shaped. So some of the aspects are still open and have to be clarify in the upcoming months. So I think that it's really important to explain to businesses and like work together as consultants businesses, all of us, and regulators of course, to make sure that we shape these AI regulations in a way that works well for safety but also for innovation. To my understanding, the aim of these regulatory documents and approaches from Europe, but also other countries in the world is to basically protect impact on humans, right? So anything which could harm your life in terms of physical life, but also digital life, your reputation, or your general, let's say being an accepted member of mankind potentially falls under regulation, which makes it for me massively hit industries like legal finance and health tech. Whereas legal and finance, most people might not see straightforward as being falling under this regulatory. But I think if I'm correct, it's also about that legal like contracting or biases about people not being allowed to get certain contracts or benefits because the AI categorizes them as not quote unquote worthy has a significant impact on your private life, same as finance. Like you don't get the loan granted because you're a person of color and you come from a socially critical area of a town. Am I correct that this also falls under regulatory clients needs? Right. So as of now, if we speak of AI Act, AI act has two parts of the high risk AI system definition. So the first one is, as you said previously, if your product is already a product itself or a safety component for product which is already regulated by one of the directives or regulations listed in the Annex 1 of EIA act, then your AI system will be considered a high risk on the AI act and will be basically regulated. And here we will have obviously like MedTech with MDR IVDR regulations, for example, in Europe. The second part of this definition is an extreme of the AI act which lists specific use cases of AI which considered to be higher risk. And under the use cases we have biometric identification and categorization of natural persons, we have critical infrastructure support, we have different AI systems for support of employment and educational environment. So as you mentioned previously in your example, like law enforcement, control of migration, border control and different systems for insurance, administration of justice and democratic processes, all of that According to Annex 3 of EIA as of now will be classified as high risk EA system. But does it mean that if you fall into the any of these areas where you need to have regulatory compliance or you want to have regulatory compliance, that a real strong impediment for your solution to be built. I mean that's usual concern of people, right? That's also a good question. So in this latest version of AI act which was published in my opinion, I don't know how effective it will be in the long run, but it is very mild on the business I would say. Because when we get back to those two definitions of AI, there is one where products were already regulated, the second one of specific use cases. So there are two different paths to market. For the first category of products you can say they will not be highly affected because they anyways had to go through conformity assessment procedure under their already existing regulations. And AI act will just bring them additional scope of these conformity assessment requirements for those additional systems like educational or employment applications of AI. AI act describes different pathway in general. So it will be possible to kind of self declare your compliance with AI act, kind of draw up a declaration of conformity and make sure that you follow all the requirements. But it will be not as strict and I mean not in terms of that you don't have to follow it, but rather it will not affect so much business because they will not have to sign contracts with additional certification or notified bodies pay for those conformity assessments. So it will be a little bit easier, but still they will have to be compliant with AI Act. And what I have seen is a pretty, let's say non intuitive observation when it comes to having compliance or governance established. As I mentioned, a concern of many companies is that if you need to be compliant, it will slow down your speed and your development for creating innovative solutions. Because many people feel that you will be too limited. You can't do this. And if you want to do that, you have to do a go through a long term of bureaucracy and all this stuff. But what I have seen actually is that if compliance is established early, maybe not even enforced by regulatory needs, but because just the company is aware of it, it actually can speed up the development. And the funny reason is that compliance takes away certain dimensions you don't have to explore anymore because they're already, let's say, defined or limited. Right? If compliance defines, you're not going to introduce a solution or AI around judgment of people's cultural preferences. You don't have to explore this dimension for your product anymore because it says this is a no go area which can help actually focus your energy and your drive into the still allowed dimensions. And you could actually turn this into a win win, which means you have lesser dimensions to explore. And when you do so, you already have a kind of compliant product because you followed the rules and the guidelines you set up yourself. So I agree to what you said. And it's interesting perspective how I look at it. So I look at those different compliance management system as a framework which basically gives you ways of working. And instead of trying to move in the dark and come up with processes on the go, you are establishing some kind of map and then you move by that map. So first of all, it makes it simpler. It limits, as you said, potential number of trials and fails. But it also gives you, for example, specifically with machine learning, I've observed in some of the companies that sometimes if there is no process, sometimes we are not able to even repeat our own experiment. So it's very important to be in control of what is happening, to be able to basically repeat what you achieved and explain what you achieved. So from that perspective, having some kind of framework in place makes your life much easier, not harder. Yeah. And the end, I think it all comes down to trust. Specifically with AI, how do you trust the solution you have built? Practical example, right? If you have an innovative employee, he uses some of the AI technologies, builds a kind of solution, which sounds cool and great for the cases he's demonstrating. Everybody says, yeah, cool, I want to have this. But management or accountable people should ask themselves, can I trust this solution? Can I trust the aspects of data privacy? Can I trust the quality of what it's producing? Can I trust that people will use it in the sense it was intended to and not for some, let's say, shady kind of applications. And building this trust can be supported by the frameworks you just mentioned. Right? Because if you can prove that it has been following a set of rules, a set of guidelines, a set of regulatory needs, it's easier to trust the solution. But I still feel that it's not a proof. So following all this compliance regulations, et cetera, is not a technical proof necessarily that the solution is absolutely safe, but it's a good hint that it's very close to be compliant. Yeah, completely agree. So it's the same as with quality assurance. So in software quality assurance, one of the principles is that there is no software free of bugs. So this is kind of a rule. So the thing here, when you implement the product using some kind of framework, it helps you to, as you said, it helps you to achieve some product which could be trusted more because you, you were somehow in control on every step of developing it, but still it's not a bulletproof thing. So if you're using the AI management system and for example, you develop a product which is not needed on the market, you won't succeed. Or even if you developed it, applying all different aspects of risk management, impact assessment and all of that, you might easily oversee some of the bias in your system because technology, ML and AI technology itself is very complex. So that's exactly why we should start building this kind of systems and start working together to learn more and understand more and also adjust our processes and how we approach this, because it could easily go out of control. And to my experience, and this is not only because of the current AI trend, but in general, when you build products out of the software realm, is that having some guidelines, some rules, or let's say I call it principles in place always helps to make you a better and more transparent project. And it also guides your team and give them some guardrails, where to go and what to do. So what I've heard, I think you and I agree that specifically for AI, it always does make sense to have this frame, even if you're not having regulatory requirements at this point, right? So even if you know your product you're making will never fall under regulatory needs from AI act or from any other country, it would still help to make you a better and maybe even faster product if you put them in place. You call it aims. So AI management system, would you agree to this? Yes, I would agree to this. Especially as of now, this kind of approach with like when organization would be able to demonstrate responsible approach to developing AI, it could be really strong signal to the market, not only to regulators which might be not applicable, but also to the end users to show that we as an organization treat this seriously and responsibly. But even not speaking of that, having some process in place will help the organization to be more in control of what is happening. So I definitely think that it should be valuable even for the organizations which are not regulated as of now. And I have been working on ISMs and IMs and quite some time in previous positions, but also here at our company at Star, and I feel that the term, just the term management system is scaring away people quite easily because they think it's 1 billion forms, 100,000 processes to be documented, a ton load of pages, et cetera. But would you agree that setting out just some simple things like we will not use AI to judge people's race, social status, we will not Use AI to judge the performance of our employees. We will only use AI where we can check the quality and prove what it's doing. Just these few statements is already an AI management system. Very, very lightweight. But it is, right? Yeah. So you can at least introduce some of the aspects of AI management system. You can for example, create AI policy so you can describe your policy when it comes to using ready AI solutions. You can, for example, whitelist which solutions are allowed to be used for which kinds of business tasks. You can explain your people what are the consequences of potential, you know, hazards related to using AI, like really the data privacy or many others. So definitely it is not all or nothing. Even some parts of it might be applied to different processes in the organization to make it more controlled. And I would even compare it to cybersecurity, honestly. So one of the biggest challenges is in cybersecurity is many companies want to have it technically completely controlled and safe, which is impossible. I mean, maybe it's close to possible, but then you would be working like Lockheed Martin in the most secure bunker area somewhere, developing some very top notch secret stuff, which is not for the majority of industries. So in cybersecurity you have to trust on the behavior of people. You set out some certain technical guidelines, certain things you can capture and reduce the risk. But at the end it's all about behavior of people. And to my understanding, it's the same with AI. You can set out as many rules as you want, but there's only very limited technical possibilities to actually control it or to limit it. So it's the same here. So you need to teach people how to properly apply it or how to properly develop it. AI is even more difficult to prove, let's say correctness, because it's not unpredictable. But usually AI systems are non deterministic. And it's not like in qa. You put this in, you get this out. It doesn't work for many of these AI applications. Yes. So yeah, so you have very different aspects to what you just said. So yes, even with regular software, which is deterministic, it's not possible to test it all. So we always say that bugs are there. With machine learning which are not deterministic, it becomes even more complex because now you have to apply different statistical approaches and come up with slicing data from different directions to be able to demonstrate the performance on each subset of data, et cetera, et cetera. But when we get back to your first part of your comment about information security management system versus AI management system, my opinion is that the Best way is to basically implement some kind of integrated management system with some aspects focusing on information security, cybersecurity, some aspects on AI. And when it comes to the AIMS itself, the approach is very similar to isms. You have to draft your statement of applicability, conduct risk assessment and identify those key areas of criticality for your organization. So similarly to ISMS and assets, and draw up this statement of lickability explaining which processes are implemented in your organization to basically manage AI. And do you have any practical experience of how companies might deal with the AI? Specific challenge of biases? So saying that the system is being not objective in every perspective of its use, how to detect it or how to deal with it. Yeah. So what I've seen from, for example, as of now, I support esteem. We support many companies with medical device submissions to regulators and many regulators are already now checking for some of the good machine learning practices. So for example, what they see, regulators want companies to make sure that when we speak, for example of intended population for which will be using the medical device, they want us not only just test it as a whole, they want to make sure that we cover specifically, you know, different genders, different age subgroups. They want us to take into account what is the source of the data. Because bias could be even coming from. If you source data from the same place and like split into training and testing data, so it is being controlled from different angles. And there are also ISO standards drafted for managing bias in the ML systems. So bias very intangible, same as quality or security. But definitely you can establish some kind of process to at least try to be in control of it. And one of our previous companies where we were doing machine learning, so before the, before it was called AI area, we discovered that detecting bias is actually nothing you will get out of detect regular statistics like you're used to from traditional quality assurance. I would say because the training of a network or of a model is basically intransparent for you as an engineer, right? You put in some data, you check how it's performing. And we have seen that there are certain biases can be introduced and never been realized unless you specifically stumble over one edge case where the result seems to be odd. And what I have learned is that you need to actively search for these edge cases, which is a totally different approach than in traditional qa because these edge cases do not come from functional definitions or requirements, they come from investigating more deeper. Basically technically speaking, all the dimensions the network is addressing. Very briefly, practical example, we work on image recognition, on object detection out of images for road markings. So, you know, these dashed lines on the road and the solid lines and the turn arrows and all this kind of stuff. The network worked pretty well until a certain point it stopped detecting dashed lines on the road. It went totally crazy. And there was no reason. The training data was fine. The data we put in was compliant with all the standards, low quality, et cetera. And then we, after a long, long investigation, very deep technically into the network, we realized that the network actually learned not to detect dashed lines, but signposts which were beside the road, because there was a correlation between signposts and dashed lines. For us, it was kind of a funny thing, but it was very hard to find because only after weeks of using it, there was this one edge case where it went off. And I think this is the challenge when you need to prove regulatory compliance or when you work with AI in general. And this wouldn't be covered by your AIMS per se, right? You would describe the procedures and the processes you have in place. But would it make sense then to add some of these, let's say creative bias research into your management system as well? Yes, it would be because bias could be introduced on any step of the AI ML life cycle. So by having some of the process on each step, you are already by that managing bias. So for example, I've heard about similar example as you mentioned, but it was in more in medtech space, when system learned to detect if the image was coming from more expensive scanner, it learned that most likely this person will have some condition. So it took it as a feature which was not intended and it was kind of a data leak because one of the hospitals for more seriously ill people, they were sending them for expensive scanner to be able to just diagnosed better. So with that said, AIMS will help you with it because within the procedures on data management, modeling itself, feature engineering, et cetera, et cetera, when we have it as a system and it is all laid out in documentation how we do it step by step, you will at least, you know, remind people to keep in mind the possibility of this bias. And when we develop a product ML solution on the AI management system, we will keep a record of every step and all our, you know, decision making process ideally. So it will help us to even look back and investigate better than in case when we have nothing but model itself. Interesting topic question here. In this scenario, you have an IMS in place, you develop a product, you follow all the steps compliant to your aims, you release the product and after it has been released, a bias is being detected which is not in favor of the users. Does it make a difference or does it help you that you have followed AI Ms. Procedures compared to a company which doesn't have an AIMS in place and ends up in the same situation? It helps you because if you see that bias, you first of all can repeat your experiment. So you have all the data and everything step by step process recorded. So you can go back and identify where this bias was actually entered into my system and then you can learn from that for the future. So you're just more in control. We are getting back to the same scenario that developing something under the QMS or AIMS or ISMS or whatever else math, sorry for the pun. It's not necessarily ensuring that you will be having a product 100 free of bugs or free of bias. It is just doing your best to develop the solution which will be less likely containing those kind of issues. But making this question, let's say, make it more extreme, let's say this example, one company following aims, one doesn't they end up at the same bias issue. Now, regulatory people come in to check from the reputational perspective, of course, in case if you develop something incomplete house and you ended up having some bias in the system, obviously for your company, it will be much more harmful in terms of their reputation comparing to the situation when you had this issue. But then they come for you, check with some kind of audit and you can show that yes, we have all the processes in place and we did our best to, you know, detect it. We conducted risk assessments, impact assessments and all of that and we can show you all this documentation. But it still happened and everyone understands that it is like not 100% solution. And in this case it will be so much easier on the reputation of the company, I would say. And we talked about also the combination of AI and cybersecurity and data privacy. What I see is that a lot of companies still don't have a decent understanding that when you produce an AI solution, you're factually disclosing information at any point of time. Because when you take training data to create a model, the model actually knows everything which was in the training data. And there are already technologies from hackers developed to retrieve this knowledge from the network in a malicious sense to in order to get some underlying data. They have been a lot of examples in the web about this happening. So would you then recommend that these aspects of data privacy and IP protection, data security, should be a centerpiece of the AIMS as well? Yes, definitely. So there are all those standards for AI, we also mention cybersecurity aspects and data privacy aspects, and as I said previously as well, I think the AIMS itself is not sufficient. It has to be integrated into information security management system, quality management system to be like standing on four legs instead of one. What is also quite interesting that the needs for data protection and data privacy, et cetera, are actually very strong technology drivers. So let's say if you have a company which has very specific knowledge, which is highly, let's say, a great USP in the market, you definitely do not want to have this as training data for a model which is used from every employee or even outside the company. So, meaning if you do not want to represent the data as training data of your model, you have to follow a totally different technical approach to still make it usable and to still make it accessible, but that you can keep control of it. This is an interesting new aspect of software development activities and engineering and architecture work, which companies also still need to learn. Because companies like OpenAI and Claude and Anthropic and how they call, of course their marketing strategy is it's very easy, give me your data, I train your model and you can do everything with it. But this basically violates everything so regulatory, potentially data security, data privacy, et cetera, et cetera. I feel that this is also an educational path. A lot of companies still need to go and you need to do this very early because once you're set off on a specific technology setup with a solution, it's pretty hard to go back to square one and do it differently once you've learned you missed something. That's true. Have you come across this in medical specifically? Most likely not, right? What I observed in medtech companies, obviously data is crucial part of ML, so without data they won't be able to do anything. So it's like very important assets for the companies. I wanted to also mention is that technically speaking. So again, from the engineering and software perspective, if you need to work under an AIMS or an isms, whatever it usually means you want, or you need to introduce additional technical means to ensure compliance to what you have set out in your management system, and then in traditional software development, it's well established technologies like regression tests, automation tests, black box, smoke test, whatever. I feel that for AI based solution, it's going to be slightly different and also slightly more difficult because as we talked earlier about finding biases or making sure that the output of a chat agent or a system is really compliant to what you want is much more difficult. And what I see is that it still requires a decent amount of human work actually because the assessment of something is applicable or good or reasonably good is still more efficient in human brains than building a system to come up with this conclusion. And I have seen also that companies start using a second AI system to assess the quality of the first AI system, which seems to reasonable but also is actually creating a kind of recursion. Because if your product is an AI system and you're checking with another AI system, how do you know that your checking system also works compliant or following to the rules? Is this, let's say recursion pattern, something which can be addressed by a management system as well? Yeah. So I've seen these so called feedback loops when you're reinforcing some of the biases, for example, by using the outcomes of one system and hitting it into another could kind of multiply and magnify some of the issues. When it comes to the testing of AI systems, the area is actively developing what I observe in some of the companies, sometimes with specifically testing of mls, there is an issue of this testing Oracle. So testing Oracle is basically expected outcome for the given input in the system. So with ML systems sometimes you don't have those Oracles and there are already standards describing some of the established techniques for AI ML testing, where they suggest some of the techniques like metamorphic testing, A B testing, which allow you to basically approach this and somehow manage these issues with AI. Pretty cool. So my basic takeaway then from our very interesting walk through this stuff is that you should definitely have any kind of AI management system in place at early stage, but you can do it quite lightweight so you don't have to have already a thousand Wiki pages and 500 process descriptions, et cetera. Start with something, but start do it lean. And also start making sure that you have people understanding that there will be management system compliant needs. Always you cannot build any AI solution, just build it and use it. Even in the smallest case it's helpful. Do you think this is a valid conclusion? I agree with you. And one step further, AI act also establishes some of the responsibilities towards AI deployers. So if you have business deploying high risk AI solution for your business needs, you will have to comply and meet some of the requirements established by the act. So as Martin previously suggested, I think that the first step of drawing up AI policy in the organization would be a good idea. Yeah. And to wrap this up, my experience, not specifically for AI management systems, but in general is if you start early, start small, it's much less scary than when it comes around the corner on surprise, and you have to introduce it, because if you grow with it, it's getting natural. And then if you really fall into one of these regulated categories later, it's much easier for you to fulfill all the requirements and the documentation and provide all the stuff, because you start already building kind of a mindset around it. Yes, I totally second this.