Scaling What Works - Without Breaking What Doesn’t, with Justin Kenney and Katie Bockwoldt

Speaker A: Foreign. So in our first iteration of recorded trainings, it was a lot more passive. We were doing the best we could with the knowledge and tools that we had at the time. But, um, some of the feedback is, you know, we definitely want more interactivity and more engaging content to keep your attention. And keeping that short. Right. Like 30 minutes or less is probably the best, um, for recorded content. So moving forward, I think that's going to be a larger component of it, um, because I think that's how adults learn best and it's easier to stay engaged in that way.

Speaker B: Hey, everybody, I'm Elizabeth Swan.

Speaker C: And I'm Tracy o', Rourke, and we're from the Just in Time Cafe. And welcome to our podcast. At the cafe, we wrestle with tough questions, talk to groundbreakers, discuss great books, and get insights from leaders and practitioners who are making a difference in the world.

Speaker B: We let you in on helpful apps, we bring you the news, and we challenge the status quo so you can build your problem solving muscles.

Speaker C: So, Elizabeth, what is on the cafe menu?

Speaker B: I'm so happy you're asking me, Tracy. Today's highlight is our interview with Justin Kenney, Chief Performance Officer, and Katie Buchwelt, Statewide Director of Performance Improvement for the state of Vermont. Two leaders quietly reshaping how government delivers results. So what starts as a conversation about training programs quickly turns into something more systemic. A shift toward communities of practice, performance, logic models, and a more human centered approach to continuous improvements for hot apps. Little detour. We're going to take a closer look at Justin's homegrown praise board. Built inside Microsoft Teams, it is a deceptively simple tool that raises a bigger question. Can recognition be operationalized? What happens when it becomes visible and shared? And for Q and A, uh, we are going to discuss what does it really take to sustain a lean six Sigma deployment, and how much of that comes down to leadership behaviors around recognition and engagement. Okay, Tracy SoCal, you're gonna be outnumbered by the Northeast today.

Speaker C: All right, well, I'm looking forward to it. I won't talk about the weather.

Speaker B: Yeah, no, zip it, zip it.

Speaker C: Up next is hot apps.

Speaker B: All right, quick heads, uh, up. I mentioned before, we jump into hot apps, this one's a bit of a format break. Normally, uh, Tracy and I take a nap for spin. We poke at it, maybe break it a little bit, and then we report back, right? What worked, what didn't, and whether or not, of course, there is a free version. Um, but today, we're going to go straight to the source. So instead of reviewing someone Else's tool. We're going to be talking with the person who built this one, you know, our guest Justin Kenny, or one of our guests today. And he created what he calls a praise board right. Inside Microsoft Teams. And he'll describe what possessed him to build it and then how it works. Right. And since it was crafted within such common software, most of you, or many of you are in organizations that use teams, um, this will set you up so you could build your own if you were interested. So let's get into it.

Speaker A: Right.

Speaker B: Okay. So what we're going to talk about now is not necessarily a traditional app, as Tracy and I have covered over the years, but this, um, uh, Justin is with us today and he used Microsoft Teams to build what he calls a praise board, uh, for the state of Vermont. And we are dying to know, you know, how, how you did it, uh, what led to it, what's come of it. Um, so let you start there.

Speaker D: All right. Yeah. So, uh, really excited to show this off. Happy with the final product. All right. Not even final product because it's always evolving and improving, um, agile development here. But this is a concept that actually came out of, uh, a speaker series that Elizabeth did with us a little while back talking about the importance of praise. Um, we had, uh, almost 200 people on that call. And as Elizabeth was talking through her content, uh, really fruitful conversation in the chat, and someone asked a very simple question which was, is there a place that I can provide some praise to another employee? And I thought about that and I was like, not really. Like, you can send them an email.

Speaker A: Okay.

Speaker D: Um, you could use the, like, there's a praise app in Microsoft Teams where you just send them praise, but this is going to go between you and them. And then there's the merit award system, which is usually used by managers and leaders. Right. And usually monetary components to it. Um, so it's not something that a regular employee is just going to do. So I was like, no, it doesn't really exist, but it would be really cool if it did. Um, so I really love all the Microsoft suite of products and I like to build stuff that's utilizing those. So we don't have to pay any extra for licensing because we're already paying for it. Uh, so I said, I think we can make this work. Um, so we originally started with a Microsoft form that people filled out. Um, and then. Right. Some Power Automate flows in the background that would then pass that off to an individual. Um, and then we said, well, there's some issues with that, because you can't pick a person if you get their email wrong, it's not going to go to the person. There's going to be all sorts of errors that happen in the flow. And then there was also this potential desire for someone to remain anonymous. Like, maybe I want to give Katie praise. I don't actually want her to know that I sent the praise. I want it to be anonymous. And that was a little bit difficult to form. So we scrapped that as initial idea. We had some people test it. Um, and then we went to SharePoint as SharePoint List feature. And there's a new feature in SharePoint List that allows you to just create a form from that list and anyone can fill it out. So we did that. That was working great. And then we had some authentication issues with certain users who couldn't access it for some reason, even though they were supposed to. There was glitches. I didn't like that. I wanted it available to everybody. Um, so then we took it one step further and built a power app that allows people to submit this information. I'll show you what the final product looks like. Um, once we've got all that set up, we actually built a page on our SharePoint site. So this is what houses all of our information about training, our CI Toolkit speaker series, all of that built an entire page that's focused on the praise board. It talks about what praise is, that talks about reasons for sharing praise, and then prompts them to participate in sharing some praise. Um, and this is the app. It's one screen, super simple. So again, this is built in Microsoft Power Apps, uh, one screen that provides some information with a people picker. Uh, so from here, this connects to our active directory, uh, where you can just start, uh, typing somebody's name and it's going to find it for you. You pick it. Um, you choose what you want to say in terms of praise. Awesome work on the, uh, Just In Time podcast. Yes, indeed. Katie, that was amazing. Um, and then there's a toggle. Do you want to send your praise anonymously? And says if you choose yes, your name will not be shared with the recipient. So I want her to know what's coming from me. I think she's going to be able to guess. Um, and then you hit submit. Um, so that goes out little bit of a congratulatory here. And then you can view the praise board, give more praise, or exit the app. So what happens on the back end is that form field built in PowerApps, then goes to a SharePoint list that only Katie and I have access to, so no one else can see it. Um, so it goes there and then it fires off an email, uh, to Katie. If I said that I wanted to be anonymous, it will not include my name. Um, if I didn't, then it will say, justin has shared some praise with you. It'll share the praise message and then it's going to give Katie some options. At the bottom it's going to say, do you want to post this publicly? And I'll show the praise board in a minute or do you want to keep it private? So people get that choice of whether they want to be recognized publicly or not, which is really important because some people don't want to be recognized publicly. Um, so they choose an option and then if they hit, yeah, put it on the praise board. Uh, it will go down to the praise board that is here. Uh, so that information then goes to a secondary sharepoint list that everyone has access to view. We're using the board gallery here. Um, almost looks Kanban style. We've organized this so you've got the recipient the message and then who shared the praise. And this would be blank, like in this case if it's anonymous. Uh, and you can see here, uh, we've had, uh, since we launched it, 27, uh, public praises that have been put here. And there's more than that on ours for the ones that aren't public. So you can go in, you can see, uh, whatever you want. There's a couple different views that you can organize. I'll go to see all so you can check this out. Um, so there's the gallery view, which is the one that everyone gets to see. We also have a view called My praise, uh, which allows you to scope down the entire list to praise that only has you as the recipient. So you have a repository of all of your praise right there and then you can go back to see all items and kind of a grid view. Um, but yeah, that's how it works. People like it. We wanted to keep it very, very simple. And again, we went through a couple different iterations. We learned, we tested things, didn't work, tried something new. Um, and it's great to be able to do all of this within the Microsoft suite. Didn't, uh, require any massive coding or development work. We were able to stand this up very, very, uh, quickly and then get it out to the entire organization.

Speaker B: The tech whisperer strikes again. Um, so I, I love that aspect of being able to have praise be public or private. And it sounds like you said the ratio is more private.

Speaker D: Yeah, part of it might be people don't take action on the email. Um, so if they don't take action, we do nothing. 30 days goes by, it times out and then it just stays private. Um, so that could be it. People don't take action on that. Um, but there are some that click. I don't want it to be public.

Speaker B: Yeah, what a cool thing. This is great. Thank you for describing us. You let me know it happened kind of shortly after the, uh, speaker series and I just love that that was an outcome. And I think you had a discussion at a community of practice after that happened. Oh, yeah, you just said, uh, that someone said, well, how, how would we do that? What's an easy way to do that? So, um, and yeah, there are some, you know, simple ways, but you've again, kept it simple but made it accessible and uh, get given just enough versatility. Right. Stay, make it public or just enjoy it yourself.

Speaker D: Thanks for the impetus. Appreciate that.

Speaker B: Um, thank you so much for sharing that. This was a good twist for an app review. And uh, yeah, I think I'm going to come back to you regularly for apps. Justin, I suspect you are a fount of technology at this point. You are having way too much fun.

Speaker D: We should share our Power BI dashboards at some point. I love our Power Bi dashboards.

Speaker B: Um, well, then I make a note and you're coming back. I'm Elizabeth Swan and you're listening to the Just in Time Cafe podcast. In a short while, you get to hear our interview with Justin Kenney and Katie Buckwalt. This is a return trip to the cafe for this pair since they gave a record breaking webinar about their no Belts continuous improvement program in the past, which you should check out.

Speaker C: Up next, it's Q and A.

Speaker B: Today's question is what does it really take to sustain a lean, lean six sigma deployment and how much of that comes down to leadership behaviors around recognition and engagement? And Tracy, I know you've got opinions about this, so what have you seen that truly helped sustain a deployment?

Speaker C: Well, you know, this is opening a can of worms, I feel like, because there are so many things to think about with successful deployment or unsuccessful deployment and a huge part of that is under the narrowly, narrowly scoped leadership behavior section, or category, if you will. But I, you know, in a nutshell, one of my favorite quotes that I share a lot with my green belts is, uh, from a book called Boundaries for leaders by Dr. Henry Cloud. And in that book he Shares one of my favorite quotes. And it's culture is created by what is promoted and what is tolerated. So if we promote CI continuous improvement with recognition, and we rethink leadership behaviors around promoting continuous improvement, that falls under the what is promoted category. And, you know, it's interesting because most of the time, if they don't have a CI culture, it's pretty obvious that leaders really aren't doing much to support a CI culture. So there's a direct correlation. And then there's also.

Speaker A: And.

Speaker C: And here's the thing. I feel like sometimes behaviors are just hard to remember doing behaviors like people say, oh, I gotta recognize people. I'll be sure to do that going forward. Well, if you don't operationalize it, like Justin was saying, or if it's not tied to a process or a habit, it's gonna be difficult to remember, because most of the time, when we have time to remember something, we're pretty tired. So the question becomes, how do we operationalize it? Or in other words, how do we get our processes and habits to help us continue to do the right things, such as recognition and praise? Maybe you can, uh, put it on your calendar, or maybe you could put it on. On an agenda for a meeting to establish some sort of cadence and consistency around praise and recognition. But I feel like recognizing that it's a habit that you need to work towards. I feel like people don't. Don't, um, either they don't really come up with that or they. They forget about it. But I will say as a sidebar that one of my favorite recognition programs was a leader who was, uh, in the Washington area, and she worked for King county, and she really wanted to promote recognition of removing waste, really. And so she would go around and she bought these little Gumby figures, you know, the. The little. That character Gumby. And she would recognize people by giving them one of these little mini Gumby figures figurines. And do you know why it was a Gumby figure, Elizabeth? Because Gumby has no waist.

Speaker B: Oh, my God. That is. That is so sneaky. That is so hilarious. Uh, and. Oh, God, I can't believe that hasn't spread like wildfire. That is awesome. That is perfect for so many reasons. But I feel like. And, you know, I'm writing about this, but it's. It's. There's a real balance there between kind of a program, a recognition process, and then also authenticity. And I feel like the Gumby figure thing like, that is so, like, particular to that leader and hilarious in its delivery. Like, it's got a really nice twist to it. Like, I would definitely want a Gumby. Like, I, I would be so honored. I would be like, I've got to get, I gotta get like as many, many gumbies as I could. That's an awesome one. That's hilarious and, and on point and I think very personal. Um, I recently heard one. This is like on the other end of it, kind of not so good. But I heard the story. It was a global health and industrial imaging company that looked like it had, you know, on the surface, a phenomenally motivational recognition program. Right. But it ended up having the complete opposite effect. So each quarter, leaders were asked to nominate high impact projects. Uh, the criteria were, you know, measurable results, cross functional teamwork, strong leadership. And then they had an executive cohort vet all the nominations and forward them to the CEO who made the final call. And the winners were all recognized at a global forum. They got a significant financial reward, always nice. So on paper, seemed like that would easily drive motivation. And it totally collapsed. Uh, and the two things that totally undermined it. One, the approval process turned into a kind of a cross examination. The CEO wanted to ensure that the process had rigor. So he would dig into the details. He questioned decision making, looked at the specific approaches that were used. So it was intended as quality control. Felt to project leaders, uh, like having to defend their work. Right? You're going in before a thesis board or something. Instead of feeling recognized, they felt like completely scrutinized. And the second thing was that by design, the recognition was concentrated only on the project leader and the team. The people who absolutely did a lot of the work were invisible as opposed to celebrated. So the effect was project leaders felt uncomfortable taking all the credit. Right. And the teams, uh, felt totally overlooked. And over time, you know, nominations dropped to zero. Right. People, the word got out. You were going to get like go before a, uh, you know, a grilling squad. And the program didn't fail because of lack of structure. It failed because of how the experience felt to people who were part of it. Right. And it strikes me as well intended. And you and I both push the people we guide and coach to raise the bar. Right? We work with project leaders, we, we question their metrics. We push folks to dig into root causes and do their homework and drill into the process. But what happened here was different. And the projects had already been rigorously vetted by an executive leadership cohort. So when the CEO introduced yet another, you know, another additional level of vetting, he, uh, unintentionally undermined his leadership team. So I'm going back to Gumby on this. So I think that's one of the eight ways. Right. This is over processing. It's like requiring multiple signatures in a process. Right. It puts a drag on momentum. In this case, it degraded the experience to such a degree that people opted out of receiving substantial financial rewards and any recognition at all, which is kind of fascinating. That is fascinating to me. Yeah.

Speaker C: Well, you know, there's a lot that's going on, and we could unpack that a lot. But, you know, I kind of felt like in hearing that, the hearing what the CEO is doing, it's really like he kind of forgot what his role was. Right. In that instance, he really didn't need to be the scrutinizer. He needed to be the cheerleader and the, you know, showing appreciation and inspiring people to continue and kind of miss the boat on that. And, you know, I think this is probably something that works really well somewhere. And I think it's kind of paying a nod to the fact that the culture is a puzzle. And, you know, when we try things for recognition and praise, it is an experiment. And, you know, there's so many nuances about, you know, here's what we're going to do to. For praise or recognition, but the nuances around that sometimes don't work well. So it's really, you know, assessing is this effective or is this not effective? And do we need to be doing something different? Are we going to stick to this and keep doing it? So I feel like everything is sometimes a cultural experiment when you're trying to, you know, embed these in the culture and you have to tweak. Um, also reminds me, uh, like when you mentioned the financial reward piece. It does. It did remind me of that book, the Surprising Truth about what Motivates Us by Daniel Pink Drive. And, you know, talking about how motivation relies on intrinsic factors rather than traditional carrot and stick rewards. And sometimes they say when you do pay people for something, it actually demotivates them to do the deed when they're not getting a reward.

Speaker B: Right.

Speaker C: So it actually has the opposite effect sometimes, uh, because they're like, well, I'm not getting a reward this time, so I'm not going to do it anymore. And so, you know, it's just really interesting culturally how these things land and to pay attention to it. So I feel like the best thing you could do is experiment, try things, and really pay attention to what's happening culturally. Is this working? Is it not? Are people put off by it, you know, I think, um, there was one program that I saw. It was. We called it the Paper Plate Awards. This is from this organization. And they would actually literally draw on a paper plate with markers. You know, awards like you're the cherry on top. And it was great. And people loved it. And they would put it in their, you know, in their cubicles and. But somebody, uh, other people might think that is so awful, and it's so cheap. You know, you never know how things are going to go.

Speaker B: You're raising a great point. And that book by Daniel Pink Drive was so, so seminal. Like, it really brought out that reality in terms of what. What really motivates people. And the other thing you said, which is, you know, hey, paper plates over here, Gumby over here. Big financial reward didn't quite work out. And it's like people saying, you can't go to Toyota and then sort of take a snapshot of their culture and bring it back over and implant it into your factory. It's all. It's all of a piece. Right. It has to be part of your culture, and your culture has to grow its own.

Speaker C: Right.

Speaker B: Maybe it's a paper plate, maybe it's Gumby. You know, it. It has to be, uh, about that particular group. And I think that's important to remember as you build these, because it has to be a mix. I think it can't just be, uh. It can't be all prescribed is how it's. Then it's not authentic. And I think if it's not authentic, it's not worth much, but.

Speaker C: Right.

Speaker B: Yeah, this is. We're going to keep talking about this because it's so fascinating, but, um, you are. You're dead on. It's nuanced and.

Speaker C: Yeah, nuance of the nuance. And then. And I'll just say this last piece, and then we can move on. And then there's this whole idea of novelty. Right. So do you go with standard? And this is how we're going to standardize and make sure we continue to do praise and recognition. But what about novelty? Right? So sometimes Gumby gets boring, man. We've been sending out Gumbies for three years. What are you going to do now? Right.

Speaker B: Uh, yeah, you know why? Because it's the current best way. It's not the only way. And you have to, like every other process, continuously improve it.

Speaker C: Exactly. I'm Tracy o', Rourke, and you're listening to the Just In Time Cafe podcast. We host these monthly, so you can go to our page at www.jitcafe.com and hit us up.

Speaker B: Coming up next, it's an interview with our featured guests, Justin Kenney and Katie Buckwell. Tracy, why don't you tell our listeners a little bit about our fabulous guests?

Speaker C: I would love to. So Katie Buchwald is the statewide Director of Performance Improvement for Vermont's Chief Performance Office, where she leads efforts to build a culture of continuous improvement across state government. She oversees professional development and enterprise wide performance frameworks, equipping leaders and staff with the skills and structures needed to improve outcomes. And through her work, Katie helps translate strategy into practical actions, supporting initiatives that drive measurable impact at both the enterprise and local levels. Justin Kenney is Chief Performance Officer for the State of Vermont, where he has spent more than 13 years advancing lean and continuous improvement to strengthen programs, processes and outcomes for both staff and the citizens they serve. Previously Director of Continuous Improvement and Planning, he combines operational rigor with a practical people first approach to change. Known informally as the Tech Whisperer, Justin has a knack for helping teams adopt and adapt technology in ways that actually stick. Can't wait to hear what they have to say when they come to the cafe.

Speaker D: Oh yeah, foreign.

Speaker C: Welcome back to the cafe.

Speaker B: Yeah, welcome guys. The last time you visited the cafe was to run a webinar about how you had opted out of the Belt system with Continuous Improvement, which proved to be an incredibly popular webinar. And we have been, we've got this kind of, uh, way back machine segment where we've been doing this new Year. So, uh, we're, I suspect, going to be highlighting that one again. Uh, nice. Absolutely. So what I want you guys to do is, um, anchor us, right, because there's a lot of things going on at the state of Vermont, right. In the state of Vermont, uh, across the state of Vermont, all around the world. Um, and you guys have, um, you know, obviously evolved since we last spoke. So what's, you know, what's materially different for you now? What prompted the changes?

Speaker D: Katie, do you want to take on that one? Because that was your baby.

Speaker A: Yeah, sounds good. Um, it's been several years now. I'm trying to think when we first revamped the training program, probably at least three years now, um, but we chose to go away from the belts and build a system that was a lot more flexible and adaptable to people's needs. Um, so, uh, for brief historical context, our current program has 34 courses and about half of those are, um, recorded and available on demand for people to take any time. The other half of those are Delivered live. Um, and people can really jump in where they want. If somebody wants to get started, right, with improving processes content, they can start there, or they can start with strategic planning or performance management or culture development. Um, it really is kind of an a la carte, pick and choose what you want type of program. Um, and that has been really popular with our staff and managers across the state. And in the last couple years, we've seen tremendous growth in that program. Uh, prior to this launch, we were averaging about 200 to 300 people trained per year in our white belt and green belt program. And in our last fiscal year, we hit over 900 people trained. So we more than tripled the, um, amount of participation that was in the program, which was excellent. And it's really interesting when we, when we break down those data as well too. Justin has built a fabulous dashboard that tracks all of our data by agency, by, uh, organizational position, by training course. Um, and by far our recorded courses are our most popular courses. Um, followed by, um, our documenting and standardizing processes class and our improving meeting productivity class. And it's been interesting to see over the years how those like, by far, are always the most popular, which tells us a lot about our organization too, right? We need more documented and standardized processes. And we all have our pains with meetings, right? And we want more time back in our day. So we continuously review those data and look at trends. We adjust the number of offerings each year to match that. So our more popular courses we run four times a year, whereas ones that are less popular we run only once, uh, or twice. So we're continuously improving and evolving that program, using data as our guide, um, and being really grounded in customer feedback. We collect feedback at the end of each training as well as part of our annual surveys. And, um, we take that feedback very seriously. We share summaries with our participants as well as plans and like action items that we've completed based off of the feedback that we provide, because we believe closing that feedback loop is so incredibly important. So I would say our program has pretty much stabilized. We haven't added any new content necessarily, uh, since the initial launch. We're more just kind of running it as it is now. Although we have been looking at the data recently and starting to consider where might we want to record additional courses to help free up some of our capacity to work, uh, on other initiatives as well. So in the next year or so, that's going to be an increasing priority. Uh, also probably looking at those most popular classes, moving those to recorded so we can Reach an even broader audience. Um, but really all the decisions that we make are based on the data, um, every single year. Justin, anything you'd add?

Speaker D: No, I think that covered everything. I was just looking at the data recently. I mean in terms of net promoter score, which we calculate, that's typically in the 60s, which is pretty darn good, Close to world class. We're very excited about that. Um, and we have a uh, participation rate on those of about 30 to 40% percent. Folks take those after training surveys, which is pretty amazing. Um, and also, you know, we have an average number of trainings per participant of about five. So of the folks that are coming, they're taking on average five trainings with us. And uh, and those to what Katie said. Right. Some are recorded. So this may be an hour, an hour and a half for some of those. Ah, the live ones tend to be between three hours and four hours. So it's a mix of those. But feeling pretty good about that participation rate.

Speaker C: I have a question about. You said there's 900 people trained now. Um, does that mean that they went through all 34 courses or does that mean that there's some level of, of finishing amount of courses? Like how do you, or is it really just 900 people have engaged in the training programs?

Speaker A: Yeah, great question. So the metric that we calculate year to year is just unique. Participants trained and that just means um, any number of courses. So anybody who took one course is counted there. Um, Justin, do you have the total number of participants? It's closer to like 2, 3 or 4,000. Um, when we added up across all the years. So that 900 was just for our fiscal 25.

Speaker D: Yeah, we've had, I see about 3,000 people touch our training program over the past couple years, which the state of Vermont for context, about 8,000 state employees, more or less, depending on who you're counting. It's about 8,000 state employees. So feeling good about that number? Uh, we have, you know, uh, some classes of employees that um, we found are probably not going to come to our training program. Some of the field based staff, state police, those a little more difficult for them to take time out, corrections officers, et cetera. Right. So we tend to work with more like guest based employees. Um, but still like we'll feel like we're hitting a pretty good number for a small team.

Speaker C: Yeah, that's great.

Speaker B: Yeah. I'm wondering, uh, is there anything that didn't go the way you thought it would. Right. In terms of what your assumptions were around the people and the systems you're using, um, Katie, what was that?

Speaker A: Justin?

Speaker D: The complexity of the learning paths.

Speaker C: M. Maybe.

Speaker A: Yeah. So while our course can be navigated in any order at any time, we developed a few different learning paths to serve as, like, recommendations as to what people might take. Um, and on one hand, it's really great to have that as, like, a goal for people to work towards. Like, yes, I want to get this operational excellence leader certification or advanced problem solver certification. In some ways, that's great because that helps people take more courses. But I think the complexity of managing that in our learning management system has been a bit challenging. Can be hard for people to understand statuses, where to sign up for courses. Um, and I think that's probably more of a function of. Of the system itself. But also we could, you know, take a. Take a deep look. If we could go back, say, you know, do we need this level of complexity? Could we just make it as simple as just, here are the 34 courses and not necessarily have learning paths. So there are also badges, and I think the badges are a really great level for people. So you can think it's like we have courses, badges, and then learning paths. Um, and if we had probably just stopped at badges, that probably would have been enough to motivate people to take more content without necessarily needing another level of achievement beyond achieving a badge. Um, so if I could go back, I would probably try to simplify the program in as many ways to make it easier to navigate and communicate. Um, and also because as we make changes to the program, then that's more to communicate. Right? We. We probably made it too complex in the beginning. So now if we make changes, now we have change management and communications management around. Okay, why did we make more changes? Um, and that's actually kept us from, um, making all the changes that we've wanted to make because they're like, oh, now we're going to confuse people more. So at a certain point, it's good to hit a stabilization point, let people get used to it and not just, you know, while we want to relentlessly improve, we recognize there's a customer management component to that that can be challenging.

Speaker B: Certain amount of whiplash.

Speaker C: Yes. Acceptance of the change.

Speaker D: Yeah.

Speaker C: Um, so I have a question, and this is kind of a segue.

Speaker D: You.

Speaker C: You kind of mentioned something you wish you would have done different. What are. So let me preface this by saying that I have a lot of people attempt to design training, and I always get very nervous for them because I feel like they always underestimate how much it takes to develop training, good training. So what would be some of the best practices you've learned? Like, what has worked for you and the state of Vermont and you know, you have popular courses. But in terms of offerings and course development, what would you say are like the top three to five things that you felt like, okay, these are the things that we, the guidelines we kind of follow when we develop the training and have created these offerings.

Speaker A: I think one thing that was important for us is like being grounded in what our customers want. Like, what content are people most interested in and designing around that and understanding your audience as much as possible. Um, for example, our audience really, really likes virtual training. So most of our stuff is virtual, um, or on demand. So keeping the customer at the center. Um, but I'll say what, what's hard, been hard for me at times in this role is that, like, I'm not a curriculum developer or training professional, like, by education. Like, I don't have any certifications of this. I don't have a master's in education. So at times I felt like it's been hard to know, like, am I delivering this content in the right way? Um, and that's where our customer feedback mechanisms have just been really helpful for us because that serves as like our real time validation is like, yep, people really enjoyed this course. This is working. Or, okay, maybe we need to change out a few of the exercises here or there. So as somebody not formally trained in education, just keeping our people and customers at the center has been really helpful for us. Um, Justin, what would you add about

Speaker D: all the different learning styles, um, and trying to embed those within how, uh, we develop our content and our curriculums? Right. Um, we've got the whole vark category is visual, right. Auditory reading and then kinesthetic. And so trying to create trainings that meet all of those different needs can be really difficult, especially when we're dealing with a majority of our trainings being virtual. Right. So it's like, what are we conveying through visuals? What are we conveying through words? What is actually written text on there? Is there any opportunity for people to apply the concepts? Um, which activities are going to be polls? Which ones are going to be group work and breakout rooms? Like, thinking about all those dynamics, that's a lot. And you're trying to embed all of those within a given training. That again, three to four hours can, uh, be a little bit difficult. I think by and large we do a pretty good job of that. Maybe better in some areas than other others. Um, but it's definitely part of the design process. Like, how's this going to land with everybody and are we going to be able to meet everybody's needs because they're different?

Speaker B: Just, um, out of curiosity, because you just mentioned a few things, do you try to work in, especially with the on demand versions of your training, a fair amount of checking for knowledge, checking for comprehension.

Speaker A: Ah, great question. Um, so this also takes me back to like, what would we do differently? So when we were recording our trainings, um, we essentially recorded videos of ourselves in PowerPoint talking about slides. And then every like five or seven sides or so, we would have a quiz or a knowledge check question to serve as the interaction interactivity within it. Um, so the interactivity was relatively limited and it was more of kind of like watching a recorded, um, webinar of somebody, like presenting. And at the time we didn't really have a lot of deep knowledge or expertise in like, professional training development tools like Captivate, um, Adobe Captivate and others. So in some of the new trainings that we had been working on recording, we were building those now instead in Captivate. So almost every slide has some type of clickable component like expand to explore, um, quizzes, um, click to reveal type things to make it a lot more interactive. Um, so in our first iteration of recorded trainings, it was a lot more passive. We were doing the best we could with the knowledge and tools that we had at the time. But, um, some of the feedback is we definitely want more interactivity and more engaging content to keep your attention. And keeping that short. Right. Like 30 minutes or less is probably the best, um, for recorded content. So moving forward, I think that's going to be a larger component of it, um, because I think that's how adults learn best. Um, and it's easier to stay engaged in that way.

Speaker B: Yeah. Tracy and I both teach at UC San Diego, um, Health and then, uh, the broader university. And during COVID which is when I got brought in, suddenly everything that was in person was online. But of course, as you've probably experienced, you can't just go, okay, now this is online. And, and you've gotten more subtle as you've gone along, like, in terms of the interactivity. But, um, yeah, we, we had this like, real revelation, like, okay, we've got to work in discussion, we've got to work in some kind of check for knowledge and we've got to like. And then people got used to during COVID like, well, you can make dinner while you're watching some video teaching you how to teach Japan, uh, speak Japanese. So it's like, you know, people have this expectation of like, yeah, well, I'll just be over here, you know, knitting while this goes on. It's like. And so I think what you found is so critical. It's like, no, we've got to engage with them. And then you're founding, you found the um, technology to help you do that. So that's cool. So I wanted to switch and ask about. Because you also have a community of practice. Right. So I wanted to know sort of what was the genesis of that and, and what happens there now.

Speaker A: Yeah, so we have several communities practice. I'll talk about our Continuous Improvement Network and Justin can, can share about Tech Hub. Um, but this actually originated out of what was originally our green Belt network when we had um, the belt program. And that was essentially, essentially just a place for green belts to connect after training, to stay engaged, um, share improvements and connect with each other. And when I came into this role, I saw the opportunity to, you know, say, hey, why don't we just do this for everybody in state government? Why does this necessarily just need to be exclusive for green belts? So then we opened it up to everybody in the state and it's now the Continuous Improvement Network and has about 1500 employees in it, which is about 20% of the workforce. And, and in that community we have, uh, ongoing webinars like our Continuous Improvement speaker series. We've had like quarterly series at our deep dives on specific topics in government, um, ranging from how to use tools like DocuSign to accessibility, uh, to um, uh, digital, other digital tools in government. And something we've started doing recently that's been a big hit are the Share and Learn sessions. So these are where anybody from the community can come and just listen or participate, participate around any given topic. So the one we did last month was around email management tips. So the topic is come and share your email management tips or your task management tips or your meeting best practices. Each month we'll have a little bit different of a topic. And it started out as an experiment and I honestly thought it was just going to be crickets. But everybody is so excited to share. We always run out of time. We have more things that we want to cover. We usually do them for like 45 minutes just in case we get some crickets and we have a little easy out after 45. Um, but that's been a huge hit. And I think that succeeds simply because it's just easy, practical things that people can take away, you know, and start using then, right, like, oh, I didn't know I could like automatically take all of my um, auto replies and send them to a folder so I never have to look at them. Like really practical tools and tips that people can use right away. Um, especially around like Outlook or Calendar management, things like that.

Speaker B: Um, so that's sudden urge to stop this interview right now and just say, okay, give me all the best tips that came out of that because we need to know them all. Auto reply to a folder. I need, I need that too.

Speaker A: Yeah, and we do. Yeah and we do like AI generated summaries after that as well too. Take the transcript and have it summarized. So even if somebody couldn't make the session, they still can have a record and notes of all the tips that were shared. And I learned something every single time myself. And that's what's amazing is like normally we're the one sharing tips and teaching and I'd love to be able to learn from the community because there's so much great knowledge out there. Um, so that's our continuous improvement network. Um, but within that we also have functional communities of practice around core functional areas in government such as like grants and contracts, project and change management, um, community engagement, like cross cutting. Things that are happening in all different corners of government. Yet you know, somebody working on grants and contracts in agriculture is facing similar problems and challenges as somebody in natural resources. But they historically have never had a place to connect. So those communities have been a big hit. Um, and those are run entirely by our volunteer facilitators who are subject matter experts in those fields. Which is great because like I am not an expert in grants and contracts so it would not make sense for me to facilitate that community. Um, so we are a bit reliant on their volunteerism and ongoing engagement to run those. And we have had a bit of turnover in some of our communities. We've had to close a couple of them just because we didn't have facilitators willing uh, to continue to run them. Our financial services and our data management, uh, analysis group we had to close within the last couple of years. Um, but that's okay. We still have a lot of other communities that are, that are running strong and they're monthly meetings that are just great knowledge sharing spaces for folks in government. And Justin, you want to share about tech?

Speaker D: Yeah, so I should probably also share tech training which we didn't talk about in terms of kind of new things on the training side. So I'll hit that briefly. Um, but uh, we Also do tech training, uh, because we saw a gap in what was available at the statewide level. And like, our focus on tech training is really about productivity. Folks have these really awesome tools at their disposal. They don't necessarily know how to use everything. Um, we feel like if they did, they could save themselves a heck of a lot of time. Um, so we have a, uh, 12 course progression that I run in Excel, teaching the basics of, like, here's how you organize your data to. Here's how you connect to different data sources and scrape this website, get this data in and do pivot tables and reports and all this stuff. Uh, so we've had a lot of success with that. People love those. We typically have wait lists for those courses. Uh, we just launched a new SharePoint badge, which is six courses. Uh, three of those recorded, three of those live. Instructor led with me teaching people how to use pages and document libraries and lists. Uh, we've got another badge probably coming out soon on Power Automate and then a final one on Power bi. I'd, um, say between the two different types of training, I probably prefer the tech training. I love the CI training. I did it for years before Katie and I've done it with Katie. But the tech training is so tangible, people walk away with a direct skill. They're like, yep, I can now do a pivot table. I could not do that before. And that feels so good. And I get a lot of great feedback from people that are like, oh my gosh, this has saved me so much time. So I love the tech training for that dopamine hit. Every time that I get one of those emails from somebody. Um, so in any case, like, that's happening and then similar to what we have on the continuous improvement side with trainings and then these communities of practice, we also have a tech community of practice, which is called Tech Hub. Uh, that started around the COVID era when, uh, everybody, um, home and suddenly you got to use Microsoft Teams and nobody knew what they were doing and there wasn't a lot of training available. We stepped in and said, hey, we've been using teams for a while now. Let's start doing some training, get people together, talk about this. And so we used to have two different teams. One on the Office 365 suite, one on Power platform side. And we merged them somewhat recently to just make this overarching Tech Hub team. We've got 3,000 people in that network. Uh, we have channels for all of the different applications in the Microsoft suite, so people can ask questions about those Like Excel or Outlook. We also have things like DocuSign and Adobe and a few others. Uh, but similar to what's going on in the CI network, you know, we have monthly sessions that occur. I will do a tech training series, 45 minutes every month, uh, on some new thing that's come our way, uh, or a topic that people don't know much about. So we just covered quick steps and quick parts and Outlook and Word and SharePoint lists. Uh, we've done a lot of other things in the past. Uh, we do an Open Office hours once a month as well. So myself and a bunch of other tech experts from our agency of digital Services, uh, get on a call and we answer people's questions. I think about it as like tier zero triage. How do I do this in Word? Oh, uh, I'm getting this error in teams. Why am I getting that error? The user interface is different. Why is that? We just get on, we answer these questions for people. Uh, they tend to be very well attended. The tech training series is sometimes the Last one was 114 people and then open office hours is like usually 60, 70 people that are hanging on, listening, asking questions. Um, but those have become really great spaces for people to just share what's going on with technology, ask questions, get support, learn from others. Uh, I also tend to announce new features that come just today I announced the change to Microsoft Teams with this draggable divider. Um, because I want people to know about this new stuff that's just showing up in front of them that um, they might be curious about. So yeah, it's a really great community. Again I love the tangible aspect to it and I think it's a really, uh, great add on to the continuous improvement work because what Katie and I have found in a lot of our improvement projects is people's ability to dream of what could be better in terms of a future state is sometimes limited by what they know technology can do. Uh, and so once we expose them to that possibility, then all of a sudden their minds are open. Oh, we could do automated emails. Oh, we could do this. Oh, our data could go over here. And then now they're starting to dream pretty darn big in terms of potential.

Speaker B: That's awesome. Yeah, Tracy. I know Tracy's having a little bit of ah, audio. Um, if you want to come to

Speaker D: Open Office hours, Tracy, you can join in.

Speaker B: Yes, honestly.

Speaker A: Yeah.

Speaker C: Ah, but uh, I also love the AI aspect of everything. You know, there's so much coming and I feel like so many people are feeling Like, I know, I feel like I'm feeling, falling behind. And um, I'm just excited hearing that you are, that you're allowed to and you're offering this kind of stuff because it is so needed in so many places. So that is kudos to you. But yes, I'm having technology issues today.

Speaker B: Yeah.

Speaker D: And we did to that point, we did start up an artificial intelligence community of practice fairly recently, two or three months ago, because that's the new thing coming. Uh, and we wanted people to have a space to talk about it and learn from our director of AI. But like, what are the best practices? What should I be using? What's safe, what's not safe? Uh, so, yeah, that's been well attended also.

Speaker B: It's like Justin, you've turned into the tech whisperer. So you're for, for those only listening. He's whispering now. Um, so I just wanted to touch on one thing before. I'd love to hear about your, um, uh, it's a performance logic model you mentioned that's got me thinking. Yeah, let's just hear about that.

Speaker D: Yeah. So, gosh, hard to know where to start with our logic model. I will ask this question. Both of you are familiar with sipoc,

Speaker B: the High Level map, Not Son of Spock, but.

Speaker D: Yes, indeed, yes, indeed. That's a logic model. Right? It's uh, right. Suppliers, inputs, process, outputs, customers. It's a very simplified version of a logic model. Um, so we have built something similar to that as a framework for programs and individuals and teams to really think about performance. Uh, one of the things that we found over time is that people have a hard time with data. Um, they have a difficult time knowing what to track and when they have measures, which ones are the most important. Um, we were like, well, how can we help people with this? There's a lot of different methodologies out there. We've used some of them in the past, like results based accountability. There's okrs and balanced scorecard and all sorts of things out there. And we felt like they didn't quite fit at all the different scales that we wanted to use them at, whether it's strategic level or programmatic level or operational level. Um, we also felt like when we look at some of the stuff that's out there on Lean and also like PDSA cycles, that falls flat a little bit on the data side. Um, like Mark Graven has some really great stuff around data and there's a bunch of other books out there on data. But like when you look at books that are all about PDSA that like the S part of that is pretty thin in comparison to the rest of the book. There's not a lot of talk about what are you actually studying, what are the right KPIs or the right measures to gather here. So we weren't finding a lot of help in that space either. So we decided to just kind of create our own. Uh, we looked at some evaluation work and like logic model work that's uh, happened in other states. Program theory, we pulled from a bunch of different pieces and put it together into this, what we call the Vermont Performance Logic Model. So I'll share that with you now so you can get a sense of uh, what that looks like. So hopefully you can see that now. Uh, and this is a simplified version of it, we have more complicated versions and I've changed it up a little bit for this presentation. Um, but no, I don't want to leave the meeting. Thank you. Uh, so basically we've got these, these four buckets. Uh, so we think about it. Could be a process, could be a program, could be initiative. I'll start on the left hand side here. Usually, uh, have a team of people, uh, that are brought together to do some level of work. It even could be an individual. Uh, but it's usually a group of people. Um, and we have the ability to understand through measurement what the culture of that team looks like and what the capabilities of that team are. Right. So this is where like the world of workforce development typically comes into play. Uh, we're thinking about engagement, we're thinking about accountability and ownership, clarity, collaboration, improvement, diversity, equity, inclusion, all of those things that exist at the team level that we know. Thinking about respect for people are really important things to be tracking and to be managing because we want all the metrics to be really, really good. Right. Um, so we start our logic model on the left hand side with people. Uh, because we believe that people are the ones that provide the most value in an organization. They're the ones that are doing the work and that team environment is really important for actually getting things done. Um, and what we found also is when we think about the values of an organization, they typically play out in the day to day interactions of a team. So there's a really strong connection here between what's happening at that team level, how we're measuring that and that connection to values that we've put out there or have espoused. Right. So that's the first part of our logic model is let's think about the team. Are we gathering data about the team. Do you understand engagement, collaboration, all of those different components. Uh, so if you want to manage a team, you got to have those metrics, you got to know, you know what's going on there. Really important. Uh, and then we move to this next part which gets into mission, which is the work. And this is where that SIPOC comes in. We're thinking about inputs, we're thinking about process and we're thinking about outputs. All of those can be measured from a uh, quantity standpoint, from a quality standpoint, from a timeliness standpoint and a cost standpoint. So we can attach measurement to our inputs, we can attach measurement to our processes and we can attach measurement to our outputs. Right. Um, and that's going to tell us something about the work. And generally when we're looking at those measures, we're in this realm of efficiency, which sometimes lean. Continuous improvement focuses a lot on efficiency, which is great, it needs to, um, but it's not the only thing that we really should be caring about. But that is the efficiency space. That's what that data is telling us. How well are we turning inputs into outputs? Um, but beyond that, and this is pretty, I would say it's pretty specific to government or kind of social sector. Um, but outputs are not the end of the line for us. In government we are attempting to affect some sort of change. Uh, so we want something to happen for the better as a result of our work. And this is the results bucket which is telling us something about effectiveness. Um, here we're trying to measure the short, medium and long term effect. So change, uh, on either knowledge or skill, behavior, circumstance could also be status, could also be uh, condition. If we're not talking about people like customers and clients or. But we're talking about pieces of infrastructure or systems that we manage. Right. So there's a whole body of data that we can collect that tells us something about are we actually making a difference in our work. Um, and then finally we have this last piece, um, which recall in the why, uh, typically we use the term outcomes here. Um, but these are conditions that we're tracking that are beyond generally the work of our individual programs that go away from our individual customers and clients that are like kind of big trends that we're tracking. So in government, you know, it's metrics around the economy, metrics around uh, the health and well being of people, water quality metrics, like big data. Right. That's like telling us blinking lights in some cases. Hey, there's some conditions here that are not great recidivism rates are on the rise. Homelessness is on the rise. Right. Health of Vermonters not going so great, whatever it might be. These are things that we're tracking, but we don't have individual programs responsible. Responsible for. But that's really important data to let us know whether we need to respond differently in terms of our programs, in terms of the work that we're doing. Um, and that data we call indicators. Anything in the mission space, either work or results. Uh, we use the term performance measures, and then on this value side, we, uh, distinguish those as cultural measures. And that's helpful from a lingo standpoint to make sure everyone's speaking the same language. Um, and one thing I will say is hard is that term KPIs gets used a lot. We feel like it gets used in sometimes the wrong spaces. And so we're very intentional when we hear that. We're like, are you talking about trends that we are tracking that we do not have control over, but we care about? Or are you talking about performance measures where we do have some level of influence and control over what's happening? Or, uh, are you talking about the people and the work environment and the culture? Um, right, because we want to get very specific here because that. That matters to us in terms of what we might do. So in terms of, I guess, scale on this, Right? So we found this works at an operational level. Uh, you can get into the work and break down a process into inputs, processes, outputs, take a look at quality, et cetera, all of those things. You can operate at a more programmatic level and ask the really important questions. Are we actually being effective with the things that we're doing? Are we doing the right things? Are we getting the results that our program was intended originally to actually achieve? Yes or no? Um, and we can work at that very high level, strategic level, where we would say, gosh, what do we need to do differently here to get better results? Not about continuous improvement right now. It's not about doing better with the same work. What do we need to be doing differently? Because we're not seeing the changes that we want to see as a result of our programming. Um, so for us, it works at all of these different scales, uh, which we really like. And we're constantly refining this as we work with programs, we communicate it more and more. Um, but I'm really happy with kind of where it is right now, uh, and being able to go into kind of any conversation and box things in, right? Or work with teams and be like, okay, you want to develop Performance measures. Okay, what are your cultural measures, what are your input measures, what are your process measures? And then out of all of those, what are your priorities as let's say a program manager or a team? Um, what are the main five to seven here that you really want to be focused on and you're going to track in a dashboard and you're going to have regular, like business review meetings around. Yeah, so that's, that's our new logic model. It's what we've been training on for year, year and a half now, two years. Katie, anything you want to add on that from your perspective?

Speaker A: Yeah, I really love this as a brainstorming framework with teams because it makes sure that we cover like all of our bases. Because without being grounded in something like this, people have a really strong tendency to focus on the easy things to measure.

Speaker B: Right.

Speaker A: The bean counting, the number of applications reviewed, the time to review applications, but what are the results of that application process? Um, so we tend to have a tendency to focus on this blue work bucket and count really easy things when we don't have a larger framework within this to operate. So it enables us to make sure we're, we're working in each of these categories equally and not just going to measuring what's the easiest or simplest to capture.

Speaker B: That's nice. I also love the distinction between, um, are you just capturing indicators or these things we have control over or you talk about people. So I really love that you've created the buckets of, to really refine those distinctions. Um, and like Katie's saying, then you can say, okay, that's just in the work metrics. So let's make sure we look broader because you've kind of opened up the world there. So, um, well, thank you a ton for sharing that. This is fascinating and I love that you're, you obviously you've worked hard on it and then refined it to make it work for you. Um, and that's great. So, um, is there anything that we haven't brought up and asked that you wanted to tell, uh, our listeners?

Speaker A: I guess since the last time that we've connected, um, we've also been putting a lot of intentional time into recognition efforts as that's a really key component of reinforcing a culture of continuous improvement. So last year we launched a new annual improvement awards from our office. We have a, um, improvement sharing form that's been in place since like 2017. And we essentially just picked, you know, the three, um, most standout improvements from that. And it doesn't have to be a big improvement. It wasn't the improvement that made the biggest impact, but you know what was one that made a really big impact for the team that it, that was working on it. Um, so it can be small scale, small m scale improvements can certainly still have a big impact. Um, we've also started working with the governor's office to send letters of recognition to people who have completed advanced levels of training within our program. And we've also partnered with the administration on Public Service Recognition Week activities as well. So that's an area that we worked on building a lot in the last couple years that I think is having a good impact with our audience.

Speaker B: Nice.

Speaker D: And the CI network Katie mentioned becomes a really good avenue for us to communicate all of those recognitions. Katie just did one today recognition post for all of the people that have gone through our different like badges and learning paths. Um, that happens on a quarter quarterly basis. And that's great, right? Really good visibility across a lot of people.

Speaker B: Yeah, that's great. Using that to raise the visibility and also acknowledge people and the effort they're putting into this. We will put uh, your LinkedIn addresses so people can reach out to you if they want to hear more. You have opened probably a lot of eyes and ears to what is possible and you're applying this obviously to state, uh, processes. But what you're talking about is all any organization could, uh, use and would benefit by. So this has been just great to hear about. Um, thank you both so much for coming, uh, on and telling us all that's gone on since we last talked.

Speaker D: Thanks for the opportunity and inviting us back. Really appreciate it.

Speaker C: If you're living in or around San Diego, I will be the instructor for the upcoming in person UC San Diego Greenbelt class starting on May 7th. So it's every Friday for five weeks from 9 to 3pm with a project presentation in July. And the Problem Solvers Toolkit is a recommended read as part of the course. So you know it's going to be a guaranteed adventure. So we're going to leave you a link if you're interested in joining me on this educational journey. Because you know what, as much as everybody's online, I still like in person training the best.

Speaker B: Yeah. And then you get to go to San Diego. Of course, you had to sneak that in even though it's a northeast episode. Um, also, if you didn't catch. If you didn't catch it, last month's episode was a good one. Culture is not an app with emergency physician turned startup founder Greg Jacobson and Lean author, podcaster and Kynexis senior advisor Mark Rabin. We got to unpack whether the Kynexis software can actually strengthen a culture of continuous improvement. And we learned how the Kynexis journey started in healthcare but has grown to embrace its cross industry impact by drawing on habit loop science like we talked about recently and the power of the nudge. So lots of great nuggets in that one.

Speaker C: Yeah, lots of good nuggets. And lastly, my third parody music video is almost ready for release and I can't wait to share it with the world. So stay tuned for the launch party sometime in April, probably the end of April. And uh, stay tuned for all the news by joining our community at the Just in Time Cafe.

Speaker B: Hear hear. Thanks for joining me and Tracy for another episode. You are our favorite thing about the Just in Time Cafe. We look forward to being your lifelong learning partners.

Speaker C: Join us next month and every month for your jolt of lean caffeine.

Speaker D: Sam.