Ep. 018: Metrics That Matter - Data Points for Technology Leaders | w/ Special Guest Sebastian Kline of Nelnet Community Engagement

You're listening to another episode of Open Source CXO, the podcast designed to share insights on how to excel in your business using technology, regardless of the industry. Host Robert Kehoe is a self-taught software developer who has grown to the role of CEO. Renowned for his collaborations with organizations such as Stanford University, Nelnet, and Louis Vuitton, he continually seeks new challenges to conquer in the world of tech. Joining him is Don Blackburn, a veteran COO with over 25 years of experience in cultivating diverse relationships and driving innovation in various technical projects.

Each week, they'll be sitting down with some of the nation's foremost technology leaders to develop an open source playbook drawing from their firsthand experiences in the field. Let's talk some tech. One of the things that I haven't really, you know, I would love to touch on some more and could be an entire topic all by itself, but if it is, that's totally cool. But I'll come back.

The metrics used to measuring performance, I think is something that has really, it's stumps and I know there's it's so nuanced. It depends on the team. It depends on what you're building. It depends on so many different things.

So you know, I know you guys have worked with with vendors, you guys have internal staff, you guys have offshore. So there's a whole variety of pieces that you guys work with. I was curious, how do you guys measure performance and and and how do you use it to sort of make the whole metrics thing for our whole audience? So I will.

I'll share a spreadsheet with you. I you know, because I know that we were measured based that you guys had some sort of way of kind of measuring us more so than anybody I've ever seen. So you guys took it to a new level where you were really you were taking a lot of data points and a lot of comparing a lot of different things. I just think a lot of companies, they see the cost.

That's like the biggest key key metrics. And then they go, OK, well, they don't really have proper measuring tools. They take the cost is like the biggest factor. And then they look at what's being done and going, OK, well, I think we could probably get by with with making this decision or that decision.

And it's it's just it's they're terrible ways to kind of make decisions. So I'm just very curious how you guys you guys had a great way of really making some good choices. I'm just curious how you did it. Right.

Yeah. Give me a second. My my squirrel brain is like, oh, I've got so much to tell you. And oddly enough, the very first thing that my brain to is this.

I mean, I want to buy it. I want to buy this thing. It's either get a shirt or a hat. But anyway, it says freak in the spreadsheets.

And that's if you want to run it, if you want to be like you get into management, like there are when I wrote code, I would plug in and I'd have like focus music or writing code, you know, get that like trance EDM house. Right. Yeah. Tropical House Radio.

You know, and now and now I listen to the same stuff and it's like, you know, I can't wait to write this Excel function. You know, I think step one is building your acumen around spreadsheets. But when I think about some of the stuff that made us really like. There are a lot of metrics you could track.

And I think there's like Dora and space. And recently I was as president at a presentation where someone was comparing and contrasting the two and I thought, oh, my gosh, what a myth. Like they were talking about Dora. They were talking about space.

They even like went so far as to flash of, you know, a screenshot of Dora. I thought, oh, my gosh, what a myth. They could have done Dora in space. And that was like the point is like, you know, engineering metrics are a balance between metrics which track objective measures, right?

Things like what's the age of a pull request? How fast is the team moving? You know, what is our capacity? How effective are we at committing and delivering against the work or, excuse me, delivering against the work we're committed to?

But there's so much that drives that. You know, I read a book a couple of years ago called 4DX, it's four disciplines of execution and it's by Franklin Covey and it's got a bunch of really wonderful case studies in that book that describe, you know, how you achieve your wildly important goals. And when I think about how we approach metrics, it's starting with, again, the business problem. What is the business problem we need to solve for?

You know, why I've seen a lot of articles around why you shouldn't track velocity because that's the only thing they track. It's one aspect, how fast are we moving? That's one aspect. Also Goodert's law comes in, you start telling people we're tracking velocity and what are they going to do?

They're going to long a bunch of, yeah, you know, tickets that maybe are fluff. Exactly. Yeah. And so that's where you've got to have some monitoring in place and your management team can help with that.

But velocity is not the only measure. There are things like age of PR. There are things such as deployment frequency, which is one of those DORA metrics. And you know, some of the things that we track that are more broad, we keep track of the work that we said we want to accomplish relative to how much of that did we accomplish by the end of the sprint.

So on plan, off plan. And so you can look at that and when a metric is off, what do you do? You show up and you go, oh, your metrics are off. And you leave it?

Like, I remember, gosh, I made such a mistake. When I switched from being a manager, I thought, oh my gosh, my job is to help all these other managers understand our metrics. And that's kind of true. But what I didn't recognize at the time was my role in that was to help ask questions, help guide the conversation, help facilitate and craft a framework for which they could go explore the root causes.

And so when you think about metrics, you can have metrics that track everything under the sun, but that's not going to drive higher performance. It's metrics married with coaching. And so when you think about a strong management team, it's folks who can coach. I had the pleasure of managing a very seasoned manager.

He also, no surprise, was a soccer coach on the side. Oh my gosh, wonderful coach. His expertise was not necessarily in software development, meaning the coding. His was more a user experience background, but he was a wonderful coach for the folks who reported up through him because he understood his job is to observe and then hold conversations about what he's seeing.

And so metrics are your eyes when you get to be a larger organization. Metrics are your eyes so you can share observations. So you go into one-on-ones. Those are typically the, unless it's a widespread issue, one-on-ones are often the best place to dive into it.

I think the default question or the default sentence stem that people use when they explore something is why. That is a terrible question. That is actually really bad. Because like Don, why did you choose to wear a black shirt today?

That comes across almost accusatory and now you have to justify and now you're on the defense. There's so much psychological dynamic there. Whereas if I were to start the sentence, something like, oh, interesting, you know, Don, is black your favorite color? Right.

No. It's slimming. I mean, Robert's into it. It's sad that it's slimming and I still look this big.

Mine's blue. I don't know. Gee, I don't know. I would say, yeah, blue is probably closer to my favorite color, but I like these black.

Of the shirts we had, black's probably my favorite. What am I going to do to score one of those? You've got to talk to Josh. He should close it over there.

Just to kind of run through a few of the metrics we track, things that I've found important to our growth. Tenure is a big deal. You want to talk about high performance. I think retention is a big deal.

We were talking earlier about how to connect employees to the bigger purpose as part of the retention play. Why are you here? Why does your work matter? There's also- And that matters because you've invested time and money and education and all that into yourself.

Definitely. I still find it fascinating that companies would literally be like, all right, I'm going to let all of you. I can name several right here, right now. I'm not going to.

It's just sad because they'll let- I think you can drop after. I think I'm going to, but- They'll let whole groups of people who have been there for a very long time have a vast amount of experience and knowledge in the platform and then totally just shift directions, whether that's offshoring or whatever it might be. But it's unfortunate. I hate seeing it.

Tenure is a really big deal to me personally. If you want to predict team performance, a lot of times you can look at the tenure of the team and the teams that work together and have run through more cycles together will be naturally more efficient, even if their skill sets aren't commiserate. So this tenure is a big deal to me. Of course, equity, pay bands, comp, that's something that I do keep an eye on because we want to make sure that folks have a future that they feel valued.

They feel that they are paid for what they are contributing that goes into connecting them to their why and then showing their future career growth, making sure that they're growing. But when it comes to productivity, there's a real easy one, which is platform uptime. How often do you have an incident? On an eight-week average, how frequently are you having an incident or something of critical level that requires escalation beyond a regular bug?

We want to keep track of automated code coverage. I think this is one that I kind of knew a really couple of years ago. I found myself so frustrated because I care about code coverage for the reason that we have a continuous integration, continuous deployment pipeline. Our automation is the path to speed.

Having a CI-CD pipeline and being able to trust it is so important. That's my why. But whenever I say code coverage, immediately I hear critics come out of the weeds and they go, you want us to do 100% code coverage. I'm like, no, no, I want you to test the things that matter and make sure that when they break, they tell us.

I don't expect 100% code coverage, but it's good to know how we are progressing against that goal because we use CI-CD as our mechanism for deployment. In conjunction with our QA teams, the combination of those two make it so that really we're doing as best as we can to minimize the amount of regression or bugs that get into production, catch them before they get there. So one of the coolest metrics that I've come to love, and it's, I don't know, my love for QA has actually grown quite a bit within the last two years.

QA has saved my butt personally and many other people's butts. QA is your friend. I can't say that enough or loudly or, yeah, they're not there to tell you what you got wrong. They're not here to tell you what you got wrong.

But we track this. They're to keep our customers happy with the product, keep the quality high, and among other reasons. And so we track a metric, which is a ratio, between how many customer detected bugs came in this week relative to the number of bugs that we detected and found ourselves. Meaning if you're working on a ticket and you find a bug, great, report it.

Get it into JIRA so we can know about it. And the goal is to find the bugs faster than the customers. And so incentivizing that behavior and leading a little bit into Goodert's law there, we want that. That's big.

That's interesting. Yeah. And again, when it comes to feature development, I think there's the, hey, we want it to build this, and then the engineering teams got involved, we built something, but along the way we discovered things. We track missed requirements.

How effective were we at planning with some accuracy? What's our scope growth on projects? How do we bring those into retrospectives and hold conversations? Hey, our scope growth on this project was this.

Our missed requirements counts with this. Are there ways we might improve the process? Are there things that we could have done to plan and deliver with more accuracy? And I think it's just always recognizing that no one intentionally does something wrong.

If you lead with that idea and you're able to lead with curiosity and explore what behaviors existed to create this scenario check, you can explore and get away from a blame culture because it's very easy to say, well, they did it wrong. Right. It's like, well, no, it's probably just a matter of change of behavior. Yeah.

Yep. Those are very good points. I like that. Now, Robert, I'm curious.

So you used to write a lot of code. How much code are you writing right now? They don't let me touch anything anymore. I write, we have internal software we use.

We have our own CRM ERP type software. So I get to work on that. Nobody likes touching it because I don't really follow all the same patterns. I find myself a lot like, oh my God, I need to go and fix this real quick so I can do something.

So I'll go and I have a lot of band-aids in there. We really need to redevelop all that. And I get to follow our own process. But see, that's my problem.

I run a whole company though, so I don't have time to sit here. So that's my only, I don't get to work on any client facing projects anymore. I do a lot of consulting though. How did you evolve out of that though?

Because I think there's probably some red thread. I've never had anybody ask me this many questions. Sorry, I'm just so curious. I started the company with a partner early on and my job quickly sort of evolved because he was a better developer than I was.

It's very clear. So he sort of just took on that piece a little bit and then I slowly evolved into more of a business development type approach. But we split not too far into the creation of the company. And I'm like, well, I still got to do all these other things.

So I need somebody that I could trust. So I sort of managed it a little bit until I found somebody that I could trust. In that, there's so many stories there about trusting the wrong people. But we won't get into that.

So it just sort of, it kind of just started off that way. I had somebody that I could trust, so I did. He handled sort of the development side. I did some of it too.

But I had a background in design as well. So I did usually a lot of the UI, the design stuff as well. Yeah. But I was a freelancer, so it was sort of required.

I was a freelancer for a long time. So that was just sort of required. And it just happened that way. But thank God, because we're around.

You used to have to jump in and save projects every once in a while. Yeah. No, I did. It's been a while, but that goes back to the trusting people who I shouldn't have trusted.

And I wasn't really, I'd never hired anybody. I was a freelancer. And I'm like, all right, well, surely these people tell me they can do it. They can do it.

That's not always the case. So I did have to jump in a few times and really just, I would go to coffee shops until they closed just to try to get certain things done and whatnot. So I had to do what I had to do. But we've come a long way.

So that's sort of how it evolved. I'll tell you, so my career progression was, I came in, well, first off, I never wanted to write code, to be honest. I did not go to college. And it catches some people off guard.

They're like, oh, you write software. Don't do it. I guess I didn't realize that either. Yeah.

No, I, oh man, I broke my parents' heart because they were all college educated. And my senior year, they're like, where are you going? And I was like, I have no idea what I want to do with my life. How about I don't?

So then I did a bunch of stupid stuff for like six months, gotten plenty of trouble. And then woke up one day, I was like, gosh, I should probably do something. So I started a digital marketing business and I did that for a couple of years. I was just going out selling stuff.

I wanted to shake your hand and convince you that a $10,000 website was the right path. And boy, that was fun. I miss making the packets. Oh, they were so good.

Next stock, we got to a place where we'd spend like $30 on a packet. We wanted to make, you know, you're signing up tens of thousands of dollar deal. Let's make it feel that way. Sure.

So started there, got exposure to software development. And then I went to work for a startup that was called Business Instruments. A friend of mine in Kansas City had that. And you know, I was doing some mobile development for them, some skills that I had sort of kind of acquired, but was able to, you know, learn as I went.

And then got into the, you know, my first what felt like a big deal was, you know, working for a wear three, right? Doing front end development, doing back end development, doing iOS and Android development and just kind of trying to cover all there where I could. And you know, getting into technical leadership eventually, then, you know, go and being one of the early managers. And then from there, being the first manager of managers aside from my boss at the time.

Yeah. That's interesting. You transitioned easily. And I appreciate that because that's one of our, one of the things we always do in the journey to leadership is what we call journey to leadership at the end of every podcast.

We try to figure out, because it's so different for everybody. Some people have master's degrees, right? And they always wanted to be in management and they push that way. Other people totally fall into it by accident, right?

It's like, can I tell you something? One of the things that, you know, I'm working to get an executive coach right now and go down that path to help with this. But one of the things that's such a challenge is when you move up so quickly, the imposter syndrome grows exponentially. So that was one of the questions I was going to ask you is you're, I mean, you're not that young, but you're young for being in the position you're in.

You're certainly young and your, your ascension to that spot has been very quick. Right. So strapping, it was a rocket. Have you dealt with that?

I mean, outside of the imposter syndrome, have you dealt with it from peers or teams? You know, of, wow. So, you know, like getting that respect at a young age, you know, to manage that many people. And so a couple of things that I had going really well for me was that when I was a software engineer, I learned that how to, how to make a mistake and own up to it in a responsible way.

And it also built the trust with the people with specifically Joe Terry and Scott Connerly, who were my bosses. You know, I built like just at being a software engineer was that no blame culture we were talking about earlier. Right. I learned, I was like, okay, this is, and you have to understand, I grew up, you know, my parents were a little more critical of my actions and how I went about the world.

And so I was naturally like, oh gosh, where am I going to get criticism? And they created the right space for me. I'm not confident that it would have gone so smoothly if it weren't for that. Because as I, as I moved up, I was always asking for more.

You know, I wasn't, for a while I thought I wanted to be an entrepreneur. I think I'm more of an intrapreneur. I love innovating within companies as opposed to maybe like, I can, I can, you know, show up after you've got through some of the, you know, eating ramen noodles part. It's just being honest, you know, but, but I'm happy to, you know, go with you from like two people to a hundred.

Yeah. Like let's do that. That's a fun journey for me. Yeah.

No, I agree. But, but, you know, what are some of the challenges you've got to let go of responsibility? And I'm going to specifically with, I was asking you earlier about how much code are you writing and what are you working on? Because I was thinking about this very thing.

So many people who go through an evolution, like I did struggle to let go of the code. Sure. I, I'm not going to lie that that's where my love was. I said, I, I still do it because I love doing it.

You know, I just do too much now. I can't take. I think once you move above a frontline manager, they should, you know, the devs, the devs should revoke your API access and turn your GitHub account into a read only like access. Yeah.

But you should be able to get in and see the work, but yeah, honestly, we had writing code and so that was, that was actually something that was very intentional for me. One of the, we'll see who listens to this later and then is surprised. But one of the things I did a long time ago was I, I got a new laptop and I never set up the new local, like we had a local development environment where you build and write and iterate on code. I never set it up because I was like, no, no, don't touch it.

You will fall into that hole. And that was a way for me to just very consciously say enough is enough. Stop. There's something to be said though when a leader gets involved and does the job of an individual contributor, there's a dynamic at play there.

Sure. It is. And, and you don't mean it when you do this, but like if I were to jump in and start doing the, like a critical issue rolls in critical issue, oh, stuff's on fire. And I jump on it.

Yeah. That, that's, that just shows, yeah. Like, is no one, do I not trust the team? Yeah.

I can make sense. No. What is my role? My role is to make sure that the, that this gets the level of visibility that it is deserved and that the management team knows, oh, that's something that CBAS, I go by CBAS professionally.

Sure. Yeah. I still, I don't even know why, but I, every time when we talk about you, that's what I call you. I don't call you that.

Yeah. So, you know, CBAS. We've been calling you that before you. Yeah.

Exactly. And I said, this was important. Right. They know how to act on it.

And that's my responsibility is knowing, making sure that they know how to act on it. Right. And what to do. My responsibility is not to work with them.

That's good. You can have a team that you trust to do that. In my particular case, that was a bit hard for a minute back in the day, but yeah. Yeah.

No, that's, and it's a good point. But that's also unique to you because you didn't, you didn't grow up with the passion to be a developer either. That's right. You kind of fell into the development side, so it's probably easy to give it up versus somebody that- I started when I was like 13.

Exactly. I mean, that was what he wanted to do from the very beginning. That's why I was a freelancer. In fact, I told myself I'll never not be a freelancer.

Here I am with a company. I don't know if I'm an entrepreneur myself. I just kind of fell into this. Well, you've kind of done something pretty great.

So, I don't know. Thank you. Thank you. I like it.

It's not stressful a lot. It's fine. We're just big freelancers. Yeah.

That's pretty much what we are. Just a bunch of freelancers. I hope that's not the way it seems. No, I don't think this is.

You guys are a really good shop and really well put together. I appreciate that. So, one of the other things I wanted to talk about then too is because you've kind of had a unique experience in that you kind of came in. Most of your corporate experience has been well, or three, and then now that.

And now you and I have the advantage of kind of talking over coffee and stuff about the networking stuff. Yes. Right. And that's what we get involved with now a little more.

I think that's kind of important. How important is that to whether it's a leader or a dev or you know, what's your take on that? So, I was in network marketing for a hot second. I think I was selling travel vacations.

Super cool. It was a lot of fun and actually took advantage of that at one point. So, it was neat. And the thing that going through that experience taught me and I don't know, maybe this is a hot take for you, but.

I'm all about the spicy hot take. I get it. Everyone should absolutely try a network marketing business at one point. And whether you succeed or fail does not matter.

The point is to get exposed to how that works. And what it taught me is that you're only as strong, you're only as supported as your network is broad, you know, and learn to be a connector. You know, I have gotten a chance to know some of the best network marketers in the industry just through my relationship with someone I went to high school with. And you know, those folks, the traits that they embody, they're just connectors.

You have a need. I know a person. Yeah. And that's what they do.

And so I thought, oh my gosh. And really, you know, for myself, I'm looking to make a career transition here in the next six to nine months. And so really, I am looking to build up my network so that folks know who CBAS is. Because if I don't get out there and tell the story of who Aware3 is, you know, where Nelnet plays a role in the market, how lucky we've been through that acquisition, it's just it's all for naught.

You know, you've got to tell the story. And you know, I think that's part of it. But in terms of networking, when you need to hire someone, it's way easier to hire through your network. Oh, absolutely.

When you... You and I had a conversation a couple days ago. I called you about it over two days ago. So I mean, that's...

If you're a developer. Yeah. When you have a particularly challenging problem, or maybe you're getting into a space that is unknown, my favorite thing to do is to go ask other people how they're doing it. Lean on your network.

You can be a large language model by extension of your network. Look at the AI. We were going to get through a whole podcast without saying... Without saying it.

No, it didn't happen. Just bleep it out. It was a curse word. Actually, man.

That would be amazing. Let him guess. No. But that's so true.

And we're in hiring mode now. That's kind of a... If I can find somebody that I have, you know, two degrees of separation from, right? If I can make a phone call and go, hey, like I did with you, you know...

It's a built-in reference as well. It's like they're vouching for this person. Well, it's an unsolicited reference, which, you know, it helps tremendously. It takes the guesswork out of hiring.

And it's just so valuable to me. And, you know, even in... I would think even as an IT executive, having that network to where if you do run into a snag, or if you're really... If you come out of a management meeting and you're going, man, that just went horribly.

To have somebody you can bounce ideas off of and go, what am I doing wrong here? You know, just to have even somebody to, hey, can we do coffee? I want to run this by you because this is crazy. Well, you're right because it can be very...

Leadership is lonely at the top. You ever heard that? Oh, so true. It doesn't have to be lonely because you're not the only one at the top of an organization.

So, like, get out there. Don't believe that leadership's lonely. You need a network. You need to understand, you know, hey, the problems that I'm facing are problems that have probably already been solved.

Who can I lean on? Who can I confide in to help get me unstuck? I'll tell you one of the cool things about doing this podcast is we get to hear different perspectives. You know, they're still IT executives.

We're having to come in here, but they're all... Everybody's got their own story and everybody's got their own issues and their own ways of solving things. So, it's really kind of cool to hear the different points of view. Yeah, last thing that I thought about with regard to your network, and this might be a little controversial for some people, is that a lot of companies focus on retention.

And I do think retention is important, but there's a point at which retention becomes toxic. It becomes toxic when you are intentionally keeping someone from their potential. So if you're an engineering manager and you have someone who wants to get into DevOps, your company does not do DevOps. What do you do?

Do you keep that... Like, you keep that person. You have an open and honest conversation with them. But the be kind, not nice aspect of this is, hey, I don't think we have...

If there's not an opportunity to create a DevOps discipline, then the conversation is, hey, I'm not sure that we'll be able to create that kind of opportunity here. Is there some way that I can help you get involved in courses, train up, and then the really cool part is if you can be more thoughtful about the placement of that person with someone else you trust. So if I know a manager who's looking to hire, the best thing I can do is put that opportunity in front of the person who has the desire, the motivation, the want for that, because it will create a space for me then to connect someone from outside the organization to the new opening.

It's truly an abundant perspective on career progression. And it was something that I learned the hard way. There was a manager and I regret I have so many like, I hope she listens, because there was so much mismanagement that I did there. And I learned a lot of lessons.

And one of them was the fact that she had interests where there wasn't a path. And so I spent so much energy trying to convince her that there was a different path that she needed to go, but she didn't want to go that path. So like in that situation, what I wish I would have had is a larger network to help her work to say, oh my gosh, let me connect you to an opportunity that seems really aligned with what you want to do. And it's beneficial for the company too, because you take someone who no longer has growth potential here, you place them into a place where they can go grow.

They're going to appreciate you more later on. You're playing the long game here. And what goes around comes around. We all want engaged employees.

The lowest engaged employees are the ones with no motivation to keep going and where they don't believe a future or the path exists at the organization they work at. So you create an opening. Definitely. No, that's a hundred percent.

What goes around comes around is a thing. If you do somebody, not even a favor, if you do something that's right and help somebody out, they're going to remember that. And they're going to speak kindly of you out in the marketplace, but then also do what they can to help you in return. So you kind of reap what you sow there.

Now, this is great though, man. I think there's a lot of good information in this.