Automation vs. Agents: What Really Happens When AI Breaks Down

Speaker A: Welcome everybody. This is Ryan Staley and I am back. I have Tom Andrews. Tom is the VP of Tech and Ops at Pavilion. For those of you who don't know Pavilion, Pavilion is a private community designed for go to market leaders, focusing on basically growing in all areas of anything from enterprise, pricing, forecasting, demand, sales, you name it. Tom, welcome. Happy to have you on the show, man.

Speaker B: Thanks so much. I'm so excited to be here.

Speaker A: Uh, yeah, yeah, I'm excited to have you. And I know we've talked. I met Tom personally at, uh, multiple different events. He's also the dean of Pavilion's AI courses, which I'm going to be starting one actually coming up in the next month or so in June. So super excited about that and gotten to know him kind of throughout that process and some of the stuff he was working on was absolute fire. So I'm like, I have to share you with my audience, right? Have to have to bottle up some of Tom's goodness and then share with everyone and reason being man is like, I love what you're doing with pushing the edge of AI, automation as well as agents. And so, you know, that's what I wanted to go over with you today, man. But did I miss anything about you and uh, what you're up to or anything like that along those lines before we kind of get right into it?

Speaker B: No, I don't think so. Just for some context for everyone, Pavilion's been on this incredible growth journey that you've probably seen. And a huge part of that, part of that has been trying to bring more and more AI and automation to everything we're doing. Like, anytime there's a problem, we start from how could we build a solution with technology and do something really interesting that makes this easier to manage and easier to use? So I'm super excited to share a couple of those things with you today.

Speaker A: Excellent, ma'. Am. Well, let's get rolling with number one. Right, so walk us through the use case that we talked about in the pre show, specifically about Manas and how you kind of got to that point and what the use case is. And Tom's even going to share his screen and share some of the things that he figured out.

Speaker B: Yes, we are. So we, um, are a community of about 10,000 GTM professionals and we have people from across all different, um, levels of their career. For a lot of people trying to find their way, it can be relatively tough. And so mentoring is a huge part of what we want to, uh, make a promise to our members. We're able to deliver. So we decided maybe, uh, only six weeks ago, to start building out this mentoring. And we wanted to build something really special that actually found people who really had something to offer each other. So we set out to build this program that would be three things. Number one, expertly match made. So not just like a random mentor with a random mentee. We want an alignment on function, on skills, and, and really on what they were trying to get out of the mentoring journey, slash, what they could offer if they were a mentor. We then wanted to make sure that the entire mentoring journey was led by pavilion and that it wasn't just us putting them together at the very beginning and leaving them to it. So we're doing some really cool things with automation to keep up the pace, make sure we're checking in with both sides and getting data back to improve this, not just for the current cohort, but future cohorts. And then finally we wanted to do it in a way that felt repeatable and scalable so that we can get to a point where this runs quarterly, maybe even eventually monthly. And there's always cohorts of mentors and mentees getting started in that relationship. So the first bit was this matchmaking, which is where I had a, um, wonderful journey with Gemini and building out kind of how we were going to do it. So I started exploring what would be possible with MAKE agents. And we built out a really interesting workflow that would essentially invoke a matchmaking agent powered by Gemini. And unfortunately it didn't really come to fruition and we really struggled with the amount of things that were kind of going through the process getting really clear and salient results. Second, it was quite hard to process the output into a way I could then use in future automations. We wanted to really have a list of like mentors and mentees and then do something with it, send it out through email automation, the Slack automation I mentioned. So this is where we started, which was very much with make. This is the agent kind of set up. This is a scenario that's invoking this mentor mentoring matchmaker agent built out with a pretty good prompt. Uh, I've assigned it some skills where you can list the associates and list the executives. This started out life as saying list mentees and mentors. Turns out mentee and mentor words are very similar. Really struggled knowing a mentor is a mentor and a mentee is a mentee. So we had some, um, kind of like pollination between the two databases. People flipped positions in their match results. So then the agents invoked here and this is all to process the output. We pull out the text, use regex to find each of the matched pairs, iterate through them, pass the JSON and put that into a Google sheet. And the results were at best spurious, at worst, ludicrous. We had, I think I mentioned to you earlier, all the names got changed and, um, it didn't just hallucinate, it lost its mind. It would use mentors and mentees multiple times. It would recycle people. It would Maybe do like 30 good matches, and then every other mentee would be matched with one mentor. And it would just fail to give me reasoning because what I really wanted in the output was reasoning, why have you matched these people? So we could share that with who was matched and eventually, yeah, go for it.

Speaker A: Was this with Gemini 2.0 or 2.5? I'm just curious.

Speaker B: 2.5. I eventually tried 2.0 Flash as well, but I did experiment with a lot of the different ones. These are the ones we've got access to. So I was like, truly just going between them all, seeing what would give me the best results. Interestingly, I got slightly more stability with 2.0 than 2.5. Weird. M. It was slightly more predictable results, but still.

Speaker A: Shit. Oh. Uh.

Speaker B: So then what I decided to do was like, cut out the middleman, move away from, make, go directly into Gemini itself. And I had a really good, kind of deep session, building out and iterating on prompts. One of the things I really, really like to do with Gemini, which I've got a really good relationship with, I use it all day, every day. We work together a lot, is I say to Gemini, like, what do you need? How can I improve this? What do I need to clarify? And then I'll execute it. I'll tell it my problem with the outcome. It will make a suggestion to improve the original prompt, and then I'll start a fresh conversation with the new prompt. So I've got like, uh, a tracked list of all the different conversations and let's do another share.

Speaker A: Now we're checking out Gemini, seeing what he did here. I love that, by the way. So that was some gold right there that you just dropped it, I would say 99.5% of people did not do. When he doesn't get what he wants, he's asking the model. And this works across ChatGPT, Claude, you name it. But he's like, hey, this isn't exactly what I need. I'm trying to get to here. What else do you need from me to get there, right? Like, I'm obviously paraphrasing what you're saying, but like, I think that's fantastic. And then he gets the new prompt and then he retries it a new thread.

Speaker B: Yes. So this is the original prompt I'd been using and make threw it into Gemini, put it all in. I always add this at the end, like, please confirm your understanding. Let me know if you need clarification when you're ready, let me know and I'll provide the two data sets. Gemini confirms understanding is given. The data sets starts running through it all. We get loads of stuff and then it didn't really work. So then you can see I tried it all of these times and just started going further and further. This one I obviously just didn't even try. Let's look at this. Here's it showing its thinking at the very end. But again it's confirming.

Speaker A: How much time did you spend on this man? Up to get up to this point?

Speaker B: Like a few. Few hours.

Speaker A: Okay.

Speaker B: It was which good.

Speaker A: It's good to know man. Like most people think it happens on the first try every time. Right. So.

Speaker B: And this was like the final one. Here we go. This is one where I was really giving it some feedback. So again I'm throwing in my prompt. You can see this has changed quite a lot from the original one. It's got a lot more structure to it. Again, confirming understanding. I give it the data, it responds in Python, which is really not helpful. I'm not trying to execute Python. So then it executes it. Uh, and at the end, can you confirm each executive is only used once? It confirmed it. That was true. I double checked. Please make the CSV provided easy to copy with a code snippet. Finally it gave me this instead of a huge wall of text. Another really good suggestion for Gemini, tell it what you want and it will give you it. If it gives you an output and you're not happy, tell it. Um, here, uh, in the output, some of the executives seem to have been tried to be matched in the second pass. So these are people who are, uh, already used. We get some clarification and at the end it says therefore you can be assured that no executive is ever used more than once. That actually turned out to be true and Gemini didn't lie to me. But I still wasn't very happy. So we took all of this. I crystallized the prompt in this doc. So this is like the final version of it. You can see I've really split it out. I actually even put in horizontal rules, not super Useful for Gemini, but useful for me. And I want this to be something where I can write a guide doc of, use this prompt, give this data, uh, give it to Manus, blah, blah, blah. So you can see, please strictly and absolutely adhere ah to these two rules. Exclusivity. Each executive can only be matched to one associate, one associate to one executive. This was paramount. And then this bit is like the magic. To ensure strict singular usage, the matching process should be performed iteratively. Each associate is processed one by one. So I'm telling it, I want you to go one by one, and each time one, one by once that executive is matched, take them out of the list and never use them again. Pretend you deleted them. Um, then I give it matching criteria, I tell it the structure of the two data sets, I'm giving it these five fields, tell it the output I want, I tell it how I want it to structure this reasoning, and then essentially I tell it to do a secondary pass. So rerun this, Loosen your requirements so that anyone who's um, unmatched eventually gets matched. This is what Gemini was really, really struggling with. Gemini also struggled with some of these edge cases, which is why I wrote them out on how to solve them. And as usual, I put a confirmation block in. So I'm doing a lot of things here to try and make sure the output I'm getting is really, really good.

Speaker A: Mhm.

Speaker B: And my favorite thing about Manus is I can show you how it processed it. So please review the text. This is actually a download of the doc we just looked at. Manus reviewed it, we gave it the data. Uh, and then Manus is saying, I reviewed the instructions, here's my understanding. And it really plays it back to us. Like it, it really proves it knows what we would like it to do. We give it the two data sets and then here you can see it taking each necessary step. And this is where I find Manus really comforting in some ways. Like it's telling me each thing it's doing. I can see every step it's taking, I can see all of the data coming out. And obviously this is all just purely numerical. There's nothing in here that's private data. Uh, it's giving me everything. And here's the second pass going through with relaxed constraints. So I know it's actually doing what I told it. And then it gave me the output you can see, interestingly, it did all of its matching with the first pass. Gemini at no point managed to match all 156. The best Gemini succeeded with was 92. So, um, it was a really. I learned so much about building complex prompts and I would say I'm already quite good at writing them. But this just took me to a new level where I was trying to do this iteration. I'd love to now take all of this and go back and rebuild the agent to be able to do this more systematically. I'm still a little old school at times where, like, if at first you don't succeed with an agent, I will just brute force it through algorithms and in GPTs and AI until I get the result I need. Because it was crunch time and I just needed it to work and so I just gave up.

Speaker A: Well, so couple things to unpack here and I got something that I discovered I'll share with you that you might like based on what you just told. So check this out. So first of all, um, a love that you went through an iterative process, right? You tried make. That didn't work. You tried Gemini, that kind of worked. And then you're like, screw it, I'm going to try Manus. You put it in Manus and this did what looks like 158 rows of matching systematically or with 156.

Speaker B: It was one file with 156 and then the other file of mentors had like 104 something people. So it's trying to match those two

Speaker A: files together, which is fantastic.

Speaker B: Right?

Speaker A: And it went through and you finally got it done. How long did it take once you dumped this in Manus to get it done?

Speaker B: Then like five minutes.

Speaker A: Okay, so that's what.

Speaker B: And then we got the results. I celebrated and I was like, okay, what's up? But all the other work, although it felt like it could have gone faster, it got me to the point I had a prompt good enough for Manus to be able to just churn this out for me.

Speaker A: Okay, so here, here's what I was going to tell you about. And I think you like this. So, ah, that prompt. And it's funny because I'll tell you where this came from. I was basically. So I created a pretty long and extensive prompt on Elon Musk first principles thinking. Right, so like how to ask Elon Musk for advice on your problems and use his first principle of thinking and then it would give you like his feedback systematically, process wise, all that. So there was a lot involved with that. And there's been other examples where I did that. Well, somebody I was working with basically had a. They worked for a PE backed company and they had a really Big PE back company. I'm not going to say who it is, but they had a. Basically like a value engineer they worked with, which is part of the PE backed advisor portfolio. So I took the prompt that I spent all the time on with Elon Musk first principles, and then threw that in there and said, okay, I want you to use the principles from this prompt but make it about this instead. And dude, I was blown away. It made like a really fricking good advisor value engineer advisor prompt just from something that was architected really well that was not even related.

Speaker B: That's pretty cool.

Speaker A: So you probably. Where I'm going with this is, dude, you could probably just throw that in there. And this was in chatgpt03, so I haven't tried it across some of the other reasoning models, but you probably take that format of what you did and say, okay, now I want you to do this for X and Y. It'll transform it in that way as well.

Speaker B: That's awesome. Um, yeah, because I feel like, like as I was going through this historic process across, I mean, three or four different platforms, in the end it was interesting to see how it kept driving me towards the way I would probably write a brief for an offshore contractor to make it like incredibly clear. And I'd probably never been to that level of detail before in a prompt because normally, normally I'm just trying to get way less complicated things done very quickly. And so it's really interesting to think about, like the transferability.

Speaker A: No, it's wild, man. So test it out, Let me know what you think about. Unfortunately, we're up on time. We'll have you back for part two where you're gonna walk through make agents on ones that have worked and how you're using them, which are absolute ninja. So super excited about that. But before we bounce, where can people find you where they can find out more about what you're doing and then we'll take it from there.

Speaker B: Awesome. Yeah. Find me on LinkedIn, TomAndrews or/TomAndrews or find me in Pavilion if you're a member. DM me.

Speaker A: Excellent, man. Well, thanks for being on the show, Tom. It was awesome as always. Thanks for breaking out like the three sections from make to Gemini to Manus. Right, man. So appreciate having you on the show and we'll see you all on the next episode.

Speaker B: Thanks so much for having me.