AI Engineering

Hi folks, this is All Things Product with Petra Bille and Theresa Corts. And we're so happy you are here! Theresa In one of our last episodes We discussed that nowadays everybody could become a product builder and code more work on the actual products. But maybe not everyone wants to but rumors That I heard say You Are A Full Time Engineer now So maybe you can share a bit more about that journey with us.

Yeah, it kind of happened accidentally slowly over the past year but I would say... Well i still write a lot so maybe like sixty-forty is probably more accurate? I'd spend sixty percent on my time literally doing engineering work and probably forty percent in writing And engineering work. is that in terms of really working on a product?

Yeah, so it started with the interview coach. I've written and shared a ton about me just building my first AI product learning... and um.... It just grew from there!

I'm trying to think like how it happened.. I built My First AI Product starting March of twenty-twenty five And then I kind of just kept tinkering and playing, but nothing that was really real. Like i started playing with my own personal productivity workflows. That's why i've started nerding out on clod code But then i started talking to Vista Lee and vista lee makes opportunity solution tree software.

Their founders have been part of my community for a long time. So i've gotten to see like how they grow? Have they grown and gotten to know them really well? They were like Teresa.

you've been doing all this experimenting. Maybe we could integrate the interview coach into Vistali. And so, we did that and That was like a huge step up because I was like oh this little toy that i built for my course is now going to be in a production product! I better learn real engineering skills like error handling.

You know you already had them from your past. No it didn't...I actually did not.. I have been front end engineer but thats very different than rock solid like engineering in a production environment make it always work.

And so I literally had to learn, how do you good error handling? How do you do automated testing when as an engineer nobody was doing that wasn't even a thing. yet. um i didn't know anything about CICD process.

still barely knows something about the cicd process but im learning. uhmm i have learned security and efficiency. Claude's great at writing code, but it's not very great. It writing like Code that's gonna perform its scale That area where I've had to like really question it and be like what if we had?

Forty thousand records? What if we have four hundred thousand records? would if we add four million records? And usually it will design for the like four thousand records and I'll be like okay well that doesn't cut it anyway.

So started with an interview coach Talked to this company. I created this partnership and then I immediately like scared myself as like. am i even capable of doing This? And I learned a lot, and it turned out.

the interview coach was integrated It was fine. and Then we started talking about. I started running experiments About could I do AI generated interview snapshots? can't I do ai-generated opportunity solution trees in In the mean that Was Like for Vista lead?

and then in The meantime from my courses I started experiment with what else Could I add To My courses And I built a business fundamentals coach for our new course, Business Fundamentals. would love to work on? Oh, I'm not running out of ideas. So i'm building on two paths.

so there's the Vistali path...so I now have a partnership with Vistaly. they basically license AI services from me right now. those services are interview snapshot generation and opportunity solution tree generation.

but we're also working on an AI interviewer We're working on-I can't share the longer term plans Right behind. that is a really big idea, but I'm very excited about. But i'll be able to share eventually. and That's like the Vista Lee path.

And what? I love About The Vista lee Path Is um this Deli? Everybody Keeps Asking me Like Why don't you Really Skills To Help People do This? Well I Love Claude Code and I love the Personal Productivity Part of Claude code but Claude Codes isn't multiplayer.

so yeah I believe Discovery as A Team Sport and You Need a tool that allows everybody on your team to add their interview notes and the thoughts of the snapshot, see this snapshot. To add comments for the tree or contribute it has to be in a collaborative interface which is what Vistally builds. The reason why I have my partnership with Vistaly is because i don't want build all that stuff. so our relationship almost set up like an AI researcher I figure out what the AI can do.

You're the innovation lab to some extent, so that again? You are the Innovation Lab Exactly and then license it to them. And that looks really great because i don't have to worry about SOC-II compliance. they already Have That ,i dont' have to Worry About like Data Compliance with GDPR in EU.

They Already Have That. I just get To Like Play With What's Possible Build The AI Service Around It Then They Deploy In Their Environment That Got All The Compliance Going On So That Works Really Well. Then I have this second thread going where, I had this really grand vision. So historically i've run a training company over the last...

I talked in previous episode about how it cut five of our seven programs and basically im trying to cannibalize myself. so..i think the future is you'll have an AI agent that will give just-in time training as needed on the job. And so I'm building basically a Theresa bot that has all of the skills it needs to be your discovery coach.

But one is not only based on a forty minute podcast episode? No, no! Not one says chat GPT act like Teresa One That Has access To literally everything i've ever written Everything i have ever recorded. That's its foundation.

but thats not enough. Thats great that going Google and searching what as Teresa said about this The reason why I built an Outcome Coach and a Business Fundamentals coach, and interview coach. And eventually there'll be an Assumption Testing Coach and Opportunity Mapping Coach right? the reason Why i'm building all of these.

These are going to Be tools for Teresa Bot. So that when you go To Theresa bot and You say blah blah blah i'm working on this thing i have This hard problem it knows Which coached to invoke to then coach you. i love It. so this is A whole different.

so i Have like two paths and I'm working on them in parallel. But it's all engineering, like its really AI engineering. It is learning context engineering, prompt writing orchestration evals observability. Do you find good enough resources for easy to learn these new skills?

Well, I started a podcast called just now possible. And it's like my cheat code because I hear a team talk about how they solved the hard problem. Yeah and in my brain am like oh i can apply that to my work this way. so That's The first thing is?

I traded Like the best cheat code if I found A Way To interview some really good teams About How They're Building AI. yeah um what's fun about AI engineering right Now Is that It's like the nineties with the web, everybody is learning it together. There are a lot of people blogging and there're a lot YouTube videos as well. A lot of podcasts.

Slop! And a lot slopp? So you got to pick your sources for sure but honestly that way I'm learning by trying do it. then I am letting Claude teach me.

so i'll give an example...I built foundation for Theresa Botte maybe two months ago. I had never done anything with RAG, which is Retrieval Augmented Generation. Had Never Done Anything With An Embeddings Database!

I Had No Idea Like What's The Right Way To Give My Agent Access to All Of My Content? I Knew Those Things Existed...I've Been Talking To Teams About Those Things Through Just Now Possible..I've Learned Like What'S Hard About Embedding How Do You Have To Do It?

Well? So like i wasn't a total beginner but I had never done it myself. And so what I did was, just started a conversation with Claude and actually published this conversation on product talk. So if you go to producttalk.

org i have an article about vibe coding best practices. in This example of how I built Teresa bot the full transcript Of my back-and-forth With Claude is in that blog post. Oh My God! Um...

And I literally Just started like here's An idea That I Have Help me think About How can make this real? and we just went in back-and-forth, designed an architecture for it. Designed the first mini experiment which is let's put twenty blog posts on it to see how good this search was And I built a version of Theresa Bot that is basically just an embedding database with a rag step In two days! And then launched inside my Slack community and got real people testing within like forty eight hours.

Just quickly folks, can you hear how Teresa is still following best practice approaches and still having conversations with real users? We are not skipping that step. Not even when we're developing AI products. I just wanted to get everyone out.

Yeah it's actually. um You know last summer i took Hamel and Shreya class AI emails And Was barely like I was a total novice on AI products, but you know what? I love about that class. Actually think Hamel and Shreya Think About AI Products the same way.

I think about discovery. they've done A really good job of identifying What are those basic skills or Basic habits That Are Universal And then how You Apply Them Changes A lot Of Different Ways. so i feel Like I got a really good foundation. The other thing I learned from that class, and this is because i have gotten to know hamel And we've had a lot of side-camp conversations I think That I am a data scientist at heart and just didn't know it.

so like maybe if I think about discovery Like what makes your discovery well? Is diving deep like What make sure discovery good? is diving deep in the messy qualitative Data and diving deep In the quantitative data and figuring out like where's the signal in this noise? And if you think about AI products, people that just write prompts and release something don't realize this yet.

But if you want your AI product to be good, You have to log traces. Yes! Look at the data. That's Hamel's mantra...

and really dig in. not like tweak your prompt a hundred times but understand what is actually going wrong here. So I'll give an example of this. Yeah exactly to test it.

even before I put in front of users, generated like a hundred questions. I thought users would ask and they actually came from my Slack community. so there were one-hundred real question. people have asked me the past.

then ran Teresa bot against those questions looked at what are errors that made systematically identified their errors use that iterate on the prompt change orchestration changed context And this process to me is so analogous to good synthesis and discovery. They're like, yeah I'm pretty much a most of the time AI engineer right now but i don't feel Like The work Im doing Is different from the Work I've done in the past In the Past! I was working with interview transcripts and behavioral analytics and support tickets and sales conversations...

and Now im Working With Traces Like AI traces, but I feel like the process is exactly the same. And so maybe what i didn't know was that product managers if you want to be good at discovery we need to build some basic data science skills and it's been kind of fun to uncover oh! Maybe always as a data scientist? Yeah yeah...

The moment when I learned more about data science all my discovery work became way easier. innovation was really happening because I had, yeah better equipped to surround myself with all the data in forms. that really helped me see whatever we need for other users. So it actually made a difference here.

so i agree more of that. I want come back and say something you said earlier But you've been an engineer before. And so I want to talk about this because...I think some people are using it as like, yeah but Teresa is different and can't do that.

So then i wanna talk how was the engineering before? Yeah please! So I have a undergraduate degree from Stanford. Stanford has best in class computer science program.

I did take computer sciences classes at Stanford. That alone makes your difference. But here's what I took two beginner classes. Right?

I didn't take four years of computer science, i took two beginning coding classes and the rest of the computer science classes that were all the math theory side of computer engineering skills because I wasn't a computer science major, i was a symbolic systems major. So the goal of the classes that I had to take were to introduce me To this perspective on the same idea as the rest of the program covered. so like I didn't come out As a college student an expert in engineering what?

I came out Of College Jazz Was A Designer Because My Focus Was On Human Computer Interaction. Okay, I had a little bit of technical Skills But I Mostly Came Out As A Designer. The problem is, I graduated in nineteen ninety-nine. There were very few companies hiring designers.

what they were hiring was front end engineers and i had learned on my own time not at Stanford On My Own Time because the internet Was A New Thing. I Had Learned HTML. CSS Was Starting To Become a Thing? Um, the first company that I worked at had its own proprietary template language.

Which by the way, The First three companies I worked out have this because This was before front-end frameworks like we had to roll our own. and so what when i say? Was an engineer? i only did Front end template language coding html And css.

i never touched a database and deployment. but back then it was Like taking your file uftp And then you put the file on. Yeah, yeah. so basically taking things life process.

I've never worked in a front-end framework i've never touched a database. i've Never ran at sql query Like people think like oh it's because tress has been an engineer like. I can't explain to You how much of these engineering skills? I did not have that.

That's why I was asking So How Much Of The Skills That You Already Had? How Many Of The Things Are Familiar? Because my kind of engineering background is slightly more engineering background maybe, because I've really studied. It was still very front-end heavy what we did back then.

PHP was around. so that's why use a lot and we developed... We do the bit of compiler building in stuff like this. So i looked into some back-end stuff, basically.

But there was ages ago and so many things have changed ever since. sometimes it helps when AI does stuff these days that I am like. hmm maybe i should ask if Claude has done this particular step or... So it helps me with debugging some times but its not a transferable skill.

Here's what I will share. I'm not going to undervalue. when i got from my Stanford computer science classes, because here is where did learn even in the very beginner class they instill form day one this concept of elegant code. it all about how do you take a hard problem decompose create code that maintainable reusable and so like.

I do think i have engineering skills that i learned from my undergraduate degree. Like, and i learn this for multiple perspectives. so i didn't just get that from my computer science classes...i also had to take philosophy classes where i learned how to deconstruct an argument in a very logical way right?

And so..I'm not poo-pooing my education! I actually think the reason why can learn these skills is because of that foundation. But I just want to be really clear.

Like, I didn't work...I had a lot of gaps and the reason why was able to do what i've been doing is because Claude can fill those gaps. Yeah you never stop learning! I think this is for both us too.

we never stopped learning through our entire career. We always whenever feel slightly comfortable we go look for next project or yeah thing that we want to learn or get better at. And, We're quite good in learning new things and picking up new skills. Yeah!

I think that is what a super helpful skill right now. So whoever's listening practice to get better At something...I think this helps you To learn something quicker and not being kind of thrown off by Something coming out like AI. there's a couple things I think that like it's learning by doing, not learn by reading.

I read a lot! Like don't get me wrong i read A LOT. but what I love about LLMs now is you can just be like...I want to do this thing..

i have no idea how set me on the right path and then You Get Started And Then You Tackle Another Thing You Don't Know How To Do And You Can Just Do That. And so I want to tie this back to our last episode. We talked about the myth of product builders and not everybody has To be a product builder, i'm not sharing my experience because I think Everybody needs to become an AI engineer? I actually wanted to share This because I've been on this path where After I broke My ankle I decided I was just gonna do whatever it Was of most interest to me and I'd Been pulling this thread.

And it turns out, the more I pull a thread. The more of an AI engineer i become and that's a little bit surprising to me like...I love product! I loved design!

Loved being a coach!! I loves teaching!!! And I'm just finding myself wanting To be an AI Engineer all day every day. It was literally just finding Like not Being afraid Of what?

I don't know. and Just following the thing That is fun for Me. Yeah, where passion led you?? Yeah..

and so its Not the take away isn't? oh everybody should Be A AI Engineer. I think the takeaway Is One of the really powerful things in this new technology. is it helps us learn things we don't know how to do.

It helps us create things that you don't have a career, and help build skills or areas which I wish always had. Yeah! Um...I ALWAYS HAD ENGINEERING INSECURITIES TO WORK AS A FRIEND AND ENGINERED AND KNOW NOTHING ABOUT DATABASES.

LIKE EVERY JOB I HAD I THOUGHT I WAS GONNA BE DISCOVERED AS A PHONY. LIKE I ALWAYZ HAD THE SEEN LIMITING BELIEFS ABOUT ENGENEERING AND IT TURNS OUT. I DON'T ANYMORE Like I know anything that I don't know how to do, Claude will teach me how to and Claude is infinitely patient. And I have no problem shamelessly asking a million questions."

And like i said for Teresa Bot ,I put the whole conversation on the web. so if you want to see what things I had to ask about...like..i know what embeddings are but idk how an embedding database works.

It's fine to not know those things and you can still push forward. So I think like whatever it is, You're excited about? It doesn't have to be AI engineering. Think of a takeaway as you could just keep pulling on this thread And for me that super fun and you might not be surprised by where you land.

Thank you so much Teresa. amazing thanks for sharing your learnings. Thanks Petra.