TDM S3 Ep6 - Deepak Prasad: Catching the Data Wave Before It Breaks

[Music] your regular fix are the best guests from the world of data and analytics this is the data mix with brian bdden and george peter [Music] good afternoon mate how are you i am good brian how are you i'm very well it is tuesday the 8th of april at 3 p.m gmtand we were here with another episode of the datamix i've actually lost count but i think we're at number six i think that's where we are um but yeah it's exciting to be back man how are you i'm good thank you um i'm just uh catching some rays in front of my swimming pool here yeah i i did notice that i did notice that bit of a different vibe over this side uh like school holidays started two days ago so um and just to give you a bit of my bit of a look into my domestic life behind me over this wayi've got a window and the blind stopped working this morning and the desk is in the way and i had to work out a way to unpick the blind and throw it off so it's like my window looks like a right old state at the moment but um i can't see that i'm i'm in a good place i'm in a good place yeah i mean for the bdi people just in case i get accused of lying this is a background i'm actually in our london office right now uh but it's a very boring white board behind me so can confirm as i have seen thebackground recently in the um before the taping took place so yes george is where he is supposed to be well done mate commitment to the cause commitment to the cause and we talk about commitment to the cause right um we have a great guest coming on um today as well we've got the caption up there we've got deepak prasad coming on but just before we do that we have a few small things that we always need to cover so let's get stuck in um your support means a lot to us guys um we appreciate you turning up and weappreciate it even more when you drop us a we review because it helps push us up the rankings um and we've got a lot of shows coming up as we move towards some bigger events in the summer so if you could drop us a review using the qr um spotify and apple are the really big ones for us if you could drop a review on there that would be wonderful it makes a big difference um that was that was short and sweet but i think it's to the point as well so george i think it's i think it's probably time probably time for the datafix right let's do it let's get you into it all right jingle away let's go all right so it's been a while since we've been on actually so a lot of things have been happening in the world but mate what are you choosing to focus on this week for the data fest well there was lots lots we could have uh we could have done actually there's been a couple of um ipos in the in the world of technology recently um and a couple of failed ipos as well that didn't go as they were expecting to go so they pulledback um but it all all of the uh sort of technology news at the moment leads to one source and that is the uh the tariffs that are going on within the the usa so for anybody that's been living under a rock for the last week or so the usa has imposed blanket tariffs on pretty much the rest of the world now we don't talk about politics here on the data mix or the data fix but what uh we are interested in is the impact on um on technology and one of them um has been tesla so tesla was down 10% on friday umand we like to think of tesla as a technology stock um i think um nvidia i personally have nvidia stock so i was uh upset to see that uh that's that's actually lost nearly 400 billion from its uh market cap um over the last few days so let's hope that turns around because i haven't sold yet you're not a loser until you sell apparently um but i think one of the um the the biggest uh the company that stands to lose the most at the moment is is apple uh and part of the reason for that is that apple has umuh huge supply chains across the world but one of the biggest supply chains comes from china um and as we uh have been hearing china is um they're already being hit with i think 37% tariffs and um donald trump yesterday said that um if uh china reciprocate which they have done then donald trump is going to add 50% to that uh so um i did hear stories of plane loads of iphones coming quickly from both india and uh um and china over the last couple of days to try that sounds plausible that sounds plausibleyes but uh we'll see so yeah that's uh that's the data fix any comment on that brian if you've been feeling any heat in the technology world because of tariffs so it's interesting um because we talk about tariffs on products right and we talk about it on imports um but i've had a chat with a few people about how does this affect like services as well and i'm assuming at the moment um if you're delivering services you're okay but um i think there's been a bit of tension inthe uk especially from kier star armor about how we are in now entering uncharted territory in terms of global economy um never before have we really gotten to the point where we're all charging each other tariffs for everything and effectively you know prorating everything that is happening um in the world in terms of product um where is all that money going to come from that how are people going to and businesses going to buy it's fine to escalate everything by 30% but where is all this extra capital going to comefrom um and i think again without getting political about it i think the the shakiness of the stock market and the footsie um and the nasdaq and all of these are real serious signs that something is happening here um and we'll just have to see if uh the person at the center of it all decides to um soften their soften their tone but one would suggest that might not happen so mind yeah it's interesting it's interesting but we we we watch with baited breath and we see what happens um but it's the only thingwe could really go with this week isn't it so yeah i think so we'll see who squeals first because i don't think anybody wants to put their prices up yet based on this um and i think mr trump will come down quite hard on any us company putting their prices up but somebody's going to have to somebody will cuz out of desperation they either put their prices up or die so let's okay well to to be continued to be continued um and that was the data fix for this week so thank you very much for that mate we appreciate it it's umsometimes things affect data and the data that's behind these decisions you could argue is there data behind these decisions and i think that's the interesting anti- angle of what we're talking about here as well so we shall see we shall see so that covered the data fix for this week but um on to the main event mate we've got um we've got a really good guest today actually we did the recording a couple of weeks ago um and it's deepak prasad who's um like a very very experienced um consultant and um aistrategist that's been around a lot of different technologies and george i will leave you to tell the little story about what happened at the start of this recording just to let everyone know before we get yeah i'm trying to remember which uh which country was i dashing into because um i i came flying into that what about i missed the first 15 minutes yeah of the of the session um but but got in um still sweating um and said hi to dpack and we and we got going it just goes to show that in this modern world um of uh of technology youcan you can run these um you can run these podcasts almost anywhere you still have to be prepared you still have to know what you're doing but you can uh you can take your laptop anywhere well there we go there we go well look without um without further ado we'll get we'll get jumped into the recording it'll be interesting because you'll just see my face for a little while until george decides to appear but it'll keep you on edge to see when george is going to appear throughout the episode so umdeepak prasad everyone we shall see you in about oh i would say about 36 minutes all right to wrap this up speak to you very soon guys cheers well oh and obviously keep an eye out for the comments and stuff as well we're watching the chat and we'll pop any comments and stuff as they come in all right we shall speak soon hey everyone welcome to another episode of the data mix in season 3 and as i mentioned before we've got deepak prasad with us today how are you mate hey good how are you and you are from edinburgright from london in edinburgh yeah right right beside the airport so if only it were physically in my budget to come and fly over and and see you and do it in person i'd be like set for it but no um sadly not um but i think we're like in way different time zones so i think you're 11 hours away from me right it's first thing in my day and last thing in your day so yeah yeah all good almost 8:30 thanks thanks for doing this podcast with me oh no problem man no problem we're delighted to have you hereand i say we uh normally by we i mean me and george but george will be arriving imminently he's just in in transit right now but um we're going to get started i think mate and we're very excited to have you and you've had a very um very decorated path through the world of data and ai so why like for for our um you know listeners and viewers who don't know who you are why don't you give us a bit of an intro and let us know like some of the tech stacks you've been working with throughout your careercould name deepak prasad i've been consulting since 2008 um so since being uh in 2008 uh i i was in when the moment i joined it the first project i took was in unix and oracle so it's so since you say that i my path is across multiple technologies this that's because i started from black screen and now i am in a very colorful screen so i started with unix oracle informatica cognos bivo micro strategy g and then click tableau powerbi then snowflake gcp and like ths spot the list goes on but if you ask memy total career right that lifespan of the span of around 15 years if you ask me the two highlights that i always remember first time i ran something with a bash script and it got executed in a black screen and second thing is when i have been given a demo of in-memory analytics which has i mean current at that time in 2011 powerbi wasn't that famous it was click and tableau um that that's my second high and the third high was when i saw snowflake uh first in 2018ish um what they have done what'swhat's what's your career path like well well look it's it's a pretty cool set of highlights and i just have to say that um a black screen is a terminology that only people of a certain age will understand right but uh what we mean by that is command lines and terminals being able to run things um by actually typing and not just speaking to our ais which is the way that we most people seem to um to to work that that's a bit harsh not most people work that way but it's becoming more important to be ableto work that way right so i think that's the first thing kudos for the reference to the black screen um in memory analytics well you don't need to talk to me about that because you know we're on the same path there and we we understand that i i don't have hicks book behind me for nothing so it's um that that's a like a path that a lot of people have followed and obviously you know the the landing of the data warehouse and the online data warehouse in snowflake and um sources and targets uh nowadays arebecoming much more prevalent in data pipelines as well and a few episodes ago we had um we had ben rojan who is like pretty much at the very very top of that that that data pipeline chain of making all that happen so that i mean there's a lot to unpack there but i think you know for me i've sort of become a specialist and it's quite interesting for me that you started off as a kind of let's try all of these things or maybe your career dictated that and now i guess you probably settled on a few things ratherthan everything which is where your career started so like what are the things that you settled on now and that you like doing now that's what the consulting is right you you can't say no often so you they would ask you can you do a prototype in this technology and if and and they come to you knowing that it's a new technology so um that's why they're asking for a prototype to see the feasibility how it's going to work end to end and at that time you can't say no i don't have prior experience no it'snew for me it's new that's why they're coming to you you have to do it right so i mean i think once or twice if you got through that initial cold war with the new tech i think then you will understand that maximum what's going to be there inside you will understand the total blueprint of uh it and the infrastructure the storage and compute and whatnot you understand that this is what's going to be inside maybe in a different order or in a different stack but let's go and see whatever it is so ithink that mindset would come after after a while i i think i think mindset is really important to talk about right especially from a consultancy perspective because this is controversial but i think tech doesn't matter and i don't mean that tech is not important but i mean when you become a consultant your brain has to work differently from when you're an employee because when you're an employee you have a set amount of things that you need to know about but when you're a consultant you need to know about like boatloads ofstuff that you've never used before and just to give you the example on that on a practical basis you know when we set up the podcast i knew nothing about youtube i knew nothing about like wavepad i knew nothing about how you published buzz sprout and how you published a podcast and when i started my newsletter i had no idea about beehive and i know had no idea about ads and i had no idea how to plug all these things together but as a consultant at some point it clicks and you just that's not a pun by the way it just make startsto make sense the way that you think um and the way that you start to connect the dots so like how did how did you like at what point did that happen for you where you stopped treating things as a piece of software or a tool to work with and just think this is another cog that i have to have in my wheel to make things happen when did that happen for so i think i think the moment i understood that we can solve everything is because i understood what is the core ingredient for example let's say we are recording this right u we are recordingthis because you can hear me i can see you and everyone is going to see uh every one of us so if but if you go crux it down crux it down and come to the last point what we really have is light sound and internet which is network right so that's that's all the whole podcast is but if you apply the same thing to computer software there are three layers application layer system layer which is the os layer and network so if you take any technology any tech stack these three components so if you know how they interact with each otheryou can go till the bios level you can go till the os level and if you go till the network level i think the one area which i haven't explored much is network layer but application layer and system layer i think pretty much pretty much confident yeah and look i i think that in in the spirit of becoming a consultant it's a it's a decision that you have to make whether you're going to niche down and that's either on you know you could say you're just niching down on one of those layers you could say that you're nichingdown on a specific software or a specific tech or you could say that you're niching down on like working out software that's required for an industry so there's lots of ways that you can do that and i think that a lot of people like don't don't know like how how to make that decision and i think that's an important thing and one i think one of the things that's interesting for you is that the decision was made for you by like working through the tech stacks until you find i guess you find thingsthat you like and find things that you want to spend more time on is that kind of how it happened or were you kind of like as a consultant sort of like i have to take this work and i have to i will tell you why why i so c for consultant c for challenge right so that's how i see it right and then the first six years of being in consulting what uh happened to me was so you got a neck of it you understood you spin up a vm install a software test your workflow it's done and then you scale it up for you do itfor thousand rows then you will scale it up for million rows then you will see whether it goes for billion rows and do the feasibility study and it all goes on so what i realized after six years is um if you want if you are when you are catching the wave is very important all right so if i am talking about cognos and micro strategy in 2025 i i can be in it i can be the best person in that what are all the amount of opportunities that is available to me and how valuable i am to the company and what am i going to doing to the companylike there used to be a fantastic um piece of software called hyperion which is used for planning i'm not sure whether you heard it i used to work for ibs so i know hyperion quite well yeah yeah so the reason why i remember few of the technologies like hyperion and abinishio is because when i used to work in cognos and micro strategy there used to be set of consultants who was sitting with me they would be like flying from somewhere from us uk and um even from india northern part of it and then they say they are specialized consultants andthey do only hyperion and abinio and things like that i think that gave me a high i think if if you are in the edge if you are catching things a bit early than others then you are good you are good to go you can ride the wave i think you will pick it up once but without that six years i wouldn't have learned it so 2008 to 2014 was all about how can i so am i enjoying the challenges that's the first question i answered myself yes i do then how can i be making sure whenever i take a new challenge thatgives me a free ride for the future right so i think someone at when i was like 6 to 8 years um they gave me one important tip i think that's the tip that i give it to everyone take take one one uh text stack or take one word let's say data for example and let you ask now we have charges many gp it is um you can ask give me top termin 100 terminologies in data and if it gives you the 100 terminologies in data how much you know and that would exactly tell you where you stand and how much you don't know these are all theareas that should give you enough curiosity to conquer right i think that tip in 20145 helped me really a lot so whenever i start i always start with a blueprint okay this is the blueprint i'm starting starting with and i don't get deviated because i have a blueprint i have done enough work for the blueprint and until i cover it i think i don't go around even nowadays if i take up a udmi course or linkedin i finish it and then i go to the next one so i respect my own blueprint i might have missed here andthere but not completely right i think these are all the learnings that you take away catching the wave making sure you cover the blueprint and move on i i i like that i i like that catching the wave um i think it's very important in tech and you know one of the most important things in tech is that george has arrived in the background so hopefully he's ready i'm going to i'm going to like switch him on whether he's ready for it or not and we'll oh there he is hello george good morning or deepgood afternoon to you or is it even good evening good evening almost and thanks george i know you caught two trains and walked a bit to reach the office and finally made to the podcast and we had some hiccups too and i think everything helped together the the only the only thing that george did not do deepac was catch a wave which is what we were just talking about when you arrived to catch a technology wave so that's the discussion that we're in the middle of right now mate so so yeah i mean so i guess that's a big thing foryou it's like finding something a line in the sand following it through and picking that as your stack right that's something that served you well throughout your career yeah so if you take if you take the wave right so 2000 till 2008 it is all operational reporting sap bo templates ad hoc report no ad hoc it is all bespoke reporting and then came the in-memory analytics that gave way to ad hoc reporting and then in-memory analytics happened and then now the same thing that happened to bi layer is happening to the data layerwhere cloud came to the evolution and said that you can scale and you can scale to the level you wanted because uh the main thing that snowflake did very well although apache iceberg is the reference point for them they the moment they uh decoupled the storage and compute one can scale to the level that others doesn't have to scale so compute can scale to any level and see what happened at that point in time uh i think the wave that you catch is very important let's say that i would have taken some other technology other thaninmemory analytics i think i wouldn't have caught the wave and i wouldn't have gone through all the technologies that came on the way mhm so that's why i say catching that wave is critical yeah so so we've just been george we've just been talking about deepak's career path and um we we started by saying like he's probably dealt with if you think about like the like the hundred core texts that have happened over the last sort of 10 years right from a data stack perspective like dag's pretty sure he'sgot most of it covered like and i say most of it like 60 to 80% so and we also talked a little bit about black black screen programming and command lines and stuff so let's hear your perspective george how much have you been through with that that level of stuff have you had any like have you had any of the command lines or the bashes or the visual basics to to deal with earlier in your i think you would be starting from main frame that cuts deep oh you're can't hear you we still cannot hear you ah we couldhear you a minute ago but um this is too cool this is too cool we're just gonna i think we're just going to turn this episode into watching george now can you hear me now yes yes perfect okay so i work in it but i don't know how to set up the audio on my computer so so i was able to hear you perfectly going right right back to the start um uh i i am am i ashamed to say it i don't think i am but my first programming was with microsoft my first database programming was with microsoft access uhso that's uh and i was able to connect that to a website and that's where i got the excitement of um dynamic web pages really so to make a web page dynamic back in the day and we're going back 20 years you needed a database at the back end um now i have a twin brother my twin brother started with flat files um so literally text files and using text files and append and update of text files um to do this i remember talking to him about it um and uh it sounded exciting but difficult and i've seen theentire wave um i got properly excited when uh clickview 8 came along um i got involved with that that then saw me going further and further upstream to no downstream i always mix up down to the actual database level um started playing around with um uh um mysql back in the day so that was um mysql is still around but i think it's now owned by oracle maria db is the is the equivalent um uh and then that got me into sort of more and more um of the mainstream uh enterprise things like oracle um and microsoft sql and here iam now i don't really get my hands dirty anymore i just talk about it and probably deep packets and just important that like we we always bring on people that know more than us and i mean that in the with the greatest of respect because we are not as technical as we used to be and also our audience loves hearing about things that we don't know about they like revel in the fact that we don't know what we're talking about at times so but i i had a question for you like in terms of interesting challenges nowadays and we talk abouthow you get deeper into that stack and the more technical that it becomes right but we're kind of at a we're kind of at a crossover moment right now i think with ai and that having detailed knowledge about the depth of a stack is to a certain extent slightly less important than it was a few years ago because if you could and we talked about this at the start of the show right if you can get deepseeek running or you can run deep research and here's another thing that just happened like literally in the last few days deep research wasmade available to paid users on chat gpt that means that you can just say you know i want to know be a command line expert i want you to write this script that that fires up a database with d6 tables has these seven relationships and data engineering is changing as we know it so how do you how do you feel about that having gone being like a a black screen og if you like how do you feel about that see i think i think it's an enabler that's how i think so i i the reason why i felt it's an enabler islet's take something like 201617 right at that time i wrote a metadata driven framework what it does is um we have set of tables and today we have 50 tables we have to generate 50 qvds tomorrow we need 100 if you have 100 tables i can generate 100 qvds or table data extract or csvs right so if you see the repetitive patterns in this there are lot of repetitive work you have to write 100 extractors today tomorrow you have to write 100 more and then it goes on how about we have only one extractor but it is driven by aconfig file and it it does iterative actions and create the outputs right and you can have variety of parameters to control it whether you want all the columns whether you want to check for duplicates whether you wanted to alias some columns you can do all that with a configuration file and to do that i think it took uh two and a half to 3 months end to end okay now if i take any gpts any open ai tools and type a exact requirement i think i will get the query in maybe 20 seconds and then i have to change the maybe it would give me arigid query i have to variableize it parameterize it put it as a function pack it up and change to coding standards of my company and principles that my developers can understand at commence maybe one day three day to test and deploy i think i received uh accolades and appreciations and almost an award as well i think now if i do it today i think the first thing people would ask is is it done by you in one day or something else right i i i get it i've got one more followup and then i'll let i'll let george jump in but i thinkthe like the base of a data engineer or a data consultant nowadays is very different from the starting point that we had 10 or 20 years ago and not bringing no i don't want to bring up the past too much right but you know probably all of us on this call cut our teeth like in front of a terminal like working out command line stuff to a certain extent we didn't even really have the internet to help us through some of that journey at least or the amount of documentation that was freely available just didn't just wasn't outthere and nowadays it does make me like maybe not reminisce but it does make me wonder about whether the same mold of data like data consultant will grow in that way because if you've got the choice of it's going to take you six months to do your dissertation uh by doing it by hand or you can pay $200 a month to get deep research to give you 50% of it before you even start how many people ethically are going to pick the right thing and you know i know that i know there's lots of things for plagiarism and that kind ofthing but i think it's a super interesting dilemma because like you say we're almost in the land of infinite compute now so asking a very very complex question like can you write your dissertation for me is something that is possible to be perfectly honest so like how do you how do you feel about that how do you see any of that the shape of data engineering changing with the with the role of ai nowadays see i think uh judge very were you about to say something no i'm i'm listening but i'm very interested in this topicokay so let's let's talk let's talk the journey of data engineering right so why data engineering came to the rescue okay let's start with database it was all referential integrity problem and then they solved it with primary key and foreign key and they wanted some history they switched on changes and there are some history table and they have a current table they solved everything and then they understood that they still need to take the historical snapshots of the data and they want to do sed typetwo and do the current um uh current uh record marking and whatn not right and then what happened was this uh infformatica and everyone it when some one person is working on a workflow the other person was sitting idle so same persons together couldn't do the co-development and they couldn't ci/cd their pipeline and but there were some advantages in the informatica pipeline it could parallelize your data you can do the component parallelism you can do many parallelisms that you can achieve but what it didn't give the high is wecouldn't shift the software engineering principles to data warehouse i think that's where analytical engineering came into the rescue and people said that you know what you write your own sql when you want to build a pipeline step one will be done by deepak step two will be done by brian step three would be done by george and whenever they make some changes in their own sql it would if you wanted to affect the whole pipeline it would affect if you want it to affect only certain branches certain certainmerges that would do i think that's what dbt did it very in a hassle-free manner i think db dbt that's what they are they are growing very fast and then and then what happened is the cloud cloud equation was just before dbt and then cloud engineering came in where infrastructure as a service platform as a service software as a service came in and then they have to spin something which says bringing software engineering to the data and we have sqls and then we wanted to spin up uh serverless things just so whenever i need it so then thenbecame an amalgamation where data warehouse data engineering and analytics engineering boom it's all born together and then you can write your python there were notebooks earlier azure notebooks data brick notebooks i think data bricks can win a big accelade there saying that they brought the notebook to the production grade notebook was there i mean i think um i think it's called uh suddenly i forgot the notebook name where everyone the orange color notebook if you know what i'm saying but i forgotgot the company name jupiter sorry jupyter notebook before yeah so jupyter notebooks and then people used to write r python and then they used to scale it but uh they couldn't achieve the production grade uh pipeline so what happens is the amalgamation of data engineering analytical engineering and data warehouse together is what where we live and anything can happen in this um if you hire a good software engineer and ask him to do build a proper ci/cd pipeline and use a sql guys to do the sql work and dbt did certain things verywell for example the macros where if you have a fantastic sql developer and he if he can write one sql that can help you lot of tlops reduced lot of transaction seconds reduced so let him write all the macros so for date conversion time conversion uh date time days calculation seconds calculation he will do all the macros let other people just call the function in their pipeline mhm and the ci/cd so when i write a code and it runs for 3 seconds and judge writes it go for which goes for microsconds 300 microconds and then there would be achange in the function in the function library but people would be still calling the same function and look at if i can i recently heard uh someone who's working for a sports company uh that gave me so high um the whole pipeline they developed in python and they ran it for 90 minutes it used to track where the football uh player stand how the ball gets passed from person from x and y of one location and x and y of another location and how many passes were there and when the distance was less were they able to pick up when the distance wasmore they were able to pick up so very complicated pipeline if you think it and visualize it right it used to run for 90 minutes okay so like they took the same pipeline rewrote the code for rust and they developed it and can you imagine how much time it took to run oh what do you think john 90 minutes 90 minutes to one second 65 seconds 65 seconds 65 seconds so i was blown away and then i asked him whether you learned rust for this one no he said that there is a a library now called as polar rs which does the python to rustconversion in the compilation so they don't need to learn a new technology new text stack so the library when you import the library polars and then they do the same three i think it was 3.

5 million data points mhm where where we were before and where we are now see just imagine the same time savings and put it and scale it across the universe you talked about infinite compute and we talked about sam alman asking $1 trillion to the government to run open ai i think it will all get reduced yeah so your hypothesis then is thatthis sounds more like um augmented intelligence so it needs it still needs a human to almost orchestrate um and was it a human that said "okay rewrite this in rust."