Data Engineering: The Backbone of Modern Manufacturing with Samar Khan

Speaker A: Foreign.

Speaker B: M welcome back to Industry X Stream where we navigate the currents of innovation shaping the future of industry. In today's episode, we are venturing into the data driven heart of modern manufacturing with our special guest, Samar Khan. With a distinguished career that spans Google, Microsoft and now a leading role in manufacturing powerhouse, Samar brings a wealth of knowledge on how data engineering is revolutionizing the way we produce, innovate, excel in the manufacturing realm.

Speaker A: Hi Musarat, how are you?

Speaker B: Yeah, I'm doing good, how are you?

Speaker A: Yeah, I'm also fine and I want to like, thank you for inviting me to this podcast. Yeah. And uh, yeah like I think it's a great way to like share the knowledge and yeah, I'm grateful for getting invited here.

Speaker B: Yeah, pleasure is all mine. You are helping me out. Uh, we are trying to educate the world about Industry 4.0 and today we are going to have a conversation with you about data engineering and how it shapes the manufacturing industry. So can you share with us your journey into data engineering and what drew you to specialize in this field?

Speaker A: Okay, yeah. So basically before going into data engineering I would just like to provide a little introduction. So myself, Samar Khan, I hold over eight years of experience in IT industry and uh, I started my career in 2016 as a software engineer working for bank of America. And uh, if I talk about data engineering and how I got into. So initially I was working as a software engineer where I mostly, as I was working in bank, my task was mostly to build softwares and build UIs for the um, for the finance industry. And that time data engineering was not known or I would say it would exist. But this term data engineer didn't exist that time. That time we used to have like database analyst or you can say like databases specialist. But eventually as the market changes and lot of artificial intelligence, machine learning, all those things came into picture. And one more thing was that the amount of data that started getting generated earlier companies used to have very limited amount of data. But with this change in the industry a uh, huge amount of data started getting generated and we needed more experts who can handle data and process data at a faster or I would say in a more meaningful way. So then this term data engineer started and when I joined Microsoft in 2019 I started working on this field, data engineering. That time I didn't know that, okay, this work is related to data engineering. I knew that, okay, I have to work on database, I have to write a store procedure, I have to do some things like get the data and do the processing. But yeah, eventually I got to know that. Yeah, it's all work is related to data engineering. And if I say like, what is data engineering? Basically it's all about uh, processing, analyzing and generating meaningful results from the data.

Speaker B: Yeah, so that's, yeah, that sounds interesting. So, uh, yeah, data engineering. So, uh, so what I have understood so far is that we have to process the data in such a way so that it can provide meaningful results.

Speaker A: Yes, that's correct. So yeah, what happens, what happens is like, like when the, this trend started of artificial intelligence and machine learning, the companies always had the data, but they never had a platform where they can basically generate business insights from that data. All they were doing was showing the data or using it for their own purpose. But to build that platform, basically those AI or ML engineers, they needed that platform. So to build that platform where they can have data from maybe different systems or from different domains, they needed data engineers. And that's how like they, this trend of data, ingenious, like started.

Speaker B: Yeah. So having worked with tech giants like Google and Microsoft, what unique perspectives did you bring from those experience to the manufacturing sector? Since you are working in a manufacturing company right now?

Speaker A: So.

Speaker B: Yeah, what do you think that, how, how that data engineering fits into a, uh, manufacturing company?

Speaker A: Yeah. So if I talk about manufacturing, right earlier, how manufacturing used to happen was like it was all kind of little manual. Not, I would say not manual, but I would say it was. They were not using the data in a right way. They were maybe getting like in manufacturing. Like if they are, if they have a manufacturing system and they know that, okay, if one of their system or one of their loop is not performing in right away, they had little checks where a person used to go and manually fix those things or the amount of products that they were generating. Right. The sales that they were having, they knew that, okay, they had a fixed set of clients and those were the client that they used to like deliver the products, but they didn't know that. Okay, like if they want to know, okay, what will be my sales in January or which month of the year we expect more sales or which month of the year we have a cooling period. Also, like, if they have an issue in their system, can they like, do they have to do it manually? Or if there is an automated way of detecting that issue before itself, or if not detecting maybe mitigating it in a, um, automated way. So that's where data comes into picture. They can they, if they are like they have a data of last 10 years, 20 years, or how long they have been running. They can use that data to analyze the trends or the charts and analyze the um, sales. All those things they can now do with their data. Right. Like rather than a person going and pulling the lever or pushing the button, they can have the AI or they can have the machine learning to do it for them. So that helps a lot in terms of manufacturing, that helps them in increasing their efficiency, in making them more robust. Yeah.

Speaker B: Okay. Okay. So what uh, I have seen is that earlier they were not uh, companies were not using um, data engineering that much because that they were not aware of the power of data. So in your view how has the data engineering transformed the uh, industry in recent years? So what changed? What caused this change?

Speaker A: Yeah, so basically yeah, uh, as I said before like it depend on the amount of data. That's the first thing. Like the amount of data that's can generate. And the second thing is the availability of tools. Basically earlier they didn't have so many tools to process the data. Like I would say like they didn't. Now there are new cloud providers, Azure or if I talk about like uh, Google, uh, cloud or Amazon. They are providing some readily available services in which you can just plug your data and they can give you business insights. So earlier there was no such tools to do this. Anyone has to do things manually or write an if and else statements or maybe I would say do it in a more old fashioned or I would say like a manual way. But now with this availability of tools, with things getting easier like the, and with the faster processing of data earlier because the rams or I would say the processors were not that fast to process such. But now like even in a small place you can fit a large amount of data as well as you can process it. And now clouds are there, you don't have to maintain earlier work m to do. They used to delete the history or maybe like kind of what I say is like they used to uh, uh, archive those data but now they have. Yeah, archive those data. But now they don't need to do that. They can just put it on a cloud or they can just put buy a system and put it there and they can process it. Just it's kind of right now like a plug and play systems and now algorithms are there. You can just provide the data and they can give you the insight. So that has caused this uh, change over the last 10 years.

Speaker B: Yeah, yeah, that's a really good insight. Actually the availability of tools was the issue that they were not able to process data and now machines have upgraded in such a way that we can transform data into meaningful insights. So can you provide an example, uh, where you have used uh, uh, data engineering and improved some manufacturing process or outcomes?

Speaker A: Yeah. So currently if I talk about my recent experience, like right now I am working for Void, which is basically a, uh, paper and pulp manufacturing company. What they do is like they build machines which are installed in paper mills and helps them like basically measure the pulp, basically the viscosity, the density of the pulse and their control loops that they have in which they maintain a particular temperature. They want to know the brightness of, of the paper that they are producing means if it's light white or if it's really white. So earlier what they used to do is like they used to take the sample of the paper and they used to basically check through, send it through a lab or something to check the brightness level on. But now what they have done is like they are using image processing. Like they have installed the cameras which captures the image and automatically through an algorithm, um, they get to know that, okay, what is the brightness of that paper? And if they don't, uh, are not getting the correct brightness, the software will automatically adjust, uh, the viscosity of the pulp or they will add more chemical into it until it gets the right amount of brightness. So that's where you can see a long process which might used to take one or two days has just gone into maybe it's a matter of few seconds. So that's, I think. Yeah, uh, one.

Speaker B: Yeah, yeah, that's really interesting. Yeah, yeah, that is really interesting to look into how computer vision is transforming the quality process that is really good inside.

Speaker A: Yeah, yeah, yeah.

Speaker B: So, uh, coming to challenges like what are the biggest challenges you have faced while implementing data engineering solution in manufacturing?

Speaker A: Yeah. So basically with data engineering it works apart from other software engineering, it works in a little different way. So in data engineering, in software engineering, okay, they want you to build a website or they want you to build uh, maybe a shopping website or anything like that, you know. Okay, like, okay, I need to use this in the back end. I need to this. But with data engineering, little. With data engineering, first you need to know what you really want. Basically you need to know the problem that you want to solve and then only you can start implementing based on that. So it's not a very. What I say is, like, it's not something very trivial. That. Okay, for data engineering I need to do this and this. Basically if you are building your database, you, before building the database, you should know what knowledge you want out of that data. So suppose if you want to know, okay, so if I take an example of Uber or anything you want to know, okay, how many drivers are there in this particular location? Or what is the average time that a driver took to reach the uh, pickup place or reach the destination. So first you should first, before dealing into data engineering problem, first the person should be asking the right questions, okay, what knowledge does he want? And once he have those questions, then he should dive into basically building the database. That is one key factor here. It's not like that, that, okay, I have built a table and later I will say, okay, like I want this because then doing the, it will be doing the reverse engineering. That becomes little difficult. That is one thing. And the second challenge that in terms of data faces is the. What I say is like, is the delay or I would say the timing, like how fast you want the data? Do you want the data to, okay, if you get the data today, you process tomorrow, or do you want the data to be in real time? So those kind of things matter a lot in the beginning of uh, any project. So yeah, that those things I think are like, if I talk about like these are the challenges that one should be like facing. Yeah, yeah, yeah. Before starting like any work in data engineering. Yeah, yeah, yeah.

Speaker B: When it comes to data, there is one word which is in fashion right now. It's called big data. So how big data, uh, comes uh, into picture in uh, industry, manufacturing industry or any other industry?

Speaker A: Okay, yeah. Uh, so if I talk about like big data basically in a survey which was published like quite some time back, I don't remember, but the amount it says that the amount of data generated till maybe I would just. I will give an example. I don't remember the exact year. Uh, maybe you can say that amount of data that generated GoT till 2015 is exactly the same as the data generated from 2015, 2020. Or you would say like the amount of data generated in the last five years is equal to the amount of data generated entirely since then. So as, as you can see, right, like technology is weaving into the our lifestyle and uh, everything that we touch or we feel or see generates amount of data. So big data is all about that enormous amount of data that is present maybe in terabytes is a small thing, petabytes. And big data comes into picture because when we started building tools or when companies started building tools for the data, they understood that, okay, like we cannot do it in an old traditional way because the amount of data is so huge. We need to Think of different ways to handle the data. Maybe it's a parallel processing or it's an asynchronous processing or those kind of things. Maybe using um, multi nodes or a cluster of servers to process those data parallelly. So when those kind of things came they invented this term called as Big data. Basically these softwares or Spark Apache, Spark Hadoop, they are built mostly to handle such amount of data and do that processing in a faster manner.

Speaker B: So uh, so with big data comes AI. So uh, you have told one AI example uh, in your uh, a company that where you are using computer vision to analyze the quality. So do you see any other applications that can be done in terms of manufacturing where we can put AI or machine learning?

Speaker A: Yeah, I think there are a lot of places there we can use. One thing that I think manufacturing in terms of manufacturing would be like to basically each manufacturing companies goes through like um, a shutdown period or some maintenance period. So I think there they, rather than like stopping business for that particular time, maybe they can analyze those things which they do over the period of time and see like if they can be done like in an automated way or AI can take a decision for that. Like I'm not, I might not be very right but I'm just guessing. Yeah. And one more thing that I can think of. Yeah Would be. Yeah like predictive maintenance. As can be a word to say that. Yeah like one can know that okay. Based on the data that those machines are generating what time their maintenance can be done. Like they can do that maintenance in slots or in some other way. And uh, another thing would be to use it the sales figure. Right. Like based. Suppose they know that okay in January they have the major sales for the product that they are building. So they can basically by analyzing this data on month basis or on week basis they can uh, basically increase or reduce their workforce. That I think would help them a lot. Maybe they can see the data for last 20 years. Okay. In last 20, every time this month they have less amount of say. So they can maybe give a cool off period or a time off period for the workforce or things like that. Yeah, yeah, yeah. So that's one thing yeah could think

Speaker B: of that's really interesting to know. I, I think many people will find out that there are many applications of AI and big data uh in various industry. But man we, we are a manufacturing based podcast so we are focusing from. On picking. On picking your brain on this uh, uh, only.

Speaker A: Yeah, yeah.

Speaker B: So let's talk about like collaboration between IT and manufacturing team See there are manufacturing guy who work guys who work on shop floors and they don't know much about data. They just want uh, they just want to see some meaningful insight. So how important is it for data engineers to have an understanding of manufacturing processes and vice versa. Like if how important is for shop floor guys to understand about data.

Speaker A: Yeah, yeah, yeah. So I think it's all about like I think for the Yasha floor guys I think what they only look for first thing is that if they are in the manufacturing industry for quite some time, maybe 10 years, 20, they are working. So the problem is that they don't like a lot of change in there. Like if they are pulling a lever of they are pushing a button every 30 minutes or every one hour, uh, they don't want to change that. That is one thing. So to basically build their trust on uh, like newer technologies or an easier way first we need to make sure that we are providing something very simple to them. If we are even providing rather than a button, if we are providing a ui, we should make sure that it has to be very user friendly and for them to easier to understand that is one thing. And they don't. They might not go into data but yeah they can help the data engineers to understand or build such system which they can use it easily. And for data engineers I would say like it's really, really important to understand the problem. As I said before, right. For data engineers to work first they need to know okay what knowledge I need to be, I need to extract out of data or what questions are uh there or what are the problem statement to understand that. I think any company like Cipla or any manufacturing companies, they basically send their engineers to their plant and show them each and everything and explain them in detail. So that helps to build a better system for the people who are on the ground.

Speaker B: Yeah, yeah, yeah I understand. Like uh, having process understanding will give you an insight into the brain of the manufacturing people and they will like you more.

Speaker A: Yeah, yeah, that's correct. Yeah, yeah, yeah.

Speaker B: Actually what I do is uh, even in my projects what I do is I first make them understand what uh, facilities or what solutions does a uh prop tool provides and then we show them a few demos and then they will be having better understanding and the requirements will be more clear and final. Otherwise it will be an agile project. Changes coming at any times that.

Speaker A: Yeah, yeah, yeah. That's what basically manufacturing is not like it or software industry where you can just push a change every month, right? So you need to be very, very Robust and kind of like very, very. I think it should go through a lot of testing and checks before like you push. So yeah that's the difference actually between normal uh software or service provider, service industry or manufacturing. Yeah.

Speaker B: So coming to the emerging trends like there are a lot of technologies coming, bubbles are there, there are some long term players also. So we hear about IoT blockchain, edge computing and all those things. So what do you think? Which emerging trend in data engineering do you believe will have the most significant impact on the manufacturing sector in the next few years?

Speaker A: Uh, okay, yeah I think uh. Okay if you stalk like I'm little like very new to manufacturing so yeah like I'm um, like let me think about it, it's not on top of my head but yeah with I think blockchain or I would say like edge computing is still a far fetched goal for manufacturing because they are right now still like very uh, I would say traditional and they are still like very hesitant to like um, yeah adopt new technologies but one thing I think they should be really doing is that like putting a lot of uh effort into like utilizing their data or do real time data analysis of their machines and system. That is I think is really, really important for them to basically understand. Also one more thing would be to like involve a lot of sensors or I would say like involves a lot of IoT devices. That is I think very important for manufacturing. Yeah to incorporate, maybe not use them right now, just incorporate them in their machines so that they can get the data and later they can use, think of a problem or they can think of using it the right way. So I think IoT devices is something like they talked about image processing or I would say video processing or I would say like basically showing the data as a chart or a trend or in those way. I think that would help them a lot and they uh, yeah they are quite afraid also to do any new change which I think they should not be like they should try to maybe adopt because if I talk about things like if you say Nokia, right. Nokia was such a great company but they were not very um keen into using the Android or adopting their phone and now they are nowhere. So I think only those manufacturing companies will last which will basically adopt The M, these IoT and data engineering or machine learning technologies because that's going to help them improve their efficiency in a way they couldn't think of it right now.

Speaker B: Yeah actually in right at present I get a lot of uh calls for uh IoT integration and AI and uh ML applications of the data. But the things when I visit the plant is that the data is not there, they are not capturing data right now. And if you want to train AI models, you will be needing lots and lots of data. So uh, so any advice for aspiring data engineers who are looking to uh, enter the field of uh, data engineering? What skills and knowledge area would you recommend focusing on?

Speaker A: Yeah, I think if you want, if anyone wants to get into data engineering, the first thing he should be very good at like is uh, I would say like SQL because I know like old but it's used widely and people say they are good at school but actually they are not like SQL is very vast. That is one thing. And the second thing would be to understand how to like. Lot of people focus on languages, like they want to learn Python, they want to learn Start, but that's not the real concern. I think database modeling is one of the main important concept in data engineering which everyone should focus on. Like right now people are not getting theory. They know 10 tools, they know Azure, they know GCP, they know lot of things. They know horizontal vertical scaling, all those big, big words. But if they ask them, okay, can you build me a uh, database model for my shopping application or for my uber systems or for my manufacturing? They might not be able to do that. So I think they need to get into their theory, the old theory, the schema, the snowflake and the um, star and all those things. They need to get into that before like getting into data engineering. I would say so yeah, they've dive later into theory and then go for talking about like multi processing or clusters and all those things. Yeah, yeah, okay.

Speaker B: Okay, yeah, that's a really good advice. First clear your uh, core concepts and then dive into tools.

Speaker A: Yeah, then there uh, I went to tools. Yeah, yeah, that's correct. Ah.

Speaker B: So are uh, are there any resources or pieces of advice you wish you had when you started your career in data engineering?

Speaker A: Yeah, yeah, yeah, I think uh, yeah, like when I started actually I didn't know about data engineering. And then uh, yeah that's, that's the one thing like I thought, I think like I should have studied little bit before like working because sometimes you know what happens like when you start working, then work becomes so much that you don't, you just do the work and you don't deluge into like theory or getting to know what you are doing. I think that's one of the major problems that everyone faces because they are doing their work nine, uh, hours, uh, a day and then they don't want to know that why or how they are doing this work. So that is one problem that I think like almost 90% of the guys do. So I would recommend that before starting any work or even if you have started also maybe read little bit about it. 10 minutes or 15 minutes to get a better. That will help anyone to build a better system and it will save them also a lot of time. So yeah, I think that is one thing. That is the one advice that I could think of. Yeah, I also think sometimes like I should have studied. Like I have worked on Spark for almost a year. But yeah, I did the work, I understood it from the top. But yeah, like I sometimes I think I should have dig deeper, studied more so that I could have like right now I think, okay, I faced that problem that time and I solved it that way right now. If I had that problem I would have solved in a different way because now I know things more. So that is one thing that I think like everyone should be doing.

Speaker B: Yeah. Yeah. Actually that's, that's what. When we look back and uh, think about the challenges we faced in the beginning of our careers and how we dealt with, dealt with it and we feel quite stupid and see that we could have done it better if we would have studied better. So.

Speaker A: Yeah, yeah, yeah.

Speaker B: So um, basic concepts are very important to uh, master on first.

Speaker A: Yeah, yes, yes. And yeah, whatever we work on, we should always know like, okay, like this is the problem and this is what we are doing basically like what we are trying to solve here. That is one thing that's really important that one should know. Yeah.

Speaker B: So that brings to the end of our podcast episode and ah, Summer, you have been really providing very good insight into the manufacturing and data engineering role available and the challenges we face in this industry. And I hope that you will be coming again soon and shower us with more of your knowledge.

Speaker A: No, no, that's, that's. I'm really grateful and yeah, thanks for this opportunity. It's been quite some time since I gave any podcast and yeah, thanks for inviting me and yeah, thanks a lot for your time also.

Speaker B: So this was Samarkhan and Musar talking about data engineering and manufacturing and until next episode of Innovate X Stream, have a nice day and a week. Thank you.

Speaker A: Um, all. Sam.