Susan Shu Chang on Bridging Foundational Machine Learning and Generative AI

Transcript

Roland Meertens: Welcome to The InfoQ Podcast. We are live at QCon in San Francisco, and I'm joined by Susan Shu Chang, who is working at Elastic as a principal data scientist.

At QCon San Francisco she is hosting the track AI and Machine Learning for Software Engineers: Foundational Insights. Welcome to The InfoQ Podcast.

Susan Shu Chang: Yes, thank you for having me.

AI and ML for Software Engineers: Foundational Insights [00:43]

Roland Meertens: So should we go over what the talks were, which you had in the track, and which ones you liked, or what's overall foundational insights you were hoping to instill in the attendees?

Susan Shu Chang: Yes. Yes, so first of all, very glad to be here. We wrapped up the AI and ML track just yesterday, which was the first day of the conference. With the track, I had hoped to curate various topics in foundational machine learning because nowadays, we're hearing a lot about generative AI, and we do have a track for this on the second day of the conference. But still, a lot of things are powered by these fundamental machine learning, which were also huge topics and still are very, very relevant these days. So for example, when people are going on YouTube, or Spotify, or Netflix, we're getting served personalized recommendations such as, oh, you might like this, or just the home feed of all these social medias and content sites. That's all powered by machine learning and has been for the last many, many years. So I wanted to cover these foundational techniques and machine learning families like recommender systems, reinforcement learning.

We also had talks about machine learning in production, which is one of the main concerns we have, because sometimes we get a fancy machine learning model, but you're actually not able to run it in production, which is serving users, in which case it's not actually bringing in the monetary value. So we want people to be learning about some of these nuances of getting machine learning in production. Then we also had some talks about knowledge graphs, and large language models, and how you can merge the two techniques together to build a better experience.

So just in summary, I wanted to cover various areas because the engineers here, they're working at all these companies that definitely use machine learning, and they have definitely been adjacent to it or heard of these techniques, but I wanted to help them bring concrete example from people who have been doing it on the machine learning side, but also help them understand, just behind the scenes, how this is working.

Recommendation engines [02:49]

Roland Meertens: Yes, so there are, especially, for example, the recommender in search engines. I think the talk by Netflix was very interesting, about how to get better recommendations and really tailor them to specific people.

Susan Shu Chang: Yes.

Roland Meertens: That was great. Also, you mentioned reinforcement learning. That was also very interesting, that some people who are working in Meta came up with an idea on how to use long-term interaction and see it as a reinforcement learning problem.

Susan Shu Chang: Yes, yes, exactly.

Roland Meertens: Are there any specific tools you would then recommend to people, or any specific topic you would recommend someone to look at first if they want to get into machine learning now?

Susan Shu Chang: Yes, I think still, if we're not talking about generative AI, specifically, just for the foundational, let's say, neural networks and things like that, I still think PyTorch and TensorFlow are still very valid to play around with and get started. And they have a lot of tutorials online too, to build an initial end-to-end app, so not just a Hello World model or something like that, but rather you might have a model and then you build a simple lightweight web app. So this connects more to what software's know and are familiar with, and how it's actually being done in industry, because if you just have the model, it's not doing anything, right? You always have to connect to the UI, which all of our software engineer partners at my job, or at other companies I've been with, or in these large companies like Google and Netflix, without the software engineers, the machine learning is not even touching the customer.

Roland Meertens: Was it in your track that someone talked about how to actually get your model deployed? So-

Susan Shu Chang: Yes. Yes. Wenjie Zi.

Roland Meertens: Yes, okay.

Susan Shu Chang: Yes, the speaker from Grammarly was talking about common pitfalls that prevent machine learning models from getting into production. And actually a lot of the challenges have to do with finding the right problem, and communicating with management, and finding the right solutions as well. And that's actually something for me, as a principal data scientist, I do a lot at work too, which is find out what is overkill and what is more practical.

Sometimes we actually really like to start with the rules-based approaches in practical machine learning, which is we have now a baseline to compare the model with. If a machine learning model is doing worse than just a simple rules-based approach, then that means either we shouldn't use machine learning or the machine learning needs to be greatly improved in which we would really need to go back and see, is the resourcing we're spending on this even worth it? Is the ROI worth it? Things like that. So early stages, we actually really focus on some of these, it's not technical per se, it's these rules and business logic, but it's essential so that we don't waste millions of dollars down the road if we're solving the wrong problem.

Roland Meertens: Yes. So do you think that in terms of AI machine learning nowadays, there's a lot of tools on the market, especially with this GenAI, everyone can call the GPT-4 API now, do you think it's still good to have a prior knowledge about AI, machine learning, specifically, understand how these models work? Or do you think it's now open to everyone?

Susan Shu Chang: I think it's more open to everyone now. So one example I was giving to attendees in the unconference that we did, so the unconference at QCon is a open session where people can come in, they put up the topics they're interested, and then they go into groups and discuss each topic. So one of the examples I was giving in the unconference topic, which was I think how software engineers are evolving in the landscape of ML. I forgot the exact wording, something like that, but one example I gave was that our company, we built an AI assistant. So this is like a chatbot, and people can bring their own API key from a vendor, like OpenAI, or Anthropic, or something, and they can put that into the Elastic, let's say they're using the cloud version of us, they can put the API key in, and then we have the chatbot, which will answer questions about the Elastic product.

So we're not in the business of competing with OpenAI, right? It's like a collaborative thing where the users are using OpenAI and us, but the engineers were able to basically built out the MVP completely by themselves, because they can just handle the stack and then they can handle building the connectors to OpenAI, right? The calling. So they can build all of this completely by themselves. Where we come in more so is regarding, let's say, improvements or evaluations. And if we had to do fine-tuning, that's also where we would probably come in as well, which is scenarios which involve opening the black box and tweaking things. The reason for bringing up this example is that engineers can build a lot of things without knowing exactly the internals these days. It didn't use to be like this, and I think this is a good direction. I'm very excited about this.

Roland Meertens: Yes, but do you have a feeling that if people don't understand what's really happening, that it's setting them up for failure? So if someone doesn't understand what a embedding is, then maybe their retrieve for augmented generation becomes a bit harder-

Susan Shu Chang: Yes.

Roland Meertens: ... if you just treat Elastic as a magical black box, which gives you some random documents back.

Susan Shu Chang: Yes, Yes, that's a good point. I think that's a good way you put it, which is it's easy to build an MVP. It's difficult to debug unless you know how it works. So I think that's where it's important to know about the fundamentals. It's important to know about how these various tools and techniques are working. Yes, so it was for if you want to tweak, tune and debug, that's where that comes in.

Roland Meertens: Yes.

Susan Shu Chang: MVP, simple, I think.

Roland Meertens: Yes. I think in general, often in software, the MVP is easy.

Susan Shu Chang: Right. Yes. Yes.

Roland Meertens: Actually getting it to deliver value to your customer seems to be the harder aspect.

Susan Shu Chang: Yes. That is true.

Elastic and Generative AI [08:44]

Roland Meertens: So, Yes, given that you work at Elastic, are you then mostly working on tools for generative AI, or other aspects as well?

Susan Shu Chang: There is a mix. Myself and my team own different types of use cases and models. So I'll share briefly what a few of the cases we have. So one is we ship supervised models, that people can run on their data, that is in Elasticsearch. So we do not have access to Elastic customer's data. So the way we are training these models is through our own data gathering, and then we will train the models. So because of the Elastic Common Schema, we know that the customer's data, if they're ingesting it in the Elastic Common Schema, the data will look exactly like this. The field names will be exactly this. It'll be in this format. So that's how we're able to ship these models, train them on our end, and then we ship them downstream so the customers can use it in their environment without our involvement at all.

So that's thanks to the Elastic Common Schema and the fact that people love to ingest their data into Elastic. So that's one part that we own, and the other part is augmenting the AI assistant piece.

Susan Shu Chang: So as I mentioned that the engineering team we're able to build out the AI assistants by themselves, and then we'll conduct tasks like evaluation or research, R&D, into what other functionality can be added into this. We have to prove out that it actually works before we can add it into the product. Yes.

Roland Meertens: What do you mean that you have to prove that it works? How do you prove that it works?

Susan Shu Chang: Yes, I can share one proper example that myself and team have blogged about. So I'm just thinking of the public example, where we used large language models to conduct session summarization. So a common way people use Elasticsearch is to ingest logging. So we have observability vertical as well as the security vertical. So this is because people ingest logs into Elastic. So the data is complex, right? So we'll have these large JSON objects, which might have information like command line logging or performance logging, and what user are you, what host are you? And we wanted to see if large language models are able to understand, given a stream of logs, is there any, let's say, what did the user do in this session? What did they do?

So we did an R&D project where we grabbed real logs that we had, and then we did some data preprocessing. We tried different ways of preprocessing the data. We fed basically these logs through a large language model and see if the large language model's able to summarize from these logs that maybe the user SSH'd into here. They ran these commands. What did these commands do? They're not just spitting it out.

Roland Meertens: Yes, but these logs can be extremely large, right?

Susan Shu Chang: Yes, yes, exactly.

Roland Meertens: So the context window has to be very big as well.

Susan Shu Chang: Yes. That's one thing we had to do. We did a few things to narrow down the content that we're feeding into the language models. So at the time, of course, I think 4o, we're using GPT-4o, and then that had a larger window. Oh, and we tried the Gemini one, which had a huge, I forgot, 1 million, was it?

Roland Meertens: Yes.

Susan Shu Chang: Yes. Yes. It was like we also tried that, but actually it didn't mean that it performed better. We had to go back and extract, let's say, relevant fields. So we noticed that some fields were more... They're not super important or they had repeated data. A simple example is, let's say, we have timestamp, or some sort of beginning time, or whatever. We can deduplicate some of those, as well as some highly repeated events. If it made sense to consolidate them, then we would have to consolidate that. We didn't spend too, too much time on this, it's just very first pass heuristics and things like that, to narrow down what we were passing into the language model. And how we evaluated this was we had a subject matter expertise. We had a security researcher. They would summarize it in the way that they would normally summarize it, like what is the relevant process here? What is happening in this session?

Roland Meertens: As in they would summarize what the user did in the session?

Susan Shu Chang: Yes, using how they-

Roland Meertens: In natural language.

Susan Shu Chang: Yes. Yes, we had a bit of, almost like a survey, like what did you think that they did, or what do you think was the relevant piece in these logs? And things like that. And then we compared that to the response that the large language model was giving. Yes, so-

Roland Meertens: Yes, interesting.

Susan Shu Chang: ... that was interesting. I would say right now, it's still just an R&D project. We were not merging it in to the capability at this time, but I don't know, we might revisit it in the future, but these are some things our team does, which is we'll run these research experiments and then we'll do evaluations, and then we'll see. We might have a few on the docket that are in various states of maybe being in the product. So I'm just sharing this one that we have shared about publicly. Yes, but it doesn't mean it'll be in the product. It just means that we have tried this out and we've learned our lessons from it, because actually this project, we created a template for the evaluation, we were just using that for the subsequent projects. So it's very repeatable.

Recommended tools to start with Machine Learning [14:27]

Roland Meertens: Yes, and also the AI tools and the AI methods are going so fast nowadays. It's really hard to know what the next big thing is, or how you have to adapt your database to make such work faster or better. It's quite difficult. Are there any specific algorithms or tools which you would recommend people to start with if they are starting with machine learning, or anything specifically they should try?

Susan Shu Chang: I think these days for creating some GenAI MVPs, I think a lot of people starting out, they're using the tools like Streamlit. So that's a, I guess it's a very... I don't know how to describe it.

Roland Meertens: It's a front-end for Python.

Susan Shu Chang: Yes. Yes, Yes. Very abstracted front-end, where they put all the boilerplate into... Wrapped it all up, so it can create-

Roland Meertens: Serving of your function as a website.

Susan Shu Chang: Yes. Yes, exactly. So people can create a interactive chatbot with a few lines of code. It could have a drop-down for some basic responses or built-in templates, and then you can have a input box. I don't know how it is in web dev terminology, where you type.

Roland Meertens: Yes.

Susan Shu Chang: Yes.

Roland Meertens: Yes, Yes, st.input.

Susan Shu Chang: Okay, okay. Yes, and then the outputs. Then you could do that real quick. We can use that for, actually in industry, people who are in ML, they might use that to spin up a real quick prototype, real fast. It's not pretty, it's not scalable or whatever, but it's just to demonstrate how it might look to someone. Let's say we're communicating to a product person. It's better to show them with that rather than, let's say it's outputting some JSON. Showing them that is a bit less tangible. Yes.

Roland Meertens: Yes, and also convincing other people to run a Python script and have the right dependencies always takes more time than just sending them the URL to your local Streamlits.

Susan Shu Chang: Yes. Yes, exactly. Yes. So that's a tool I think has been useful. Really, it's more at the toy learning, right? But it's got to start somewhere.

Roland Meertens: Yes.

Susan Shu Chang: Yes, Yes, not building some gigantic, end-to-end for your learning project. Definitely not.

Roland Meertens: Yes. Yes, no, I also use it quite a lot. I really like it. Any other tools you recommend then, or-

Susan Shu Chang: I think that's the main ones, because I think just, let's say, if we set aside GenAI for a second, right? In the past, I would recommend people run anything of their choice, really, they could run... I am talking more about the Python stack because that's where I work mostly in, but just pick your own thing of your choice. But I would recommend people, let's say, they can spin up something with Django, they could do a FastAPI, they could do Sanic, whatever. There's a few, sorry, it got loud, but it just, there's a few pick one you like, and then just try building a simple app with it. It's a bit more complex than just, let's say, using Streamlit, but it's the same concept, which is you understand the machine learning inputs and outputs, and that helps so much with people learning, how does this fit into the entire infrastructure?

Roland Meertens: Yes. Yes, and then also that way you can really have your Python function finally as a service, or as an API people can call.

Susan Shu Chang: Yes.

Machine Learning Interviews [17:45]

Roland Meertens: That's quite nice. You also wrote a book, Machine Learning Interviews. How did you come up with the idea to write that?

Susan Shu Chang: Yes, so this is an interesting story. I guess I'll try to keep it brief. So I was doing an online course for O'Reilly, as I mentioned. So it's a live course. So I answered the questions live, and the topic was around machine learning interviews. So it's to help people that are quite new or they just want a quick refresher, but I guess, the main target is to people who are either, let's say, new grads or career transitioning. And I was running this live course, it seems like it was doing pretty well, got high ratings. So then the publisher asked if I would do a proposal for the book, and actually initially, I said, "No, I'm busy. I'm not putting in a proposal". But then after some time, I eventually did go back and create the proposal.

I think sometimes for different topics, they might have various proposals at a time. So let's just have a simple example. Let's say they're looking for a topic on orchestration or whatever, right? They might have some people who have different proposals all on orchestration. So of course they can't always accept all of them, so they have to pick the one that maybe fits the best at that time for them. So then I was lucky that mine was selected. So I set about writing the book, it took me a year, and I actually do really like writing.

Roland Meertens: Did you spend a full-time year on it, or did you do it during your spare time?

Susan Shu Chang: Spare time.

Roland Meertens: Oh.

Susan Shu Chang: In my spare time, Yes.

Roland Meertens: Very impressive.

Susan Shu Chang: Yes. Yes, I think it's a lot of time management. So three to five hours a week, sometimes more, sometimes less. But I would say, I'm more like a slow and steady person, which is I just do a bit, do a bit, do a bit, and I prefer not to crunch on the book.

Roland Meertens: Yes. Yes.

Susan Shu Chang: Yes, so it's less stressful, but it was a lot of time management and energy management too.

Roland Meertens: Yes. Interesting. So if people are listening to this and they have an interview upcoming, is there a fixed set of skills nowadays which everybody needs to master, or is it still, does it depend on where you're applying and what kind of machine learning job you're applying for?

Susan Shu Chang: Yes, it's very dependent. So one of the things I was going for with the book is I first start with describing a very general machine learning life cycle and workflow. So you might start with the raw data and you would have the data pipelines that are coming in. You would have then, let's say, model development, and then later on you would have the model deployment. So given this simple life cycle or workflow, the jobs will be corresponding to one or more of those responsibilities. But one thing a lot of job seekers, and even I had made this mistake before was, I might search online for a, let's say, machine learning interview. And the same article with the title Machine Learning Interview could cover something about MLOps. An article online called Machine Learning Interviews could be covering stats, and probability, and those classic algorithms. They could also be covering stuff like data analysis.

So people might think, "Oh, I must prepare for every single one of these", but the fact is, it really depends on in the job description, what part of the life cycle the job is targeted towards, and then prepare for that first, prioritize those questions. So this is what the book covers, how do you select what topics to prioritize, and the main machine learning algorithms, the main general ops skills and tooling that are common in machine learning, and so the responsibilities of the various roles. Yes, because I mentioned that it's targeted towards beginners or people who...

Actually, there is a funny thing in ML, which is people who are experienced, depending on what kind of thing they were doing, they might still want to read this book, and I've heard people do that. They still found the book helpful, even if they're already working in ML, which was, I wasn't surprised, but I also was a little bit surprised, because people transition between model training to just more on the software side, back to model training to more data analysts to... Back and forth, back and forth all the time. So they want to know what they should cover in their interview.

Roland Meertens: Yes. Yes, but I think that's... Now, I'm going to go on the rant, but it can be frustrating that if you are a software developer, you know that you will have some kind of LeetCode interview. So you can practice LeetCode and then you're probably set for the next 20 years, in terms of skills. The machine learning field is rapidly evolving and is growing and growing on a daily basis. It makes it harder to really learn the skills. Also, one of my favorite things is to interview people who have been in this field for forever, because they always have very different answers to questions you ask, compared to people who just graduated from courses. Are there any questions people frequently ask, which you hate, where you're like, "You should stop asking this. This doesn't add any signal or any value to your interview process."?

Susan Shu Chang: Oh, I think one mistake that might be made is focusing maybe a lot on the specific tooling. So when I'm reviewing resumes and when I'm the interviewer, let's say, they mentioned they use PyTorch. So in my mind, I'm literally equating that to TensorFlow and some other tools.

Roland Meertens: Yes

Susan Shu Chang: But then I think someone who focuses too much on the exact tooling, they might hear, "Oh, you didn't use this", so maybe it doesn't count. Or there was some of those data warehousing tools, like let's say, you were using Presto or some other things. I forgot, Teradata, or... This was a few years ago. So it's some other tooling that was hot at the time. So if you use one of them, you use all of them. You can go from one of those two BigQuery immediately. You don't need to learn BigQuery, you just need to write SQL, and you can write it anywhere. It can be written and you're good. So I think someone who cares a bit about those, it's not super helpful for finding the right candidate. It limits the pool unnecessarily.

Roland Meertens: Yes. Okay. So the tip is to not focus on specific tooling, but focus more on engineer skills.

Susan Shu Chang: Yes.

Roland Meertens: Yes, and then maybe as a last question, if there are companies who are setting up the machine learning interview, do you have any tips for what questions they should ask, or what they should focus on, or how they should structure their interviews better?

Susan Shu Chang: So I think it's really, really, really important to know what skill you want to hire for. So I mentioned the general machine learning life cycle with, do you want this person to be doing maybe the early steps of bringing the data in, so more like data engineer type questions? Or do you want them to be focusing on the model training? Or let's say, with GenAI, it's a little bit different, but just let's say, we're focusing on ML model training. If their primary responsibility is that, then ask about that. And if they are more responsible for machine learning, the operations, MLOps, then ask about that. And that's going to be really important to see if they can do this role that you have in your team.

I think one mistake I've also seen is, it's not even new, it's not a new problem, but since ML was quote, unquote, "hot", people would maybe think, "Oh, our company needs machine learning". But then they didn't sit down and really define what this is, where this is going to sit, how it's going to fit in the product. So their interview is all over the place because, as a friend of mine mentioned, they want a unicorn and they don't even know where they want this unicorn in the company. So-

Roland Meertens: They don't have a space for the unicorn.

Susan Shu Chang: Yes, Yes, exactly. So just in summary, it's useful to know where you want this role to be, what skills that would make them contribute to the role, and then build the interview from there to ask the target questions. If you're hiring for more senior people, then the common senior things apply, like, oh, lead a project, mentored people. But I think for ML, for more senior people or just mid-level, I would expect them to actually have more insight also into the other responsibilities.

So let's say, someone who's more like mid-level to senior and they're responsible for model training, they will start to have more of a understanding of MLOps and how that works than a more junior person who's focusing on model training. So the more senior you get, the more visibility you need to have, end to end. So I think the interview questions could also try to capture that. Have you worked on maybe some other parts of the pipeline, or have you been very focused on this, let's say, model training? But, again... Yes.

Roland Meertens: Yes, so more like the breadth of skills rather than just a narrowness only on focusing model training.

Susan Shu Chang: Yes, or T-shaped or something like that, because it'll help with the scaling up as people grow into senior, which is the more you've seen end to end, the more you're able to communicate. Because I did a brief stint doing MLOps, and I don't normally do that, but it helped me communicate so much more to anyone that I was working with in the future regarding how I'd like this, let's say, design to work and the things like that. It helps immensely for people who are scaling up.

And, oh, another thing, I have some more thoughts, actually, because for even larger companies, there's definitely more specialization happening. So let's say someone can go really, really deep on model training. They're a research scientist or something. So there's definitely exceptions where maybe they don't need to know as much about the deployment process, but it still differentiates them. It will differentiate them given another candidate with the exact same skills.

Knowing a bit more about what comes before and what comes after is going to be really useful, but Yes, I think this is less relevant for the newcomers, but more about growing in the ML role. I think actually another thing I want to mention, which is people could interview really well for a certain skill, but if that's not what the company needs, a company with a well-realized vision of what they want to hire, they could still get rejected. So I'm going to bring up a, I can't share details-

Susan Shu Chang: ... but our team has two broad sections. One requires more, let's say, ops skills, and one requires more of the ML modeling skills. So at each time we're hiring, we'll hire someone who might fit one profile more, the other profile more. So what makes an ideal candidate today might not be the ideal candidate next month if we're hiring for the other profile, because we know what we need on this team and we know what we're missing right now.

So I think for people who are interviewing, it's also a matter of luck, almost, or you have to really align with the role in which... Let's say, they really want to, for people who are coming from a software background, there are teams who, at this point in time, they want to leverage your software skills, so then maybe your other skills will matter less. And then sometimes they want to leverage other skills more. So even if the job description doesn't make that very clear, that might be something that holds people back in interviews as well, but it's a bit of luck in what is needed.

Roland Meertens: Yes, in terms of luck, it's just about too, is there a good fit, right?

Susan Shu Chang: Yes.

Roland Meertens: And I think the fit is less clear for machine learning jobs than it is for development jobs.

Susan Shu Chang: Yes. Yes, yes, yes.

Roland Meertens: But in that sense, maybe we just don't have enough specializations yet, unless people want to apply as prompt engineer, then... Yes.

Susan Shu Chang: Yes.

Roland Meertens: Thank you very much, Susan, for joining The InfoQ Podcast and joining us at QCon San Francisco, and hope you have a good time.

Susan Shu Chang: Yes. Thanks again for having me.

Mentioned:

• The book Machine Learning Interviews