What Most People Don't Get About Machine Learning Engineering
The diverse technical and soft skillset needed to become a machine learning engineer
👋 Hi, this is Sarah with the weekly issue of the Dutch Engineer Newsletter. In this newsletter, I cover data and machine learning concepts, code best practices, and career advice that will help you accelerate your career.
When I first entered the data field, there was no such thing as machine learning engineering. The roles available were data engineering, data science, and data analysis. Since then, the machine learning engineer role has rapidly progressed and will only continue to grow in popularity, especially with large language models. However, this popularity has also highlighted, at least for me, the disconnect between what the industry needs and the traditional definition of the role.
As a machine learning engineer at a high-growth startup, I am starting to see how those traditional definitions have fallen short in reality. I feel a responsibility to uncover the truth about what it is really like to be a machine learning engineer.
In today’s article, we will cover
What is it like to be a machine learning engineer?
Machine learning engineer is an odd position. You are literally sitting at the intersection of disciplines. Ben Wilson, a principal data scientist at Databricks and author of Machine Learning in Action, describes that machine learning engineers need to know:
“enough software development skills to be able to write modular code and implement unit tests. They don’t need to know about the intricacies of non-blocking asynchronous messaging brokering.” I do want to emphasize that we do more than just write modular code and implement unit tests. We have to be able to understand how services are built (think how APIs are built and used or running your machine learning pipelines in Docker).
“enough data engineering skills to build (and schedule the ETL for) feature datasets for their models, but not to construct a petabyte-scale streaming ingestion framework.”
“enough visualization skills to create plots and charts that communicate clearly what their research and models are doing, but not to develop dynamic web apps that have complex user experience (UX) components.”
“enough project management experience to know how to properly define, scope, and control a project to solve a problem, but need not go through a Project Management Professional (PMP) certification.”
His list describes the diverse skillset that we as machine learning engineers need to have, and I would like to add one—data science. Machine learning engineers need to know enough about data science skills to be able to calculate statistics, model, and identify problems with modeling.
What is it, exactly, then what a machine learning engineer does? Machine learning engineer mediates between all of those disciplines (and teams as well) to set up data and feature engineering pipelines, manage infrastructure, deploy those models and pipelines, monitor them, and find out what is causing models to degrade.
What other skills are needed?
Machine learning engineering skills combine a ton of unique skills together. It is the hotpot of the data world.
I am unfortunately oversimplifying what data engineers and data scientists do, but if you smash together their technical skills, you will find what a machine learning engineer needs to know. Think about SQL, Python, API development, Docker, and Spark from the data engineering side. Smash that together with how to build features, the most commonly used types of models, the machine learning lifecycle, and statistics to evaluate models from the data scientist side. But that will only cover the technical skills of a machine learning engineer.
For this role, you will need to have strong soft skills. Knowing how to communicate effectively is a necessity as we are often addressed to answer two main challenges. The first challenge will be to educate and present what machine learning can do to further the goals of the business. And most of the time the problem is that they have not seen the power that machine learning can provide to the business (or they have been stung in the past by projects that did not address what the business needed because it was too small in scope or cost more than they needed to). This brings me to the second challenge. You will need to clearly articulate the needs of the business and identify where machine learning can help. Sounds pretty similar to the first challenge, right? It is because you are diving in more deeply and actively presenting the use cases. Addressing these challenges requires solid communication skills, as we must collaborate across multiple teams (both inside and outside of the data organization), and clearly present and write about what machine learning is and how we are addressing business needs with it.
On top of communication skills, machine learning engineers will need a strong sense of curiosity, learning ability, and think critically and holistically. You and I are constantly bombarded with the new tools and developments in machine learning (hello ChatGPT!) and how we can implement them into the business. We have a responsibility as machine learning engineers to demystify what those new tools are, learn them, and then share that knowledge in layman's terms to understand what folks need in the business. Finally, and honestly, most importantly, we are often given complex problems and thus are required to think critically and holistically to solve those problems. If we want to build a model, we first have to define the business problem we are trying to solve, present that use case to get initial feedback, get the data, get the data into the right format for the model, make sure that we are using similar business logic (and understanding) as the rest of the data organization, build the model, integrate our model with the product (engineering), get the results from the product, run statistical analysis on that to see how our model is actually doing, figure out when to do retraining, and then train the model and restart the deployment process. This will require continual problem-solving, figuring out how to get everything working to fit together into one piece like a masterpiece.
What has my current experience been thus far?
I have been in this role for the last 3 months. And what a wild ride it has been.
Up to this point, my biggest asset has been my desire to learn and how to help others become better. I was also very fortunate to have been a data engineer and a data scientist before this role, so many of those technical skills translated.
What surprised me the most is how much I did not know about this role. Like many of you, I thought that it is literally just trying to get models together and monitor them. Instead, I have had to collaborate closely with colleagues from other teams to ensure that we were needing necessary data for our future models and infrastructure and integrate those into the organization's broader data ecosystem. These interactions have allowed me to gain a more holistic understanding of the machine learning process and to develop a greater appreciation for the collaborative, interdisciplinary nature of this field.
The second thing that surprised me the most is that you are literally the guide for your own team and others. For the data engineering team, you are sharing with them how you will be using the data (which is different than how data analysts are using it). For the data science team, your goal is to make them better stewards of the code that they are producing. For other teams, you are carrying the future of machine learning and its black boxiness on your shoulder, so the goal is to share how machine learning can become an asset to the business.
Final thoughts
Machine learning engineering is not what I originally thought it would be — focusing solely on models and monitoring. Instead, I have learned that this role requires a diverse set of skills, ranging from technical expertise in software development, data engineering, data science, and project management, to strong soft skills such as communication, curiosity, and critical thinking.
Being a machine learning engineer has therefore been challenging but also rewarding as I have the power to bring together teams to help the business achieve its goals.