What Is Over Fitting in Machine Learning and How to Avoid It?
In this post, I will cover the topic of overfitting in machine learning models. It is widespread for data scientists, especially those who are just starting and are not very experienced, to train models and obtain a training metric that they think is very good. Then, when predicting new data, the model performs much worse than in training. It is a fundamental problem when productizing models, since models that seem right in development and productization are a hecatomb. This is often because these models are over-fitted to the training data.
Generalization of Knowledge
As if it were a human being, learning machines should be able to generalize concepts. Suppose we see a Labrador retriever for the first time in life and they tell us “that’s a dog.” Then they show us a Poodle and ask us: is that a dog? We will say “No”, because it is not at all like what we learned previously. Now imagine that our tutor shows us a book with photos of 10 different dog breeds. When we see a breed of dog that we did not know, we will surely be able to recognize the canine quadruped while being able to discern that a cat is not a dog, even if it is furry and has four legs.
When we train our computational models with an input data set, we are making the algorithm capable of generalizing a concept. So that when it is consulted for a new unknown data set, it can synthesize it, understand it and return a reliable result given its ability to generalize.
This concept is one of the key ideas in machine learning. Overfitting is called making a model so tight to the training data that it does not generalize well to the test data. In the field of data science for developers, it is essential and vital for final products.
It should be remembered that the objective of machine learning models is to obtain patterns from the available training data to predict or infer new data correctly. In other words, the key concept is to train and get general designs that can be extrapolated to new data. Something similar occurs in human beings; the overfit would happen when we learn things by heart without understanding the concept.
When Does Overfit Occurs?
Overfitting occurs when a machine learning system is overtrained or on abnormal data, causing the algorithm to “learn” patterns that are not general. Learn specific characteristics but not the prevailing trends, the concept. More sophisticated models tend to overfit more than simpler models. Furthermore, when faced with the same model, the smaller the amount of data, the more likely it will be overfit.
How to Evaluate it?
There are several methods to assess when a model is overfitting. However, few machine learning development companies are having full access to testing mechanisms. One of the ways is through the graphs of training and test errors. The error graphs of the model can be represented in the data used to train (Train) and in the data used to validate the model (Test). Ideally, both errors should be as close as possible. That is, as the ability to capture details increases, the error in Train decreases. Still, there comes the point when the model begins to “memorize” non-general patterns, and the mistake in the Test begins to increase.
How to Steer Clear of Overfitting?
Overfitting can be avoided in several ways, the clearest of which are as follows:
- Incorporating more data: By having more data, the algorithm is more likely to generalize better, taking into account more data types.
- By changing the parameters of specific algorithms, making the algorithms more straightforward: by making the algorithm more uncomplicated, it fits less into the data, and it is not possible to overfit the training data. For example, reducing the depth of a decision tree adjusts less by making the model more straightforward.
- Incorporating regularization: There are parameters in many algorithms that allow the settings to be regularized. Thus, avoiding, to some extent, the over-adjustment. Commonly there are two types, which are L1 and L2.
AppStudio is the best Toronto app developers having full expertise over machine learning and artificial intelligence. We develop chatbots and other digital systems, using machine learning to give you complete superiority in the market.
I hope this article has been useful to you to understand the importance of making models that generalize well to new data, thus avoiding overfitting of training data. Contact us for further info without any obligation.