What Is Over Fitting in Machine Learning and How to Avoid It?

In this post, I will cover the topic of overfitting in machine learning models. It is widespread for data scientists, especially those who are just starting and are not very experienced, to train models and obtain a training metric that they think is very good. Then, when predicting new data, the model performs much worse than in training. It is a fundamental problem when productizing models, since models that seem right in development and productization are a hecatomb. This is often because these models are over-fitted to the training data.

Machine Learning

Generalization of Knowledge

As if it were a human being, learning machines should be able to generalize concepts. Suppose we see a Labrador retriever for the first time in life and they tell us “that’s a dog.” Then they show us a Poodle and ask us: is that a dog? We will say “No”, because it is not at all like what we learned previously. Now imagine that our tutor shows us a book with photos of 10 different dog breeds. When we see a breed of dog that we did not know, we will surely be able to recognize the canine quadruped while being able to discern that a cat is not a dog, even if it is furry and has four legs.

Overfit Concept

This concept is one of the key ideas in machine learning. Overfitting is called making a model so tight to the training data that it does not generalize well to the test data. In the field of data science for developers, it is essential and vital for final products.

When Does Overfit Occurs?

Overfitting occurs when a machine learning system is overtrained or on abnormal data, causing the algorithm to “learn” patterns that are not general. Learn specific characteristics but not the prevailing trends, the concept. More sophisticated models tend to overfit more than simpler models. Furthermore, when faced with the same model, the smaller the amount of data, the more likely it will be overfit.

How to Evaluate it?

There are several methods to assess when a model is overfitting. However, few machine learning development companies are having full access to testing mechanisms. One of the ways is through the graphs of training and test errors. The error graphs of the model can be represented in the data used to train (Train) and in the data used to validate the model (Test). Ideally, both errors should be as close as possible. That is, as the ability to capture details increases, the error in Train decreases. Still, there comes the point when the model begins to “memorize” non-general patterns, and the mistake in the Test begins to increase.

How to Steer Clear of Overfitting?

Overfitting can be avoided in several ways, the clearest of which are as follows:

  • By changing the parameters of specific algorithms, making the algorithms more straightforward: by making the algorithm more uncomplicated, it fits less into the data, and it is not possible to overfit the training data. For example, reducing the depth of a decision tree adjusts less by making the model more straightforward.
  • Incorporating regularization: There are parameters in many algorithms that allow the settings to be regularized. Thus, avoiding, to some extent, the over-adjustment. Commonly there are two types, which are L1 and L2.

AppStudio is among Best Mobile App Development Company Toronto. We are providing cost-effective yet innovative mobile apps development services under one roof.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store