6 Mins Read

What is Transfer Learning?

transfer learning

A History of Transfer Learning

A paper published in 1976 by Stevo Bozinovski and Ante Fulgosi addressed transfer learning in neural network training. In this article, we’ll look at transfer learning mathematically and geometrically. Using transfer learning to train a neural network on images representing letters was reported in 1981. It was demonstrated that transfer learning could have both advantages and disadvantages. Discriminability-based Transfer (DBT) was formulated in 1993 by Lorien Pratt as the first step for transfer in machine learning. An article on transfer learning was published in 1997 in the journal of Machine Learning. In 1998, a formal analysis of the theoretical foundations of “multi-task learning” was published in this area.

Transfer Learning Methods

Let’s review three general methods for transfer learning:

Training a Model for Reuse

Assume we want to perform task A, but we don’t have access to large amounts of data, to train a model for it. We can train the model for a similar task (B) with more available data, and then use the trained model for our desired task (A). The decision to use the whole model or just several layers depends on what we are trying to accomplish.

Using Pre-Trained Models

Using pre-trained models is also a transfer learning technique. These models are readily available and can be leveraged for our intended task. This type of transfer learning is most used in deep learning.

Feature Extraction

Another transfer learning method is extracting the most important features. Machine learning typically requires experts to manually create features, which takes a great deal of time and effort, but neural networks are capable of recognizing which features are important. They are capable of providing a good combination of features, even for the most complex tasks, within the shortest (optimal) amount of time. It is possible to use these learned features for other purposes. We can extract features using the initial layers of the network and modify the final layers to meet our needs. Feature vectors from raw data are generated as a result of this network.


Transfer Learning in Computer Vision

Training “deep convolutional neural network” models could take days or months. Using previously trained models from computer vision benchmark datasets such as ImageNet is a shortcut for this process and is known as transfer learning. Transfer learning often involves using models trained on one problem as a starting point for solving related problems. Pre-trained models can be used directly for feature extraction in preprocessing to train new neural networks quickly, and reduce generalization errors. The weights in reused layers may be used in the training process and adapted to new problems, in a way transfer learning can be considered a weight initialization scheme. This may be useful when the first related problem has much more labeled data than the current problem, and the similarity of problem structures is useful in both contexts.

ImageNet Large Scale Image Recognition Challenge (ILSVRC) is an annual competition in which teams enter their models to compete for higher visual recognition accuracy on a given dataset. ILSVRC has inspired novel architectures and training methods for convolutional neural networks. The models produced for ILSVRC are trained on more than a million images to recognize 1000 categories of general features. The competition has also pushed them to the limits of performance in this specific task. These models are available for download or for direct use through APIs. All these advantages make them ideal candidates for the initial model in transfer learning.


 transfer learning process
Figure1. Overview of the transfer learning process. Source
Related articles
Trauma Detection on Pelvic Radiographs using Computer Vision Algorithms
A timely and accurate diagnosis of the proximal femur and pelvis injuries in trauma patients is essential to saving...
Detecting and classifying aesthetic defects in different sizes, shapes, and positions immediately after the casting process is an essential...
deep learning
Nowadays with the help of computer vision technology and image processing we can classify broken and normal bone X-ray...
Annotating Computer Vision Projects
Choosing the right images for training, validating, and testing computer vision algorithms will significantly affect your AI project's success....
Metaverse is one of the fastest-growing technologies today. A comprehensive review of computer vision concepts in Metaverse is presented...
casting defects
Automated X-ray systems are improving quality through multiple objective inspections, and reducing labor costs, and increasing productivity and consistency...
Subscribe to our newsletter and get the latest practical content.

You can enter your email address and subscribe to our newsletter and get the latest practical content. You can enter your email address and subscribe to our newsletter.