Current Issue

Journal of Korea Technical Association of the Pulp and Paper Industry - Vol. 53, No. 2

[ Article ]
Journal of Korea Technical Association of the Pulp and Paper Industry - Vol. 53, No. 2, pp.5-14
Abbreviation: J. Korea TAPPI
ISSN: 0253-3200 (Print)
Print publication date 30 Apr 2021
Received 13 Jan 2021 Revised 09 Mar 2021 Accepted 12 Mar 2021

Paper Defects Classification Based on VGG16 and Transfer Learning
Yun-hui Qu1, 2, ; Wei Tang3 ; Bo Feng2
1Computer Teaching and Research Section, Xi’an Medical University, Xi’an, Shaanxi, 710021, Professor, People’s Republic of China
2Department of Electric and Control Engineering, Shaanxi University of Science & Technology, Xi’an, Shaanxi, 710021, Student, People’s Republic of China
3Department of Electric and Control Engineering, Shaanxi University of Science & Technology, Xi’an, Shaanxi, 710021, Professor, People’s Republic of China

Correspondence to : †E-mail: (Address: Computer Teaching and Research Section, Xi’an Medical University, Xi’an, Shaanxi, 710021, People’s Republic of China)

Funding Information ▼


There are some problems in traditional paper defects classification, such as the poor generalization performance, less types of recognition, and insufficient recognition accuracy. The deep learning method provides a new scheme for paper defects classification. However, due to the small sample size of paper defect images set, the over fitting phenomenon is easy to appear in the training process. Aiming this problem, a transfer learning method based on convolutional neural network model is proposed.Firstly, freezing the first seven construction layers of VGG16 network which has been trained by ImageNet, and fine tune the rest convolution layers with the paper defect images set to complete the feature extraction; Secondly, the full connection layers for classification are improved to meet the needs of paper defects classification; Finally, transfer learning strategy is adopted in the training process to improve the efficiency. The experimental results demonstrate that the paper defects classification proposed in our approach can improve the efficiency and accuracy of paper defects recognition. The approach will beneficial for the web inspection process.

Keywords: Transfer Learning, VGG16, Paper defects classification

1. Introduction

The modern paper industry is a technology, capital, resource and energy intensive industry, which is developing towards large scale, automation, high efficiency and low energy consumption. Improving paper quality is one of the important directions of independent research and development of paper industry. In the paper industry, the paper defect is the paper that does not meet the quality and technical requirements. Some paper defects are caused by poor technical operation or poor technical quality during paper process, while others are caused by poor environmental sanitation in the factory. Some of the paper defects are relatively continuous. If not take measures, the quality of the paper will be more impacted.1,2)

At present, web inspection system (WIS) in the production lines generally use linear-array CCD cameras to collect the paper image sequence, and then use industrial computer to detect and classify the defects.3,4)

The WIS based on machine vision can inspect the paper continuously and accurately, which plays a great role in improving the production efficiency and product quality of the paper machine and improving the competitiveness of paper mills.

The classification of paper defects is a very important step in the whole process of WIS. Accurate classification of all kinds of paper defects is of great significance for finding out the causes of paper defects and early warning of paper machine stoppage.5,6)

The traditional paper classification method is mainly composed of feature extraction and paper defects classification. The flow chart is shown in Fig. 1.

Fig. 1. 
Flow chart of traditional paper defects classification algorithm.

Before the paper defect classification, it is necessary to extract the features. There are many kinds of paper defects, which are similar, and the differences are very small. How to select the paper defect features which can effectively distinguish all kinds of paper defects is the difficulty of paper defect feature extraction. After extracting the features, we need to use the corresponding features as the classifier data, and design the classifier to classify the paper defects. Therefore, in order to achieve good classification results, we need to select a large number of different types of paper defect features, and design a powerful classifier.

According to the above analysis, the traditional paper defect classification methods mainly have the following problems:

  • 1) It is difficult to select and extract the features of paper disease image. For example, the gray level can be used to distinguish dirty spots and holes, but it is difficult to distinguish cracks and holes; Geometric features can be used to distinguish cracks and holes, but cannot distinguish cracks and folds. And there are other paper defects, which need some more complex combination of features to distinguish. Therefore, in the traditional paper defects classification, feature extraction algorithm is one of the difficulties in the whole process.
  • 2) It is difficult to design and train the classifiers. The design of paper defects classifier needs to consider the convergence speed, over-fitting, training data set and other issues. Therefore, the classification ability of the current paper defect classifiers is limited to 4-5 most common paper defects.
  • 3) It is difficult to match the features extraction and classification of paper defect images. In the traditional paper defect classification process, the feature extraction algorithm and classification algorithm are two categories of algorithms, which are completed in two stages. Therefore, the paper defect classifier designed in the later stage cannot fully adapt to the paper fault features extracted. Thus, the classification effect of the involved classifiers is not ideal in the actual use process.

To sum up, the traditional paper defect classification algorithm needs to be carried out on the feature extraction. Every time a paper defect needs to be identified and classified, it needs to research and extract its features and transform the classifier, so it is difficult to improve the classification and efficiency.

In this work, a paper defects classification method based on deep convolution neural network (DCNN) and Transfer learning is proposed. The deep-seated features of the paper defect image can be extract automatically through convolution operation, which effectively solves the problem of feature extraction, improves the accuracy and scalability of paper defects classification.

2. Paper Defects Classification Based on VGG16 and Transfer Learning

The research of convolutional neural networks (CNN) began in the 1980s and 1990s. After the 21st century, with the development of deep learning theory and the improvement of numerical computing equipment, CNN has developed rapidly. CNN is a kind of feed forward neural networks with deep structure including convolution computation. It is one of the representative algorithms of deep learning. Convolutional neural network has the ability of representation learning and can shift invariant classification of input information according to its hierarchical structure. Therefore, it is also known as “shift invariant artificial neural networks (SIANN)”. CNN imitates the biological visual perception mechanism, which can be used for supervised learning and unsupervised learning. The convolution kernel parameter sharing in the hidden layer and the sparsity of the connection between layers make the convolution neural network grid like with less computation Topology) features, such as pixels and audio, have stable effects and have no additional feature engineering requirements for data. It is found that the expression ability of CNN can be enhanced by increasing the depth of CNN, so the use of deep convolution neural network (DCNN) for image classification is a hot spot in current research.7,8)

2.1 The choice of basic network structure

Among many basic deep convolution neural networks, VGG16 has the characteristics of good classification performance, regular network structure and easy modification. The network structure of VGG16 is shown in Fig. 2.9)

  • 1) Input layer: the image size of input layer is 224×224 pixels;
  • 2) Convolution layers: VGG16 has 13 convolution layers. There is a maximum pooling layer behind the second, fourth, seventh, tenth and thirteenth layers. The size of convolution kernel in all convolution layers is 3×3. In this way, the superposition of several smaller convolution kernels to achieve the convolution effect of larger convolution kernels is equivalent to adding implicit regularization, which can improve the classification ability of the network and increase the operation speed of the network.
  • 3) Full connection layer: VGG16 has three full connection layer neurons, the number of which is 4096,4096,1000, and 1000 in the last layer, corresponding to the number of categories to be recognized in the ImageNet image recognition challenge (1000). After that, it is easy to compare with the sample data and calculate the error through the Softmax layer.

Fig. 2. 
Flow chart of traditional paper defects classification algorithm.

In general, VGG16 network integrates the belief of “deeper network architecture, higher accuracy”. The smaller filters are used to increase the depth of the network and avoid the problem of over holding parameters. In view of the above analysis, VGG16 network is used as the basic network for improvement in this work.10,11)

2.2 Paper defects classification based on improved VGG16 and transfer learning

In the classification based on DCNN, enough training samples can avoid serious over fitting problem. However, in the current paper defects classification research, there is no paper defect images database for training and performance testing. Researchers can only use industrial cameras to collect enough paper defect images on the actual production line, and then manually find defects in all images, and classify the defects to establish a database for research and comparison. Because the emergence of paper disease is a small probability time, so the sample data obtained in this way is insufficient.

In this work, the deep convolution neural network classifier is constructed by combining the idea of transfer learning and fine tuning deep convolution neural network.

2.2.1 Freeze and fine tune the parameters of convolution layers

The methods of using transfer learning will be different. There are four main situations:12,13)

  • 1) The new data set is very small and similar to the original data;
  • 2) The new data set is very small and different from the original training data;
  • 3) The new data set is very large and similar to the original training data;
  • 4) The new data set is very large and different from the original training data.

The paper defect images classification task belongs to the second of the above situations. In the process of using transfer learning to classify paper defects, the choice of transfer learning mode and how to fine tune it are the keys to improve the classification accuracy and save the network training time. The paper defect images data set is smaller than the original data set, so it is better to train only one linear classifier. And the difference between the paper defect images data set and the original data set is large, so the higher-level features are different. So in this work, the parameters of low-level convolution layer are frozen to extract the basic features of image texture, edge and shape and paper defects image data set is used to fine tune the high-level convolution layer.

The freezing and fine-tuning strategies are combined in the work. The first seven layers of VGG16 convolution layer trained by ImageNet dataset are used, and their parameters are frozen; the parameters are randomly initialized from the eighth layer, and the subsequent network is fine-tuned layer by layer by using the paper defect images data set. The specific idea is shown in Fig. 3.

Fig. 3. 
Fine tuning VGG16 network.

2.2.2 Improved full connectivity layer

The original VGG16 network has three full connection layers, and the number of neurons is 4096, 4096, 1000. According to the actual situation of paper defects classification, the full connection layer is redesigned.

  • 1) The pooling layer between the convolution layer and the full connection layer is mproved. The global average pooling layer is used to reconstruct the 3D array from the convolution layer into a 128 bit vector.
  • 2) Freezing the parameters of the first fully connected layer, adding two fully connected layers, the number of neurons is 256 and 6 respectively, so that it can meet the classification output of 6 types of images, including 5 kinds of paper diseases and 1 type of paperless diseases.
  • 3) The linear rectification function ReLU (Rectified Liner Unit) is used as the activation function to prevent the gradient from disappearing. ReLU is a piecewise function defined as follows:14)
ReLUx=max0,1=xif x00if x<0[1] 

ReLU is the most common activation function in deep convolution neural network because it eliminates the gradient super-saturation effect and is more simple to calculate.

  • 4) In the full connection layer, dropout algorithm is used to prevent over fitting phenomenon.
2.2.3 Improved paper defects classification method

In this work, the deep convolution neural network VGG16 model is freeze and fine tune with the transfer learning method. The ImageNet data set is used to train the VGG16 network. The first seven layer’s parameters of the trained VGG16 network are frozen, and the labeled paper defect images data set is used to fine tune the other convolution layers to extract the features of paper defects. Finally, the improved full connection layer and Softmax layer are used to classify paper defects. The specific process is shown in Fig. 4.

Fig. 4. 
Flow chart of paper defects classification method based on VGG16 and transfer learning.

3. Data Acquisition and Preprocessing

In this study, five kinds of paper defect images and normal paper images were collected by the laboratory web inspection equipment.

Firstly, the paper defect images were collected by industrial camera. Then the collected images were normalized. Since the collected paper images were large (The size of paper images was 4096×1024 pixels which are collected in the laboratory), and the input image size of VGG16 model was 224×224 pixels, it was necessary to pre-process the image size.

In the pre-processing process, each paper defect image should be able to represent the paper fault area to the greatest extent, and reduce the algorithm complexity as much as possible. Therefore, in the research process, the smallest circumscribed square of the paper defect area was taken firstly. Secondly, the circumscribed rectangle image is scaled to 224×224 pixels. Finally, each sample image was manually labeled to obtain the final paper disease image data set.

Because the appearance of paper defects was a small probability event, the training and testing of convolutional neural network need a large number of data sets. Although the method of transfer learning and fine tuning depth convolution neural network can reduce the requirement of sample size, sufficient training data can avoid the problem of over-fitting. Therefore, in order to increase the scale of paper defect images data set, in the process of making paper defect image data set, mirror image, rotation and other operations (such as 90°, 180° and 270° rotation of the paper defect images, which can increase the paper defect images data set while maintaining the paper defect images characteristics) and multiple acquisition of the same paper defects under the interference of different light sources were used to expand the data set. Finally, 370 dirties, 340 holes, 280 bright spots, 350 folds, 360 cracks and 300 normal paper images were obtained as the paper defect images data set, part of the image data in the paper defect images data set were shown in Fig. 5. And the training set and test set were divided according to the ratio of 4:1.

Fig. 5. 
Paper defect images data set.

4. Results and Discussion
4.1 Experimental environment and parameter setting

The experimental environment was Win 10 operating system and Intel CoreTM i7-7500U CPU, 8G RAM, 256G SSD. Matlab 2019b platform was used to build and train the model.

The self-supervised training optimizer was: Sdgm (Stochastic Gradient Descent with momentum). The initial learning rate of the model was set to 0.0001, and the momentum was used to adjust the learning rate of the model. The setting value of momentum was 0.9, batch size was 64 and dropout was 0.5.

4.2 Experimental results and analysis

The recognition of five kinds of paper defects and normal paper images without paper defects were statistically analyzed in the experiment. The results are shown in Table 1.

Table 1. 
The classification results of the test samples
Dirty spot Hole Bright
Fold Crack Normal
Total Accuracy (%)
Dirty spot 74 0 0 0 0 0 74 74 100
Hole 0 63 5 0 0 1 62 68 91.18
Bright spot 0 4 49 0 0 3 49 56 87.5
Folds 0 0 0 66 2 2 66 70 94.29
Crack 0 0 0 2 68 2 68 72 94.44
Normal paper 0 0 0 0 0 60 60 60 100
Total samples 379 400 94.75

It can be seen from the data results in Table 1 that the accuracy rate of bright spot is the lowest, which is mistakenly identified as the hole or normal paper. Through the observation of the sample images of recognition errors, it is found that there are two main cases of recognition errors:

  • 1) In the process of paper defect images acquisition, the top light source and the bottom light source were used to irradiate. If the brightness of the top light source was too high, the gray value of the collected bright spot area was close to the normal paper, which was difficult to distinguish. So in this case, it would lead to wrong classification;
  • 2) On the other hand, if the brightness of the bottom light source was too high, the gray values of the hole and the bright spot were very close, which was difficult for the system to distinguish.

In addition to the above two cases, when the light source intensity of the paper defect collection device was optimized properly and the collected bright spot image was clear, the classification accuracy was higher. For cracks and folds, when the contrast was low, the wrong classification would occur. Except for the above, the recognition accuracy of other types of paper defects was above 90%, and the recognition effect was good.

Some of the test images and the classification results are shown in Fig. 6.

Fig. 6. 
Classification test results.

In order to further verify the effectiveness of the proposed classification method, the widely used algorithms such as SVM algorithm,15) BP neural network algorithm,16) and AlexNet convolution neural network algorithm17) were compared and analyzed on the same data in terms of classification accuracy and time complexity. Among them, the results of SVM algorithm and BP neural network algorithm were obtained by reproducing references 15 and 16 on the data set which was proposed in section 3. For AlexNet convolutional neural network, the trained model in Matlab 2019b was directly called, and the data set in Section 3 was used to fine tune it. The results are shown in Table 2.

Table 2. 
Performance comparison of four algorithms
Algorithm used Accuracy (%) Testing time
(Unit: s)
BP neural network 89.25 15.35
SVM 91.30 15.26
AlexNet 93.45 37.36
Proposed 94.75 18.56

It can be seen from the results in Table 2 that compared with the traditional BP neural network and SVM, the classification accuracy of the paper defect classification method based on transfer learning strategy proposed in this paper has been greatly improved. For BP neural network and SVM, in the previous research process,15) the classification accuracy can reach more than 95%, but which was based on a series of preprocessing and feature analysis and extraction of paper defect images. In the process of this experiment, the accuracy of classification had decreased significantly without changing the previous selected feature vector after adding a new paper defect. So it can be seen that the traditional paper defect classification method has poor scalability.

For the AlexNet network and the classification method based on transfer learning strategy proposed in this paper, the test recognition rate on the test set is 96.37% and 96.46% respectively. But due to the limited number of paper defect samples, the training sample size is insufficient, which makes the accuracy on the verification set lower. Although the accuracy of proposed method in test set is also decreased, it is obviously better than that of AlexNet. In addition, the time consumption of proposed method is also significantly less than that of AlexNet network, which is close to the traditional classification algorithm.

5. Conclusions

In this paper, the deep convolution neural network model VGG16 is combined with the transfer learning method. Transfer learning strategy is used to freeze and fine tune the trained VGG16 model to make it suitable for paper defect classification. The paper defect images data set which is collected by the laboratory web inspection system is used for experimental verification, the accuracy rate can reach 94.76%, and the convergence speed is fast. There is no over fitting phenomenon between the experimental training set and the verification set, and the effect is good. The experimental results show that the proposed method has some improvements in accuracy, real-time and scalability compared with the traditional methods, and can be effectively used in the process of paper defect classification on the production line.


This work was partially supported by Scientific Research Project of Shaanxi Provincial Education Department (17JK0645). We sincerely thank for the funding of the project.

Literature Cited
1. Qu, Y. H., Tang, W., Feng, B., Web inspection algorithm for low contrast paper defects based on artificial Bee colony optimization, Journal of Korea TAPPI 52(2):43-51 (2020).
2. Qu, Y. H., Tang, W., and Wen, H., An algorithm for on-line image segmentation of multiple paper defects based on fast 2-D threshold method, Journal of Korea TAPPI 52(4):28-37 (2020).
3. Shan, W. J. and Tang, W., Multivariable dimension-reduction and synergic control strategy for cross-directional basis weight of papermaking process, Journal of Korea TAPPI 2(51):76-87 (2019).
4. Wu, G. A. and Wu, D., Application of industrial camera on paper making, paper and paper making 35(07):8-16 (2016).
5. Tang, W., Zhang, C., and Feng, B., Review of advanced control technologies and modern control systems for paper industry, China Pulp & Paper 39(8):14-25 (2020).
6. Zhou, Q., Chen, Y., and Shen, T. Y., Review of paper defect detection system based on machine vision technology, China Pulp & Paper 35(5):72-79 (2016).
7. Gao, L. L., Zhou, Q., and Wang, W. G., Paper defects recognition based on layer feature extraction of CNN, Transactions of China Pulp and Paper 34(4):52-58 (2019).
8. Huang, J. and Zhang, G., Survey of object detection algorithms for deep convolutional neural networks, Computer Engineering and Application 56(17):12-23 (2020).
9. Wang, X. Y., Wang, Q., and Cheng, D., Detection of split pins defect in catenary positioning tube based on three-level cascade architecture. Chinese Journal of Scientific Instrument 40(10):74-83 (2019).
10. Aravind, K. R., Raja P., Disease classification in eggplant using pre-trained VGG16 and MSVM, Scientific Reports 10(1):1419-60 (2020).
11. Sun, W. Z., Ma, Y., and Yin, Z. Y., Banknote dirty identification method based on convolutional neural network, Journal of Chinese Computer Systems 41(7):1508-1212 (2020).
12. Zhuang, F. Z., Qi, Z. Y., and Duan, K. Y., A Comprehensive Survey on Transfer Learning, Proceedings of the IEEE 109(1):1-27 (2019).
13. Cao, X. J., Yao, Y., and Yan, Y. L., Convolutional neural network flower image recognition using transfer learning, Computer Application and software 37(8):142-148 (2020).
14. Ansgar, R., and Milena, P., Advances in verification of ReLU neural networks, Journal of Global Optimization 77(2):1-44 (2020).
15. Qu, Y. H., Tang, W., and Feng, B., On-line detection and classification method based on background subtraction and SVM, Packaging Engineering 39(23):176-180 (2018).
16. Duan, Y., Chen, K. X., and Liu, X., Research on paper defect detection and recognition based on BP neural network, Journal Xi’an University of Technology 34(2):235-239 (2018).
17. Li,G. M., Xue, D. H., and Jia, X. H., Paper defects classification based on muti-scale image enhancement combined with convolution neural network, China Pulp & Paper 37(8):47-54 (2018).