Photometric Stereo-Based 3D Reconstruction Method for the Objective Evaluation of Fabric Pilling

: Fabric pilling evaluation has been considered as an es‐ sential element for textile quality inspection. Traditional manual method is still based on human eyes and brain, which is subjective with low efficiency. This paper proposes an objective evaluation method based on semi-calibrated near-light Photometric Stereo (PS). Fabric images are digitalized by self-developed image acqui‐ sition system. The 3D depth information of each point could be obtained by PS algorithm and then mapped to 2D grayscale im‐ age. After that, the non-textured image could be filtered by using the Gaussian low-pass filter. The pilling segmentation is con‐ ducted by using global iterative threshold segmentation method, and then K-Nearest Neighbor (KNN) is finally selected as a tool for the grade classification of fabric pilling. Our experimental re‐ sults show that the proposed evaluation system could achieve ex‐ cellent judging performance for the objective pilling evaluation.


Introduction
Pilling of textile is a very unpleasant feature that results from daily washing and wearing, which is affected by fiber properties, yarn properties, and fabric structure.For the quality control of fabric products, pilling evaluation has been always considered as an important issue.Traditional evaluation methods have some disadvantages of high cost, subjectivity, poor reliability and low efficiency.It is necessary to develop the objective and digital technology to replace the traditional methods.
For this purpose, researchers used some objective evaluation methods based on 2D image analysis techniques in fabric pilling evaluation such as the Fourier Transform, Wavelet Transform, and Artificial Neural Networks.Yun et al [1] used the Fourier Transform algorithm which divided the image information into low and high frequencies where low frequencies included the deterministic structure and high frequencies represented the noise and pills.Their experimental results showed that the method was suitable for pilling evaluation of woven fabrics.Deng et al [2] used multi-scale 2D Dual-Tree Complex Wavelet Transform (CWT) to extract six characteristics at different scales from images of textile, indicating this evaluation system had excellent performance for knitted, woven, and nonwoven fabrics.Xiao et al [3] transformed the pilling image to the frequency domain using Fourier Transform, and combined it with energy algorithm, multi-dimensional Discrete Wavelet Transform and iterative thresholding algorithm to obtain pilling segmentation images.This objective evaluation method was capable of obtaining full and accurate pilling information and deep learning algorithms achieved 94.2% classification accuracy.Wu et al [4] proposed a Convolutional Neural Network-based pilling evaluation system by extracting pill features and texture features.The rating accuracy of their model reached 97.70%.
However, the analysis of pilling based on 2D images has limitations due to lighting as well as pattern variations.For more accuracy, more and more methods based on 3D information are used in fabric pilling evaluation.Kang et al [5] developed a noncontact 3D measurement method for reconstructing the 3D model of the fabric.A CCD camera was used to capture the image of the laser line projected on its surface.Using a heightthreshold algorithm, the 3D model was converted into a binary image, and the parameters extracted from that image were used to calculate the pilling grade.The results of their method correlated well with the manual evaluation method.Xu et al [6] investigated a 3D fabric surface reconstruction system that used two side-by-side images of fabric particles taken by a pair of ordinary cameras without special illumination.To make the system resistant to fabric structures, colors, fiber contents, and other factors, robust calibration and stereo-matching algorithms were implemented.Liu et al [7] proposed a method based on structure from motion (SFM) and patch-based multi-view stereo (PMVS) algorithm for pilling evaluation.The pilling segmentation was achieved by adaptive threshold segmentation and morphological analysis.
Multi-view stereo can only reconstruct the macroscopic contours of the fabric surface without significant texture details.Laser triangulation can get texture details, but this method is usually high-cost, timeconsuming and difficult in operation.In most other methods whose 3D reconstruction is based on surface features, such as stereo matching, the 3D model cannot be generated when the surface features are blurry.This paper designs a simple and low-cost system for pilling evaluation which can not only recover the macroscopic contour of the fabric, but also reconstruct its tissue points.This system first reconstructs a 3D model of the fabric surface using semi-calibrated nearlight Photometric Stereo (PS).In mapping the 3D model to a 2D image, a low-pass filter is used to eliminate fabric texture for the 2D depth image.The binary image of pills is segmented by a global iterative threshold to obtain the pilling number and area.Finally, the classification is completed by K-Nearest Neighbor (KNN).Figure 1 shows the flowchart of fabric pilling evaluation.

System Setup
We have designed a computer image acquisition system, which consists of a data acquisition facility and image data analysis facility, as shown in Fig. 2. Imaging hardware used in the data acquisition facility includes a box, a high-resolution digital camera (NIKKOR D7200), a macro lens (NIKKOR AF-S) and eight LED light sources (1 W).The PS is sensitive to the light information in the image, and the inside walls of the box are painted with black matt varnish to minimize the reflection of the scattered light and the effect of stray light.To calibrate parallel light, the light sources were concentrated with a range of 15 luminous angles.The acquired images are analyzed and reconstructed using MATLAB R2020a.PS was first proposed by Woodham [8] in 1980, which takes multiple images with different illumination using a specific LED configuration and then infers the 3D model.The traditional PS method assumes that lighting is caused by infinitely distant point sources, and the intensity of the light source needs to be calibrated.In this paper, a near-light model is developed, and the source intensities are assumed to be unknown.Modeling near-light sources means that low-cost lighting devices such as LEDs can be used, and the calibration procedure is simplified by assuming unknown source intensities (semi-calibrated).This paper uses semi-calibrated nearlight PS technology [9] to complete the 3D reconstruction of fabric surfaces.
For 3D reconstruction of fabric surfaces by using semi-calibrated near-light PS, we designed the multilight image acquisition system (Fig. 2) and conducted the calibration to obtain camera intrinsic matrix and the light position parameters which are crucial for 3D reconstruction.The process of calibration is described in Section 2.1.The image acquisition is conducted after system calibration, and eight light sources are controlled to irradiate and shoot the samples in turn.A variational formulation is established, which describes the relationship between the grayscale values of pixel corresponding to the surface point of 3D model, and tackle the nonconvex variational model numerically to complete the 3D reconstruction.

System Calibration
According to the method proposed by Zhang [10] , taking a set of calibration board images, the camera is calibrated with the Matlab Camera Calibration Toolbox to obtain the camera intrinsic matrix.Figure 3 shows one of the corner point detection results.
Triangulation is used to calibrate the location parameters of the light source [11] .A pair of metal spheres are placed in the scene to generate visible highlights for each light source, and multiple highlight points are extracted by thresholding.Then the Canny operator [12] is used to detect the sphere contours, as shown in Fig. 4. The light source position parameters are calculated from the actual radius of the sphere, the focal length of the camera, and the physical pixel size of the camera sensor.

Photometric Model
The relationship between the surface point s of the 3D model and the 2D pixel point i = (xy) can be expressed as follows: where z(xy) is the depth value of the 3D model and K is the intrinsic matrix of the camera.
Considering non-parallel illumination features, the attenuation caused by distance F k l can be expressed as follows: where k represents the k-th light source, l represents the irradiated light vector of surface point s, and x k l is the light source position.Thus, the incident light vector l k (s) at the surface point s can be expressed as: The normal vector n(s) is the unit-length vector proportional to ¶ x s ( xy ) ´ ¶y s ( xy ) : On the camera sensor, f is the focal length, dX and dY are the physical lengths of pixels in the X and Y directions.In the pixel coordinate system, u 0 and v 0 indicate the sensor center coordinates, β represents the angle between the horizontal and vertical edges of the photographic plate.
The albedo ρ(s) expression was modified following the method of Quéau et al [13] : The relationship between the grayscale values of pixel p k (i) corresponding to the surface point s can be expressed as: where e k represents the light source intensity of the k-th light source.
Combining the above equations into a system of nonlinear partial differential equations: (10) j = 1  N are the corresponding pixels.Then, the discrete counterpart of Eq. ( 10) is written as: Optimize the discrete part assuming that Q consists of all rank-1 N-by-M matrices: where λ is the user-defined parameter of the Cauchy estimator, and in our experiment λ = 8.The nonconvex model of Eq. ( 12) is minimized alternatively over variables θ and logz.In each subproblem, we solve a local quadratic model of Eq. ( 12) using the positive definite approximation of Hessian to achieve 3D reconstruction.

Preparation of Samples
In this study, 98 fabric samples were classified in pilling grades.Fabric samples were cut into squares of 30 mm×30 mm, resulting in an image of 512×512 pixels.The pilling severity is divided into five grades.From Grade 1 to 5, the pilling degree of fabric decreases gradually until almost no pilling is seen in Grade 5 [14] .Five samples selected from the datasets and graded according to American Society of Testing Materials (ASTM) standards were analyzed as standard samples as shown in Fig. 5.The subjective evaluation results were taken based on the evaluation results of five experts who compared each fabric with the standard samples.Those with the highest consistency were taken as the subjective evaluation results.

2D Depth Image Generation
In the feature extraction phase, the complexity of extracting 2D image features can be ignored compared with the 3D feature extraction which needs 3D point cloud processing.The 3D model can be projected onto a 2D plane with a determined mapping relationship [15] .The coordinates of the point s = (xyz(i)) in the depth image are converted to the pixel coordinates of the grayscale image.Eq. ( 13) converts the depth value z(i) into the grayscale value for the corresponding pixel.
where G(i) is the grayscale value of the 2D depth image, z m is the minimum depth value of 3D model, and H is the range of the depth value.

Pilling Segmentation
To make the pilling segmentation easier in the 2D depth image, fast Fourier Transform (FFT) algorithm [16] is used to filter the texture which is usually the patterns with a high degree of periodicity.FFT converts the image to the different frequency domains for analysis by seperating the texture information and pilling informa-tion as the high-frequency component and the lowfrequency component, respectively.The low-frequency component is highlighted by a Gaussian low-pass filter to eliminate the textured background.For each nontextured image, a global threshold is determined automatically using the adaptive iterative method which selects the average gray value of the whole image as the initial threshold and determines the optimal threshold by an iterative process.Figure 6 shows the processing results from sample image to binary image.

Pilling Evaluation
In the subjective evaluation method, the size, total area, and coverage of pills are the main factors that influence the expert evaluation [2] .In this study, the number and area are selected as the features for evaluating fabric pilling.In the binary image, the pilling number refers to the number of pill areas.The values of the pilling pixels are all 1, and the connected area is searched according to the 8-connected objects.The first connected area encountered is labeled as 1, then the search is carried out successively.The pill area is the total number of pixels counted in the detection area.It is worth noting that the total area of the pill is S total .
In this paper, the KNN classifier [17] was used to classify pilling samples and perform a 2-fold crossvalidation.The datasets are randomly divided into two subsets with equal numbers of the samples.One subset is used as a training set, and the other as a test set.KNN classifies the test sample by comparing the distance or similarity between the training samples and the test sample, and the K training samples closest to the test sample are found.K=3 is set in this experiment.The category with the highest frequency among the 3 points as the prediction category of the test sample, which is the objective evaluation result of the test sample.The accuracy P ACC of the objective evaluation is determined by comparing the results of the subjective evaluation.
where N M is the number of samples inconsistent between objective and subjective evaluation, and N T is the total number of samples.
As shown in Table 1, a total of seven samples are misclassified, which gives the system a classification accuracy of 92.8%.As shown in Fig. 7, a scatter plot shows how feature parameters relate to grade.The ordinate represents the pilling area, the abscissas represent the pilling number in samples, the point with different colors represent different pilling grades, and the misclassified samples have also been marked with red. Figure 7 illustrates that KNN is effective in classifying samples with small intra-class spacing and large inter-class spacing.

Conclusion
This paper proposed an effective way to objectively evaluate the pilling images based on PS.Self-developed image acquisition equipment is used to capture multiple images with different illumination, and then the semicalibrated near-light PS algorithm is used to reconstruct the fabric surface.The 3D model is then converted into a 2D depth image for texture filtering.The transformed non-texture image is segmented into a binary image by  the iterative threshold segmentation method, and the defined feature parameters of fabric pilling, including pilling number and area, were extracted.Finally, the KNN classifier was used to identify the fabric samples.The experimental results show that the system is effective and reliable for pilling evaluation.This method performs well for plain fabrics, but it is insufficient for patterned fabrics.The following work should focus on building non-Lambertian models.

Fig. 3 Fig. 4
Fig. 3 The results of corner point detection

Fig. 7
Fig. 7 Results of different evaluation methods