DE BRUIJN STRUCTURED ILLUMINATION STUDYING WITHIN THE TASK OF RESTORING HANDS RELIEF

In the course of studies on the problem of restoring hands relief, using the de Bruijn structured illumination, methods of solving this problem are proposed. This is a method of simple quantitative detection of Hough segments on the skin of the hand, a method of qualitative visual evaluation of the effectiveness of the color palette using the dominant color, and a method of the weight coefficients of the components of the color palette. The proposed methods make it possible to quantitatively determine the optimal choice of the color scheme for generating the de Bruijn bands when illumination of the hand, to restore its relief. The work describes the stages of this study, led from visual observation to a full quantitative calculation of the quality of calibration illuminations, with the possibility of their optimal choice. In the course of experiments and observations, the requirements for the technical support of research were developed to achieve the best quality of the images of the hands. Also, the paper presents a high-speed de Bruijn sequence generating algorithm using Lyndon’s words, which excludes the search for Euler chains or Hamiltonian cycles, for various kinds of de Bruijn graphs. With its help, the generation of structured light patterns with various color schemes was carried out, with the purpose of further analysis of their use in 3D reconstruction systems of hands.


Introduction
To date, in the problems of scanning volumetric surfaces, many different variants of patterns of structured illumination have been developed.Most often, in practice, a series of changing pictures (pictures with time multiplexing) and unchanging pictures are used, using various color-coding options.
In the present study, to detect the relief of the hand, colors with color coding based on M-sequences or de Bruijn sequences are used.
De Bruijn sequence of order m in the alphabet of n symbols is a circular sequence of length n^m that contains any substring of length m only once.This sequence is used, in particular, in systems for recording 3D objects in real time (3D scanners) using structured illumination.
The aim of research is studying the behavior of the de Bruijn bands on the hand, with the possibility of choosing the optimum color palette for structured illumination using the weighting method.This will allow the most qualitative restoration of the three-dimensional relief of the hand for further use in 3D scanning systems, in particular, in systems for recognizing gestures.Since, as a result of correctly selected color sets in the illumination patterns, minimal mutual influence of the components of the entire color scheme is provided and overall recognition accuracy is increased.
The main task of the image processing algorithm is determination of the correspondences between points on the registered image, and points on the illumination pattern.The result of the work of algorithms is a cloud of points in three-dimensional space, on which a three-dimensional Computer Sciences visualization of the surface of an object is constructed.The structure of the processing algorithm is determined by the type of used template [1,2].
The implementation of the illumination used in the work is carried out using the Lyndon words, since there is a direct link between the de Bruijn sequences and the Lyndon words.The software implementation uses recursive and iterative algorithms to generate such words (roughly the same performance).The result of the work is the generation of a pattern of structured illumination for the tested color schemes [3,4].
In many image processing systems, it is necessary to solve the problems of searching or identifying objects of a given form: straight lines, circles, ellipses, etc., in real time [5,6].The relief of a given surface in real time can be obtained only using pictures of structured illumination with spatial or color coding, which can't be achieved with the use of pictures with time multiplexing.
In the course of the study, color coding was chosen.So, as a criterion of optimality of the color scheme of illumination, the number of detectable objects (color segments) on the hand, obtained with the help of Hough transforms (HoughLinesP function from the OpenCV library) was chosen.Obviously, such criterion is not optimal, since the effectiveness of using the selected illumination is determined visually -these are segments of a certain color unfilled by Hough segments.Therefore, it was used as an auxiliary, when choosing the initial color schemes, for further quantitative analysis [6,7].
Thus, a method was developed for calculating the weight coefficients of colors and their percentage losses, described in detail in the study.The application for the study is implemented in the C# programming language in the Microsoft Visual Studio 2015 environment, using the Emgu-CV library (OpenCV for .NET).
At the moment, working with the OpenCV library is well documented, there are many examples and practical advice [7,8].
The practical value of this research consists in the possibility of choosing the optimal color scheme of structured illumination, for the most effective investigation and restoration of the relief of the hand.

Review of the problem state
Detection and analysis of bands of structured illumination by color is an accurate and simple way of obtaining image depth (surface relief) [9−13].
There are many color patterns developed for the rapid identification of the relief of the object under study, achievable in one video frame.
Structured illumination methods used in modern 3D scanners and suitable for real-time operation use finite color sequence templates.To increase the resolution of the resulting point cloud (the quality of the result), an increase in the number of unique subpatterns in a single projection pattern is required [14,15].
Fig. 1 shows the de Bruijn sequence pattern based on the RGBCMY color alphabet, with a length of a non-repeating subsequence of 3.

Fig. 1. De Bruijn sequence of 216 characters in length (color bands)
Methods for generating de Bruijn sequences, using various algorithms, are widely represented in many sources [16][17][18].

Computer Sciences
Color coding uses only one picture.The position of each pixel, uniquely, is encoded by the color value of the given pixel and several of its "neighbors".When creating a color-coded picture, it is tend to get the minimum size of the neighborhood (the number of "neighbors") of the pixel required to uniquely restore, and the minimum number of different colors (to increase the reliability of determining each color).It is precisely these properties that the M-sequences or de Bruijn sequences have realized in the study using Lyndon words [19].

Materials and methods
In accordance with the logic of scientific research, the research methodology is developed.It is a complex of theoretical and empirical methods, the combination of which makes it possible to study, with the greatest certainty, the object of behavior of the de Bruijn bands of structured illumination on the hand (the specific structure and color of the skin) in order to select the optimal color scheme for restoring its relief.
In the course of solving the problem, studying the behavior of the de Bruijn bands on the hand, a number of methods have been applied that made it possible to comprehensively study the investigated problem, all its aspects and parameters.Among them let's select methods of theoretical research (abstraction, analysis and synthesis) and empirical research.
The empirical study consisted in observing the investigated object, without being influenced by the observer, under various conditions of illumination and background.In the end, this led to the conclusion that it was necessary to use an absolutely black color for it, which made it possible to improve the contrast as much as possible.
In the course of the experiments, a 3D scanning machine was used.The choice was made in favor of a mini projector Samsung H03 (South Korea) and web cameras Logitech HD Pro 920 (China).The use of devices provides both the most acceptable illumination of the investigated object, and the excessive resolution of the resulting video frame in the process of video capture.
Experiments with the resulting set of images, OpenCV tools, allowed to choose the optimal parameters and threshold values for maximally complete and high-quality processing.Below are the values of such parameters.Accordingly, the modeling method was used to create and study a copy (model) of the object that imitates the original.A comparison method was also used, which, at this stage, allowed to abandon the use of some color schemes.

Experiments
The developed Windows application allows to load images of a hand made in approximately the same conditions (the known corners of the projector-camera system, the distances between them) with different illumination (color scheme).
To obtain the maximum number of Hough sections, empirical studies were conducted, during which the optimal set of tools that exist in the OpenCV library was determined.For Hough transformations, threshold values were used, specially selected for the existing set of images.
Initially, the image is smoothed (or blurred) using the GaussianBlur function, which blurs by calculating a convolution of an image with a discrete Gauss core with standard deviations equal to sigmaX and sigmaY along the Ox and Oy axes respectively (the deviations are 2).
It should be noted, without going into details and terms, that when calling this function, a restriction on the parameter kSize is imposed (empirically selected value = 3).The width and height of the core must be positive and odd, or zero if the core size is determined from standard deviations.Also, a median filter with a core size of kSize=7 is used.
The threshold values for the Kenny boundary detector are empirically chosen.The discrete Sobel differential operator, which calculates approximate values of derivatives of different orders for the brightness function of pixels, uses a second-order derivative of X (in Y, there is no need to differentiate, since vertical and close segments are detected).
Detection of segments is performed using the application's public LineSegment2D GetVer-ticalLines (Bitmap bmp) method, which calls the library function CvInvoke.HoughLinesP.Basic operations of working with images are described in many sources, and do not require additional explanations [20].

Fig. 2. Algorithm for processing the original image
The GetVerticalLines method defines a set of vertical lines in the image, using the optimal empirical parameters transferred to the designer, for filters and boundary detectors.
Some fragments of the text application are shown in Fig. 3.

Fig. 3. Fragment of the test application
It should be noted that the developed application is able to detect hand with the help of existing trained Haar cascades used in pattern recognition.For more accurate localization of the hand, it is possible to select ROI (the area of interest) with the help of mouse.

Results
The results of detecting segments on the hand, using the Hough transforms for various de Bruijn illumination (black segments), are shown in Fig. 4, 5.The visual analysis allows to leave just these palettes for further research.

Discussion of research results
The essence of the approach is the determination of the dominant color from the illumination palette on the skin of the hand.
The illumination palette on the skin of the hand is shown in Fig. 6.
By ROI scanning (in this case, the area of the hand), let's analyze the color composition of each pixel and determine the dominant color, for example, for red color.The algorithm for the ROI scanning of the hand region is shown in Fig. 7.
Also, with the formation of lists of pixels for each color from the illumination panel, let's determine their number to calculate the specific weighting coefficients.
For visual control, the dominant color is displayed in "clean" color.The definition of the dominant color of the RGB illumination in quantitative terms is shown in Fig. 8. Weighting factors for RGB illumination are presented in Table 1.The mutual influence of colors entering the palette can be investigated by choosing the appropriate ROI (in the limit of a single color band).Fig. 10 shows that the color shade of magenta, in practice, is not subject to such influence.In Fig. 13, 14 it is possible to see the absorption of green and blue colors in red in RGB illumination, as well as yellow and cyanic magenta (CMY).
The processing of the file statistics for each palette gives the results shown in  Thus, changes in the color of the illumination on the hand indicate that the color loss in the CMY illumination is less (do not exceed 15 %), for RGB illumination they are more than 25 %.

Conclusions
1.A review of the existing complexes of theoretical and empirical methods is performed, the combination of which makes it possible to study the object under study with the greatest certainty by analyzing the behavior of the de Bruijn bands of structured illumination on the hand, in order to select the optimal color scheme for restoring its relief.
2. To obtain the maximum number of detectable Hough segments, empirical studies using a 3D scanner are carried out, during which the optimal set of tools that existed in the OpenCV library is determined.For proper Hough transforms, empirically selected threshold values are used.
3. As the statistics on color loss gradually accumulated, and the results were fixed, we can make an unambiguous conclusion in favor of using CMY illumination for the hand, which is optimally suitable for these purposes.
The obtained results help to choose the optimal color scheme of the structured illumination, in order to most effectively illuminate the hand to restore its relief.There is also a wide opportunity for experimenting with all sorts of other color schemes for such illumination.It should be noted that in many cases the advantages of such scheme can be assessed by visual analysis.

Fig. 8 .Fig. 9 .
Fig. 8. Determination of the dominant color of RGB illumination in quantitative expression

Fig. 14 .
Fig. 14.Change of the palette color on the skin of the hand (RGB)

Table 1
Weighting factors for RGB illumination of the hand

Table 2
shows the weighting coefficients for CMY illumination.

Table 2
Weighing factors for CMY illumination of the hand

Table 3
Change of CMY illumination color on the hand

Table 4
Change of RGB illumination color on the hand