Real-time Monitoring of 3D Printing Using Between-layer Structural Similarity (BLSS)

This research introduces an innovative method specifically designed for comprehensive error detection in the 3D printing process. The Between-layer Structural Similarity (BLSS) technique gauges the similarity between displacement maps, which are generated using the Structural Similarity Index Measure (SSIM). This allows for the differentiation between consecutive layers in both simulated and actual prints. Utilising the Fast Fourier Transform (FFT), the displacement maps can be converted to the frequency domain and assessed for similarity. The proposed approach involves three key processing stages: print simulation and processing, print capture and processing, and BLSS-based structural analysis and evaluation. This methodology not only reduces material wastage but also lays the foundation for automated systems capable of halting or terminating live printing processes upon error detection. Experimental results reveal that printing defects occur below 97.5 % of the similarity value, establishing this threshold as appropriate. This underscores the effectiveness of the system in error detection and its potential for real-time quality control in 3D printing.

produce intricate objects without the need for expensive tooling [1][2][3].Despite its advantages, 3D printing faces limitations such as slow build speeds and a relatively high rate of print failures.Many issues result from incorrect thermal settings, which can lead to problems like delamination, material under extrusion or over extrusion [4].Another major category of failures can be attributed to inexperienced users, improper mechanical setup, or incorrect print parameter configurations [5].In response to these challenges, researchers in recent years are developing 3D printing monitoring systems to mitigate the impact of print failures.Becker et al. [6] implemented microphones near the nozzle to monitor material flow by sound.This configuration faced challenges distinguishing the material flow sounds from background noise, which affecting the accuracy of the system.Aidala et al. [7] developed a contact sensor that employs a touch probe to compare the printed object outer shape with the simulated model.In contrast, the vision-based error detection systems have shown promise in various 3D printing processes, including material jetting and powder bed fusion.These systems often incorporate machine learning algorithms to self-calibrate printheads and offer closed-loop feedback for print correction [8].Vision-based error detection in 3D printing can be categorised into global and local approaches.
Global error detection is crucial in identifying various issues, such as geometric accuracy, thermal deformation or surface defects.Researchers have explored various approaches to achieve this goal, including the utilisation of single and multiple-view cameras for quality assessment by comparing simulated and actual printed layers [9], [10].Various camera view orientations, including orthogonal and angled camera positioning, have been tested.The most widely adopted experimental setup involves mounting a camera perpendicularly above the build plate of the 3D printer to enable a top-view perspective of the object.Other researchers have concentrated on adopting a perpendicular side view approach to detect errors that were previously difficult to identify using the conventional top view [11].Typically, these setups involve the use of binary template matching from the current layer or a 2D view of the object with simulated images [12].The primary focus has frequently centred around the identification of external shape errors.Petsiuk and Pearse [13], developed a setup with an angled view, allowing them to identify the shape of the printed object using edge detection and match it to a template, which could also measure object height accurately from this perspective.While single-camera systems can detect significant print failures, there are distinct advantages to implementing multiple-camera systems, despite the synchronisation challenges they may introduce.Multiple-camera systems provide comprehensive monitoring by capturing images from various angles, enabling them to detect a broader range of potential issues.It is worth noting that concavity remains a challenging limitation [14], [15].Local error detection plays a critical role in monitoring material flows and analysing specific segments of parts during the 3D printing process.Unlike global approaches that examine the entire object, local methods focus on individual components and localised issues.Specifically, the system primarily concentrates on monitoring the material flow from the nozzle and detecting any overlaps with previous printed lines.This localised approach ensures that potential errors and discrepancies in material deposition are detected and addressed promptly, contributing to improved print quality and accuracy [16], [17].
Neither global nor local error detection methods are flawless.Global approaches often overlook subtle, layer-specific inconsistencies and depend on camera perspectives that might fail to capture intricate errors due to their limited viewpoints.Conversely, local methods may be too narrow in focus, potentially neglecting errors that impact the overall structure.In response to these limitations, this paper proposes a Between-layer Structural Similarity (BLSS)based method to address the challenges of error detection.The objective is to pinpoint discrepancies between consecutive layers in both simulated and actual prints.To enhance accuracy in similarity quantification, the calculated displacement maps undergo transformation into the frequency domain.This innovative approach shows promise in enhancing the quality and efficiency of 3D printing processes.
In the rest of the paper, the mechanical setup, the preparatory process and the methodology are introduced in Section 2, and it is followed by presenting the experimental analysis and a precise classification of detectable errors in Section 3. Finally, the concluding remarks are given in Section 4.

METHODOLOGY
The proposed methods consist of three main processing stages, including the print simulation and processing, print capture and processing, and BLSS-based structural analysis and evaluation.In the print simulation and processing, the system generates a simulated image from G-code to preview the expected outcome.During the actual print capture and processing stage, it captures real-time snapshots of the print, enhancing and isolating the printed object in images for the further analysis.In the BLSS-based structural analysis and evaluation, the processed images are used to generate the displacement maps between consecutive layers in both simulated and actual prints, and it is followed by the similarity measure of the displacement maps.Figure 1 illustrates the work cycle of the proposed method.

System Configuration
The research utilised a Creality Ender 3 printer, selected for its affordability and dependable performance.To facilitate smooth communication, it was linked to a Raspberry Pi 2b model, enabling the integration with OctoPrint.Image capture was conducted using a standard web camera with a resolution of 640x480 pixels.The computational setup featured a computer equipped with an i7 processor, 32GB of RAM, and an 8GB video card.The proposed methodology was implemented using MATLAB.

Print Simulation and Processing
Generating a simulation of the 3D object involved the utilisation of a G-code file.This file, crafted by a slicer program, encompassed a textual compilation of printer settings and movements.This textual information served as the basis for the physical reconstruction of the object using a 3D printer or for creating a virtual simulation in MATLAB.Top views of a single layer simulation and a 3D object simulation are shown in Figure 2.
To facilitate structural analysis, images containing simulated prints were initially cropped to align with the build plate size of the 3D printer utilized in this project, ensuring uniformity for subsequent measurement comparisons.Following the cropping process, a thresholding technique was implemented to eliminate the background and isolate the object of interest.Manual adjustments to threshold values, considering colour space and setting minimum and maximum values for all three corresponding colour channels, were made to ensure standardized segmentation quality.This step proved critical for maintaining reliability in comparisons and analyses throughout the study.Subsequently, the thresholded images were converted to grayscale to enhance computational efficiency in the structure analysis.To further refine the images, a median filter was applied to the grayscale images in order to further refine the image including reducing noise and smoothing overlapping edges.Figure 3 demonstrates the above workflow for processing an image of simulated print.

Print Capture and Processing
Instead of the conventional black build surface, a yellow Polyetherimide (PEI) sheet was positioned on the silver build plate.The choice of the PEI sheet colour proved advantageous for more effective thresholding and background subtraction.Synchronisation between the top-down camera view and the print control system was crucial for ensuring precise and consistent image captures.This alignment guaranteed that each image was captured with the object and the camera in identical positions, significantly simplifying subsequent processing and analysis.This alignment was particularly crucial for accurate layer subtraction, as it ensured to capture the differences occurred from layer changes rather than shifts in positioning.This, in turn, reduced computational complexity and enhanced the reliability of the analysis.The setup for print capture is shown in Figure 4.
Similarly, to the procedure for processing the image of simulated print, the image containing the actual print was handled through a sequence of steps, including distortion correction, colour thresholding, grayscale conversion, and median filtering.The camera calibration was conducted to estimate the camera parameters and it is followed by the image distortion correction using the estimated camera parameters.A semi-automated background subtraction process was then applied to the undistorted image in which the manual thresholding values were set for the first layer, specifying the colour space and establishing minimum and maximum values for all three channels individually.These specified values masked the background in subsequent images throughout the print process.Both grayscale image conversion and median filtering processes were the applied to the masked images.The workflow for processing an image of actual print is illustrated in Figure 5.

BLSS-based Structure Analysis
Having both simulated and processed, and actual prints captured and processed layer frames, a structure analysis can be applied to determine the similarity between them.The structure analysis uses a processing pipeline that includes the Structural Similarity Index Measure (SSIM) calculation, image cropping, Fast Fourier Transform (FFT) and Normalised Cross-Correlation (NCC), as illustrated in Figure 6.
Comparing images directly poses challenges due to notable differences in colour intensity, brightness, background, and image distortion between simulated and actual prints.Such disparities, as illustrated in Figure 7, can lead to inaccuracies in similarity measurements, as acknowledged in studies conducted by Schindler et al. [10] and Aburaia et al. [12].To address this issue, an alternative approach is proposed, involving the use of displacement maps from consecutive layers in both simulated and actual prints.This method allows for a more reliable comparison of structural differences, unaffected by other substantial disparities.The study uses the result of the SSIM calculation for the similarity measure, designed to assess perceived changes in structural information between images.The SSIM considers three key image components -luminance, contrast, and structure -to encompass various aspects of image quality such as brightness, sharpness, and texture [18].
Following the computation of SSIM, a displacement map is generated to illustrate structural differences between consecutive layers, as depicted in Figure 6.Subsequently, this map is cropped to isolate the area of interest corresponding to the layers.Applying FFT and NNC to the cropped map transforms it into the frequency domain  and assesses the similarity between the maps of the same layer in both simulated and actual prints.The utilisation of FFT is advantageous because the displacement map can be represented in terms of frequency components.This representation offers a more intuitive depiction of texture and structure information based on its frequency content, facilitating precise comparisons [19].NCC evaluates similarity by comparing frequency-transformed maps, considering how patterns in one map correlate with those in the other.It proves effective even for subtle differences.The resulting metric yields a scaled value between -1 and 1, where 1 indicates a perfect positive match, and 0 signifies no match [20].
The robustness of the BLSS-based structure analysis demonstrates its strength from its capability to identify errors that posed limitations in previous research, such as detecting layer shifts on a pyramid or identifying holes in a bridging surface.These errors are documented in Table 1, along with the corresponding simulated layers and the actual structural differences, presented as a percentage to indicate the similarity.The layer shift, artificially created and measuring only 2mm along the Y-axis, ensures it remains within the outer contur of the object.This demonstrates how the method effectively addresses challenges related to identifying errors in objects where cross-sections progressively reduce in size.Another advantage of this method is that if the user decides that a detected error is not a critical failure and chooses to continue the print, the previous error will not be flagged in subsequent layer  structure analyses.This ensures that the system accommodates user preferences and allows for the continued printing process.

EXPERIMENTAL RESULTS
To comprehensively evaluate the effectiveness of the proposed method, a selection of objects was chosen for experimental testing.This set included a cube to represent simplicity, a pyramid to introduce angular complexity, and the intricate details of a 3D Benchy model, as illustrated in Figure 8.Each object underwent testing with a diverse colour spectrum to evaluate sensitivity and accuracy across various potential printing scenarios, ensuring a thorough assessment of performance.This led to a total of 30 test prints.
In some cases, intentional modifications were made to the Gcode to induce printing faults.This purposeful alteration aimed to evaluate the capacity of the system to identify common printing defects, such as layer shifts, missing layers, extrusion errors, and inaccuracies in print detail.Upon analysing the outcomes from the 30 test prints, a preliminary accuracy threshold of 97.5% was established to differentiate between correctly printed layers and printing errors.This threshold is subject to further refinement through additional testing.Figure 9 provides a visual representation of the similarity measurement using the proposed BLSS-based method across various print tests that feature print defects, including blobs, bridging errors, stringing, build plate delamination, spaghetti, layer shifts, and lost feature [2], [21][22][23].It is evident that the similarity values corresponding to the print layers with defects significantly fall below the established threshold boundaries.Figure 10 shows an example of BLSS-based similarity measurement in details.In this figure, the yellow areas represent artificial defects, while grey areas signify regions without defects.The drop in similarity values below the defined threshold is clearly visible in the two artificial areas.However, there is another drop under the threshold around Layer 37 of the print, where a visible bridging error caused by the printer itself occurred.This error is clearly reflected in the similarity measure, demonstrating the ability of the method to detect both artificial defects and real printing errors.
Another significant advantage that is clearly visible and has been explained in the Methodology section is the capacity of the method to minimize redundant notifications, which contributes to its overall effectiveness.Furthermore, this method effectively addresses a critical issue by minimizing unnecessary notifications for errors, ensuring that notifications are only triggered when there is a significant deterioration in error severity.This practical feature proves particularly useful when operators anticipate error recovery.

CONCLUSION AND FUTURE WORK
This paper introduces an innovative BLSS-based approach for realtime monitoring of 3D printing.The core method involves calculating structural similarity between consecutive layers in both simulated and actual prints, achieved through the process pipeline of SSIM, FFT, and NCC.Experimental results showcase the effectiveness of the method in detecting print defects caused by various factors, offering potential improvements to the overall quality and efficiency of the 3D printing process.Future efforts will prioritize exploring optimizations to enhance the robustness of the method.
In the course of this research, several avenues are being considered for enhancing the robustness and efficiency of the BLSS method in future work.One of the primary objectives under consideration is the optimization of error identification by repositioning the camera to different angles, allowing for the determination of optimal viewpoints for error detection.Additionally, the integration of a nozzle-mounted camera for real-time monitoring of material flow and overlapping is being explored, enabling immediate error identification during the printing process.Furthermore, the research is currently investigating the incorporation of machine learning methods.The goal is to establish a robust model using extensive training data.This approach aims to enhance the BLSS method ability to identify deformation and defects.While these advancements may increase calculation time, efforts are planned to optimize the algorithm to ensure efficient processing.To enhance statistical reliability, an increase in the repetition times of experiments is also planned, aligning with the suggestion to improve experiment reliability.These ongoing efforts aim to improve the efficiency and stability of 3D printing and enhance error detection's overall quality and accuracy.The valuable suggestions from the reviewers are appreciated, and the commitment to implementing these enhancements in future work remains.

Figure 1 :
Figure 1: Work cycle of the proposed method

Figure 6 :
Figure 6: Proposed BLSS-based approach for structure analysis and evaluation

Figure 10 :
Figure 10: Similarity measurement for the print process with both artificial and actual defects

Table 1 :
Examples of BLSS-based error detection