11 Sep The Road to Median Filter Mastery — Part 1

This article is part one of a two-part examination of median filters, why they are used, how they operate and some guidelines for their effective use. Part 1 focuses on low-pass median filters. Part 2 focuses on high-pass filters.

Median Filters, Why are They Important

Median filters are a method used to reduce unwanted signals, better known as noise. Effective filtering reduces as much of an unwanted signal, while leaving as much of a desired signal, as possible. Non-destructive testing (NDT) inspectors cannot presume to know all the signals they are looking for in advance. It is therefore necessary to be very aware of what desired signals—target defects—the median filter might remove. So, you must always use median filters with the utmost care and understanding their functions thoroughly. It’s advisable to review filtered and unfiltered data to make sure nothing is missing.

What to Filter?

Median filters are very effective means of filtering two basic types of noise: high-frequency noise, often called electrical noise or spiking, and low-frequency noise or drift. High-frequency noise can be electrical noise entering the data stream from external sources. It can also be spiking from electrical or electronic issues inside the probe. Finally, it can be from instrument and natural signal changes associated with material variations or deposits on the test piece itself. One very common source of high-frequency noise is over-amplification, such as gain, beyond the optimal settings for a given technique.

Low-frequency noise can be caused by the liftoff variations encountered by the probe inside the tube or over the test surface, variations in deposit conditions on the surfaces of the material, or changes in material, composition, geometry, or thickness.

Median Filters, What They are and What They do

Median filters perform digital signal processing and come in two types: low-pass and high-pass. A low-pass filter allows low-frequency signals to pass (filters high-frequency noise) and a high-pass filter allows high-frequency signals to pass (filters low-frequency noise). This can be a bit confusing as these names describe what is unfiltered rather than what is. Another way to distinguish the two is to remember that a low-pass median filter is most often used with a “low” number of data points and a high-pass median filter is most often used with a “high” number of data points.

The word “median” is a mathematical term used to describe the “middle” of a sorted data set. For example, in Table 1, where there are 11 samples in the data set, the middle value would be found at sample number 6 and would be equal to a value of 20. Figure 1 shows a line graph of the data presented in Table 1.

Table 1 — Example data set with 11 data points

Table 1 (part 1)

Figure 1 (part 1)

Figure 1 — Line graph of the data presented in Table 1

Low-Pass Median Filter

This filter returns the median value of a sorted set of values centered about the current data point with the number of data points contained in that set being defined as the width of the filter. Although this sounds complicated, it isn’t. To illustrate how this filter works, read the definition again looking at the data shown in Table 2 and Figure 2.

Table 2 — Low-pass median filter example data set with 33 data points

Table 2 (part 1)

Figure 2 (part 1)

Figure 2 — Line graph of the data presented in Table 2

As you can see in Figure 2, one data point in the data set has a higher value than the rest. Assume that this higher value is a noise signal and that a low-pass median filter will be used to reduce its influence.

  1. Filter width – the low-pass median filter in this example will use five data points (more about choosing the width of the filter later).
  2. Centered about a current data point — center the filter about point 17 so that the data runs from data point 15 to data point 19.
    Centered about a current data point
  3. Sorted values — take the five data points from data point 15 to data point 19 and sort them from smallest to largest. Note that the original data point numbers are no longer tracked.
    Sorted data points
  4. The median value of this set is the middle number in the row, which in this case, is 2. Therefore the value of 2 becomes the low-pass median filtered value that will replace the original value at data point 17.
  5. This process is repeated for all the data points, the results of which appear in Table 3 and Figure 3. Note that the filtered data follows the original data closely, but does deviate slightly on a point-by-point basis. Also note that the filtered data is shorter by two data points at each end.

Table 3 – 5 data point wide low-pass median filter of the data from Table 2

Table 3 (part 1)

Figure 3 (part 1)

Figure 3 – Line graph displaying the original and 5-point low-pass median filtered data

The low-pass median filter is most effective at removing spike-like noise from data streams and has a minimal impact on data integrity, especially if the the filter is as narrow as possible. The low-pass median filter filters out spike signals regardless of their amplitude, where an averaging filter retains some traces of a large amplitude spike signal, especially when the spike voltage is very large.

An example of a low-pass median filter reducing the influence of electronic noise appears in Figure 4, where the upper C-scan and Lissajous are from the original, unfiltered data and the lower ones have had a low-pass filter applied.

LP filt 01 cropped

Figure 4 — High-resolution Magnifi® C-scan of calibration tube data showing unfiltered and filtered outputs

Take special care when using low-pass median filters as there’s always a danger of attenuating the signal (amplitude reduction), but a certain degree of signal attenuation is normal. You can see different signal attenuation for various low-pass median filter widths in the three C-scans in Figure 5. The signal shown is from a 0.1 mm wide EDM notch. The noise adjacent to the notch signal was measured and found to be approximately 0.20 volts peak-to-peak (Vpp). The unfiltered signal was measured at 1.93 Vpp (Figure 5a). Assuming that the noise is added to the signal, it’s reasonable to assume that the flaw could be as small as 1.73 Vpp (1.93–0.20 Vpp).

Figure 5 (part 1)

Figure 5 — High-resolution Magnifi® C-scans of a 0.1 mm wide EDM-notch signal with a) no filter, b) a 3-point low-pass median filter, and c) a 17-point low pass median filter

Applying a 3-point low-pass median filter reduces the amplitude of the signal to 1.83 Vpp (Figure 5b). Similarly, the attenuated amplitudes using filter widths up to 17 data points (Figure 5c) were recorded and charted in Figure 6.

Figure 6 (part 1)

Figure 6 — Chart of signal attenuation with increasing low-pass median filter width

It was therefore concluded that a low-pass median filter 3 or 5 data points wide was acceptable for this application (the amplitude of the signal was not attenuated below the arbitrary limit of the original signal amplitude minus the approximate amplitude of the noise being filtered). Note that every situation is different. A wider low-pass median filter will remove more noise, but can also attenuate more signal. The degree of attenuation is a function of the inspection speed, the length of defects, and the filter’s width. The maximum width of a filter should always be ascertained using calibration samples before it’s used on inspection data. If you’re unsure, always err on the side of using a narrower low-pass median filter.

The Takeaway

Low-pass median filters are an effective means of mitigating short-duration noise events such as spiking, electronic noise, and surface conditions. Low-pass filters should be used with as narrow a filter as possible and the degree of attenuation of a filter should be assessed on known defects before use. Data should always be examined in its filtered and unfiltered states.

Part 2 of this series examines high-pass median filters.

Categories

Eddy current array (ECA) Electromagnetic testing Hack NDT Power generation Surface inspection Tubing inspection

comments powered by Disqus