Three-dimensional hyperspetral camera based on near-infrared single-pixel imaging

This work presents the development of a multi-spectral vision system that combines different technologies detecting radiation in the visible spectrum (380-750 nm wavelengths), in the near-infrared NIR (1550 nm wavelengths), in the thermal spectrum band 1.4µm, and 77 GHz millimeter band. The main motivation of the this work is to propose a new hyperspectral approach to overcome the limitations present in commercial vision systems, such as the high amount of memory needed to generate images, the low quantum efficiency of silicon sensors, noise due to disturbance of the visible spectrum, limitations caused by foggy or low visibility conditions, and the difficulty to determine the depth information for objects in the image. Therefore, we propose a hybrid vision system that combines the capabilities offered by RADAR technology at the a range of tens or hundreds of meters, the ability to trace surfaces and operate under low visibility conditions, using a vision system based on single-pixel imaging (SPI) operating in the near-infrared NIR (@ 1550 nm) part of the spectra and Time-of-flight (ToF) based system using an InGaAs high quantum efficiency detector (QE(λ)>80 @ 1550 nm wavelength). Furthermore, SPI systems have a high intrinsic compression capacity at the hardware level, so less memory is needed for image generation, reaching PSNR values > 20 db (the PSNR level acceptable image quality in the industry is between 20 and 50 dB). Developing a system with these characteristics offers a high advantage for applications such as drones, autonomous vehicles, and other techniques that operate with 2D/3D vision or video systems. For the implementation of the system, a structured lighting SPI architecture was developed using an array of 32 x 32 NIR LEDs emitting radiation in the wavelength of 1550 nm that operates in combination with other technologies such as an 80 GHz millimeter band radar, a Time-of-flight system based on four pulsed lasers, a 32 x 24 thermal camera sensor, and a camera with northern vision capability. The developed vision system has an operating range of 37.5 mm to 5 m with a maximum spatial resolution of 10 mm in the distance range below 1 m. As a single-pixel image processing strategy, the Orthogonal Matching Pursuit (OMP) compressed sensing algorithm supported on a GPU architecture is used to achieve a processing times of below 30 ms, which corresponds to a 24 frame/sec video rate.

Future applications NIR-SPI hyperspectral system

The last decade has promoted different detection technologies adapted to different conditions in rainy, fog, or low-visible scenarios. The object detection under scattering conditions has converter in a challenge to which they have proposed some solution as use of the RADAR or LIDAR . A point important to considerer in a scattering condition is the elastic Mie scattering effects which in NIR spectrum decreases, for which a NIR-SPI system is a solution efficiency with low-cost and integration which in recent years have improved, reaching processing times close to real-time and capacity of 3D images. Therefore, these offer an advantage for its integration in systems perception using today in different vehicles, where it doesn't need to have a high-quality image for application as vision sensors detect and avoid obstacles, 2D/3D pre-mapping, or navigation applications.

Our NIR-SPI system developed if adopted in UAVs (see Fig. 1) offers a solution to perform better visualization of the scene in outdoor conditions. If compared to conventional vision systems that usually use RGB sensors operating in the visible spectrum. As explained, conventional systems operating in the VIS range present much higher image degradation in scattering conditions than those operating in the NIR part of the spectra. The integration of our NIR-SPI system into UAVs increases spectral capacity detection. In addition, it provides fast deployment at remote or challenging locations, being essential for many civil and military applications such as post-disaster relief assistance. Therefore, we believe that the NIR-SPI system presented has a great potential to be used in similar applications. Furthermore, SPI could be an alternative to the use of LIDAR technology.

Fig. 1. Future applications NIR-SPI hyperspectral system recues Drone

SPI applications are not limited to vision sensors, but new applications have also been developed. For example, in VR/AR systems, we can cite MEMSEye system that works using SPI and MEMS mirror to a generation of 3D images (see Fig .2b), and applications in holography . Others applications that are studied is adaption as vision system for ROVs or underwater, SPI-RADAR (see Fig.2c), and hyper-spectral . This later field of application has great importance in the area medical [241] (see Fig. 2d), the space exploration , and CubeSat.

Fig.2. Overview of new fields for applications system vision SPI: (a) SPI-LIDAR , (b) VR/AR , (c) SPI-RADAR , (d) SPI-medical.

Single-pixel image reconstruction using active near-infrared illumination in rainy conditions

RGB silicon-based image sensors operating in the visible (VIS) part of the spectra (with wavelengths between 400 and 700 nm) are highly susceptible to outdoor environmental conditions. They are forced to deal with scattering effects caused by the interaction of light with micrometer size particles in the rain, fog, or smoke, limiting the depth of visibility. Using longer wavelengths, e.g., near infra-red (NIR) radiation with wavelengths of around 1550~nm, would result in less light attenuation effects, eliminating most of Rayleigh scattering and leaving predominantly elastic Mie scattering effects, where the size of the scattering particles is comparable to the used illumination wavelengths. Motivated by the latter, this work proposes a vision system based on NIR active illumination and single-pixel imaging (SPI) using an array of 8 x 8 NIR-LEDs and a single Indium Gallium Arsenide (InGaAs) detector device with high detection efficiency for the wavelength range of interest. The proposed system uses a fusion of the super-resolution convolutional neural networks (CS-SRCNN) and the wavelet method, adapted to different Hadamard active illumination scan patterns for 2D image reconstruction.With the objective of evaluating the NIR-SPI 2D image reconstruction capabilities in scattering environments with rain, we carried out an evaluation of the quality of the reconstructed images using figures of merit such as Structural Similarity Index Measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR) in scenarios of laboratory that simulate different conditions of background illumination and droplet size to test its feasibility as a vision sensor.

---

SPIE Future Sensing Technologies 2021, Nov. 14, 2021.

https://doi.org/10.1117/12.2601118.

2D Near-infrared single pixel imaging (NIR-SPI) spatial resolution

In recent years, it had been an evolution in the sensors used by vision systems. Different technologies from RGB camera, IR-camera, LIDAR, RADAR, or sensor combinations can work in different spectrum bands. A new alternative to the vision system is the single-pixel camera (SPC), an architecture based on the compressive sensing theory for 2D reconstruction images. SPC's feature is used as the single detector that measures the linear combinations of pixels through patterns projection of structured light over the object that would like to photograph instead of using an array of pixels as in RGB sensors or cameras. These architectures offer the advantage that they can be adapted to different spectrum bands as near-infrared (NIR), and could operate in scenarios with low visibility or under rain, mist, or fog conditions. Currently, there is little information that discusses the problem of spatial resolution in SPC in scattering conditions for which we defined a methodology to evaluate the spatial resolution SPC's in terms of point spread function (PSF), and we analyzed how it affected the spatial resolution with the distance and under foggy conditions, for an SPC with active illumination at the wavelength at 1550 nm.

Fig.1. Spatial resolution using Hilbert Scanning at 20 cm , with separation between x=10 mm and y=10 mm ∆x=11.6 mm and ∆y=12 mm without Fog (a-b), and ∆x=16 mm and ∆y=18 mm with Fog (c-d).

---

SCATTERING EFFECT

When light interacts with a water particle in the air, various physical phenomena occur as reflection, refraction, absorption, and scattering. The scattering is related to the particle size (sp), defined as: sp=πD/λ , where D is the particle diameter, and λ the incident

Fig.2 .I ncident beam scattering characteristics for

different particles size: a) Rayleigh regime λ >> D,

b) Mie regime λ∼D and c) Ray optical regime λ << D

photon wavelength. This scattering process is respectively related to the size of the particle. For small

scattering particles (see Fig. 2a), the distribution of scattered light is symmetric in the forward and backward directions. If the particle size increases, the scattered energy increases in the direction of the incident beam diminishing the amount of scattered light in the opposite direction (see Fig. 2b, 2c).

The scattering cross-section Qd is proportional to the radiation-particle interaction probability, and it is defined through the so-called effective area of the collision. For which the total scattering of the impinging light is proportional to the effect cross-section (Mie theory) multiplied by the amount of the particles present in the medium. The scattering cross-section is here related tospof the inter-acting particle through the probability of extinction of the photon, also known as the extinction efficiency, Qext, the scattering probability Qsca, and the absorption probability Qabs.

---

FOG EFFECT IN THE SPATIAL RESOLUTION

In scattering scenarios, present changes in the level of contrast in the scenes that affect the quality of the reconstructed 2D image, and deducting spatial resolution image. In the case of scenarios with fog the level contrast is defined by Beer–Lambert law , where the contrast decreases by an exponential factor with distance defined by Eq.(1), with β as scattering coefficients Eq.(2), and depend on Qsc is the scattering coefficient (calculate using matlab ), the radius of particles r andDdensityis the density of particles suspended in the medium.

Therefore, as the measurement distance increases, the SPC's ability for discriminate objects is reduced due to this scattering effect.

Fig.3 .NIR-SPI test image circle of 50 diameter: At focal length distance d1=18 cm, and d2=30 cm , a-e) basic scanning, b-f) Hilbert scanning, c-g) Zig-Zag scanning, d-h) Spiral scanning.

Fig.4.NIR-SPI test image circle under foggy conditions of 50 diameter: At focal length distance d1=18 cm, and d2=30 cm, a-e) basic scanning, b-f) Hilbert scanning, c-g) Zig-Zag scanning, d-h) Spiral scanning.

---

https://ieeexplore.ieee.org/document/9975875