Analysis of modern learning-based multimodal fusion algorithms for neuromorphic vision sensors

Hamad AlRemeithi; Issac Niwas Swamidoss; Abdulla Al Mansoori; Abdulrahman AlMarzooqi; Slim Sayadi; Tarek Bouamer

doi:10.1117/12.2653210

22 May 2023 Analysis of modern learning-based multimodal fusion algorithms for neuromorphic vision sensors

Hamad AlRemeithi, Issac Niwas Swamidoss, Abdulla Al Mansoori, Abdulrahman AlMarzooqi, Slim Sayadi, Tarek Bouamer

Proceedings Volume 12327, SPIE Future Sensing Technologies 2023; 1232722 (2023) https://doi.org/10.1117/12.2653210
Event: SPIE Future Sensing Technologies, 2023, Yokohama, Japan

Conference Poster

Abstract

Image enhancement is an ongoing research problem that the community is addressing through the development of fusion algorithms. Such techniques typically involve the reconstruction of RGB images by removing environmental artifacts and enhancing desired features. Infrared imagery is also widely used to improve situational awareness in low visibility scenarios. Recently, learning-based approaches are used for fusion purposes to extract meaningful representations from images and capture latent features that could otherwise be inaccessible using conventional image processing algorithms. The inadequacies of RGB images in these algorithms’ pipelines are still obvious, despite the fact that the viability of RGB-Infrared image fusion has been thoroughly demonstrated in the literature. For example, RGB images often have artefacts like sudden changes in exposure or motion blur when the illumination changes or sudden changes in the scene. A novel imaging sensor operating in the visible light spectrum has been developed to address these issues. In this paper, we explore the cutting-edge paradigm of Neuromorphic Vision Sensors (NVS), a class of asynchronous analog imaging platforms that operate based on the change of pixel luminosity within a scene. When compared to frame-based counterparts, NVS enhances scene interpretation, processing time, reaction time, and power consumption. Deep-learning reconstruction networks are evaluated in this study to determine the applicability of existing state-of-the-art multi-modal image fusion techniques with the addition of NVS data rather than RGB data. As a benchmark, metrics such as signal-to-noise ratio (SNR) and pixel wise error are used.

Citation Download Citation

Hamad AlRemeithi, Issac Niwas Swamidoss, Abdulla Al Mansoori, Abdulrahman AlMarzooqi, Slim Sayadi, and Tarek Bouamer "Analysis of modern learning-based multimodal fusion algorithms for neuromorphic vision sensors", Proc. SPIE 12327, SPIE Future Sensing Technologies 2023, 1232722 (22 May 2023); https://doi.org/10.1117/12.2653210

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available