The main objective of the project is to demonstrate the advantages of the MISEL holistic sensing and computing approach against state-of-the-art conventional approaches. This is realized by 1) elaborating, designing, and implementing the MISEL sensing and computing system with multi-band visual sensory front end, 2) identifying application scenarios that benefit from event-based approach, where the proposed system could become a natural tool, and 3) benchmarking. The result is a standalone multi-band vision system for advanced situation awareness.
To achieve this main objective, the project will develop all crucial components of the system, which leads to a set of sub-objectives:
- Demonstrate adaptive multi-band (VIS-to-NIR) pixels for the camera. The pixels are based on quantum dot/metal-insulator-graphene (QD/MIG) diodes monolithically fabricated on top of a silicon computing layer to work on multiple wavelengths.
- Demonstrate in-sensor computing for data reduction and adaptation Local computing drives sensor adaptation for signal enhancement, while event-based operation reduces output data stream.
- Demonstrate dense FeRAM monolithically integrated on top of silicon computing layer, used for synaptic communication and plasticity.
- Explore and implement the MISEL holistic computing approach In addition to novel devices (QD/MIG, FeRAM), the effectiveness of MISEL holistic approach stems from organizing computing hierarchically similar to biology:
- near sensor processing (cellular sensor-processor) – sensors and processors form cells; neighbouring cells interact like in the retina;
- fast spatio-temporal processing (cerebellar processor); spatio-temporal networks and hyperdimensional computing for on-line learning and inference
- high-level processing with prediction capabilities (cortical processor), feedback to cellular and cerebellar processor
- Demonstrate the competitiveness of neuromorphic computing. Map real-life use scenarios (e.g., related to sensory-motor systems) to the MISEL system and benchmark performance. Hardware and algorithms are co-designed to find the best possible trade-off between hardware complexity and performance metrics as to allow efficient implementation of a low-power edge device.