The main objective of the project is to demonstrate the advantages of the MISEL holistic sensing and computing approach against state-of-the-art conventional approaches. This is realized by 1) elaborating, designing, and implementing the MISEL sensing and computing system with multi-band visual sensory front end, 2) identifying application scenarios that benefit from event-based approach, where the proposed system could become a natural tool, and 3) benchmarking. The result is a standalone multi-band vision system for advanced situation awareness.

To achieve this main objective, the project will develop all crucial components of the system, which leads to a set of sub-objectives:

  1. Demonstrate adaptive multi-band (VIS-to-NIR) pixels for the camera. The pixels are based on quantum dot/metal-insulator-graphene (QD/MIG) diodes monolithically fabricated on top of a silicon computing layer to work on multiple wavelengths.
  2. Demonstrate in-sensor computing for data reduction and adaptation Local computing drives sensor adaptation for signal enhancement, while event-based operation reduces output data stream.
  3.  Demonstrate dense FeRAM monolithically integrated on top of silicon computing layer, used for synaptic communication and plasticity.
  4. Explore and implement the MISEL holistic computing approach  In addition to novel devices (QD/MIG, FeRAM), the effectiveness of MISEL holistic approach stems from organizing computing hierarchically similar to biology:
    • near sensor processing (cellular sensor-processor) – sensors and processors form cells; neighbouring cells interact like in the retina;
    • fast spatio-temporal processing (cerebellar processor); spatio-temporal networks and hyperdimensional computing for on-line learning and inference
    • high-level processing with prediction capabilities (cortical processor), feedback to cellular and cerebellar processor
  5. Demonstrate the competitiveness of neuromorphic computing.   Map real-life use scenarios (e.g., related to sensory-motor systems) to the MISEL system and benchmark performance. Hardware and algorithms are co-designed to find the best possible trade-off between hardware complexity and performance metrics as to allow efficient implementation of a low-power edge device.

 

EU reference