EmbeddedNeuroVision

Electronic vision and artificial intelligence (AI) already enable automatic entrance monitors and living rooms that safeguard the well-being of their occupants. By processing video data directly in the sensor (so-called “embedded AI”), it is possible to maintain individual privacy. However, complex applications using traditional AI in particular require significant computing power, leading to excessive power consumption, thermal dissipation issues, and high manufacturing costs for embedded hardware solutions. In many cases, real-time processing in applications to detect activities with temporal context is therefore not possible with embedded AI.

The application under consideration is a system that detects critical situations such as immobility, inactivity, and accidents in living spaces and can respond in an emergency by sending an alarm message to caregivers or relatives. The project partner Inferics currently offers PatronuSens, a solution based on classic cameras. In the project, so-called event-based cameras are used instead of classic ones as a solution approach. Event-based cameras output changes over time as a continuous data stream. This functional principle results in an inherent filtering of the observed area, where “uninteresting” is ignored and only image areas with activity are displayed. Aside from the fact that such event cameras alone require less energy than their classical counterparts, this filtering also reduces the amount of data that a subsequent AI algorithm must process. The aim of this project is to examine how such event-based cameras can be integrated into an existing image processing system, which system components have to be adapted for this purpose and what advantages their use actually yields.

The FZI is conducting research on hardware accelerators for spiking neural networks (SNN) and their use in real-world application scenarios. For this purpose, current advances in neuromorphic computing are studied, currently available event-based cameras are evaluated and neuromorphic AI algorithms developed.