FPGA-Based Hardware Architecture for Contrast Maximization in Event-Based Vision

Technical Summary: FPGA-Based Hardware Architecture for Contrast Maximization in Event-Based Vision

Problem Statement
Event-based cameras (Dynamic Vision Sensors) offer significant advantages over traditional frame-based sensors, including high temporal resolution, low latency, and high dynamic range. However, processing their sparse, asynchronous data streams presents challenges. The Contrast Maximization (CM) algorithm is a robust method for estimating motion parameters by maximizing the variance of an Image of Warped Events (IWE). While effective, CM relies on iterative optimization to find motion parameters that best align event edges. This process is computationally intensive, involving complex warping, bilinear voting, and gradient calculations. Consequently, existing CPU and GPU implementations often struggle to achieve true real-time performance, particularly in high-speed, low-power embedded applications, due to high power consumption and latency.

Methodology
The authors propose a dedicated hardware architecture implemented on Field-Programmable Gate Arrays (FPGAs) to accelerate the CM algorithm. The methodology is structured around a deeply pipelined, massively parallel design:

1. Event Preprocessing and Warping: Input events are filtered based on a Region of Interest (ROI). A reference time ($t_{ref}$) is calculated as the midpoint of an event batch. Events are then warped to this reference time using a constant translation motion model ($v_x, v_y$), adjusting spatial coordinates based on the time difference ($\Delta t$).
2. Bilinear Voting and IWE Construction: To handle the fact that warped events often fall between discrete pixel locations, the architecture employs a bilinear voting mechanism. Each event contributes weighted values to four neighboring pixels based on fractional coordinates.
3. Parallel Accumulation: To support high throughput, the memory architecture utilizes 12 Block RAM (BRAM) banks. These banks are partitioned to allow simultaneous writing of contributions for the IWE and its partial derivatives with respect to velocity ($v_x$ and $v_y$). A register-based buffer is implemented to resolve potential write conflicts when multiple events target the same memory address within consecutive clock cycles.
4. Gradient Calculation and Optimization: The system computes the gradient of the contrast function (variance of the IWE) with respect to motion parameters. This involves calculating the IWE and its derivatives, subtracting mean values, and performing dot products. A gradient ascent method iteratively updates the motion parameters ($v_x, v_y$) over a fixed number of iterations (100 in the experiments).
5. ROI Update: Once the optimization loop converges, the calculated motion parameters are used to update the ROI coordinates for the next batch of events, enabling continuous object tracking.

Key Contributions

• First Hardware Implementation: The paper claims to present the first hardware architecture specifically designed to accelerate the Contrast Maximization algorithm on an FPGA platform.
• Deeply Pipelined Architecture: The design features a fully optimized circuit that executes warping, IWE generation, gradient computation, and optimization loops simultaneously, leveraging the FPGA's deterministic parallelism.
• Hardware-Aware Optimization: The authors detail specific implementation strategies, including a 12-bank memory scheme to prevent write conflicts and a zero-write-back mechanism to clear BRAMs between iterations without requiring a full hardware reset.
• Validation: The architecture was validated using an event-based object tracking application, demonstrating the system's ability to estimate horizontal and vertical velocity components in real-time.

Experimental Results
The proposed architecture was implemented on a Kria KV260 Vision AI Starter Kit (Zynq UltraScale+ MPSoC) and compared against Python reference models running on an Intel Core i5-11300H CPU and an Nvidia GeForce RTX 3050 Ti GPU.

• Performance: For a batch of 5,000 events with a 64×64 ROI and 100 optimization iterations, the FPGA achieved a processing time of 0.92 ms. This represents a speedup of over 200 times compared to the CPU (185.96 ms) and approximately 500 times compared to the GPU (473.51 ms).
• Resource Utilization: The design utilized less than 4% of the available Look-Up Tables (LUTs) and Registers on the FPGA. BRAM usage was 17.7%, and DSP usage was 12.1%, indicating that the architecture is resource-efficient and leaves room for larger ROIs or more complex models.
• Power Efficiency: The system consumed approximately 0.813 W of on-chip power, highlighting its suitability for energy-constrained embedded systems.

Significance and Claims
The authors assert that this work demonstrates the feasibility of accelerating the computationally heavy CM algorithm using FPGA resources. By achieving processing speeds over 200 times faster than CPU implementations while maintaining low power consumption, the proposed architecture provides a solid foundation for real-time motion estimation in high-speed, low-power embedded systems. The paper concludes that while the current design is a significant step forward, future work could focus on increasing the operating frequency, reducing resource consumption, supporting more advanced motion models, and expanding the ROI size through external memory integration.