Technical Summary: Sarus – Privacy-Preserving Multi-Vendor Perception Fusion via Homomorphic Encryption

1. Problem Statement

Cooperative perception allows autonomous vehicles (AVs) and infrastructure to aggregate detection outputs from multiple agents, improving situational awareness, robustness against occlusion, and sensing range. However, in multi-vendor deployments, sharing raw detection outputs (e.g., bounding boxes, confidence scores) at inference time creates significant privacy and security risks:

• Model Extraction: Adversaries can infer proprietary model decision boundaries or reconstruct surrogate models by observing confidence scores and response patterns.
• Data Leakage: Detection outputs reveal sensitive environmental information, including the presence and behavior of individuals or objects.
• Side-Channel Leakage: Auxiliary data such as timing, output sparsity, and detection patterns can leak information about scene complexity or sensor characteristics.
• Cross-Vendor Inference: Competitors can analyze relative strengths and weaknesses of different models by comparing outputs on the same scene.

Existing solutions often rely on plaintext sharing or focus on training-time privacy (e.g., Federated Learning), which is unsuitable for inference-time collaboration where instance-level predictions must be aggregated. There is a lack of scalable frameworks that enable secure aggregation of structured perception outputs without revealing vendor-specific information.

2. Methodology: The Sarus Framework

Sarus addresses these challenges by transforming detection outputs into compact, encrypted statistical representations that can be aggregated directly in the encrypted domain using Homomorphic Encryption (HE).

2.1 Core Concept: Gaussian Splatting and Moment Encoding

Instead of sharing discrete bounding boxes (which require non-linear operations like Non-Maximum Suppression), Sarus converts each detection into a Gaussian splat.

• Representation: A bounding box $b = [x_1, y_1, x_2, y_2]$ is modeled as a spatial Gaussian distribution $N((\mu_x, \mu_y), \text{diag}(\sigma_x^2, \sigma_y^2))$. The mean $(\mu_x, \mu_y)$ corresponds to the box center, and the variance is derived from the box dimensions with a tunable scaling parameter $\kappa$.
• Moment Vectors: Each detection is encoded as a weighted moment vector containing sufficient statistics:
  $$\mathbf{m} = w \cdot [1, \mu_x, \mu_x^2, \sigma_x^2, \mu_y, \mu_y^2, \sigma_y^2]$$
  where $w$ is a trust-weighted confidence score.
• Linearity: Crucially, aggregating these moment vectors across multiple detections is a linear operation (summation). This allows the fusion server to combine encrypted payloads without decrypting them.

2.2 Spatial Binning and Soft Assignment

To ensure scalability and prevent spatially distant objects from interfering, the scene is discretized into a shared spatial lattice (class-specific grid).

• Soft Assignment: A detection's moment vector is distributed to up to four neighboring bins using bilinear interpolation weights based on the Gaussian center's position relative to bin centers.
• Local Aggregation: Each vendor locally aggregates moment vectors into per-bin accumulators before encryption. This reduces the payload size from $O(N)$ (where $N$ is the number of detections) to $O(B)$ (where $B$ is the number of occupied bins), as multiple detections often fall into the same spatial hypothesis.

2.3 System Workflow

1. Vendor Side:
   • Process local sensor data to generate detections.
   • Convert detections to Gaussian splats and compute moment vectors.
   • Aggregate moments into spatial bins (class-specific).
   • Encrypt the aggregated moment vectors and associated statistics (confidence mass, count) using the CKKS scheme (supporting approximate arithmetic).
   • Transmit the encrypted payload to the fusion server.
2. Fusion Server:
   • Collects encrypted payloads from all participating vendors.
   • Computes the union of occupied keys (class-bin pairs).
   • Performs homomorphic addition on the encrypted moment vectors for each key across all vendors.
   • Aggregates plaintext statistics (mass and count) for later use.
   • Returns the fused encrypted payload.
3. Reconstruction (Authorized Party):
   • Decrypts the fused moment vectors.
   • Inverts the moments to recover the fused Gaussian parameters ($\mu, \sigma$).
   • Reconstructs bounding boxes from the Gaussian parameters.
   • Performs cluster fusion: Constructs a spatial consistency graph where nodes are bin-level hypotheses. Edges connect adjacent bins if they satisfy geometric overlap (IoU) and statistical proximity (Mahalanobis distance) constraints. Connected components are merged to produce the final fused detections.

2.4 Threat Model

• Adversarial Server: Honest-but-curious; follows the protocol but attempts to infer individual vendor inputs from ciphertexts.
• Curious Vendors: Mutually untrusted; can only see their own inputs and the final decrypted output.
• Assumptions: The encryption scheme is semantically secure; vendors adhere to a shared spatial reference frame and payload schema. Active attacks and key compromise are out of scope.

3. Key Contributions

• Encrypted Moment-Based Fusion: A framework enabling multi-vendor perception aggregation without revealing raw detections, utilizing the linearity of Gaussian moments.
• Spatial Binning Mechanism: A strategy ensuring scalable fusion with complexity $O(B \cdot V)$, where $B$ is the number of occupied bins and $V$ is the number of vendors, avoiding explicit cross-vendor matching.
• Comprehensive Evaluation: A performance analysis covering vendor payload generation, server-side HE fusion, and postprocessing, demonstrating linear scaling.
• Empirical Validation: Demonstration on the KITTI dataset showing that Sarus effectively recovers complementary detections across camera (YOLOv8) and LiDAR (PointPillars, PV-RCNN) modalities, particularly in distance-dependent regimes where individual modalities degrade.

4. Experimental Results

Experiments were conducted using the KITTI dataset and a real-world autonomous test vehicle setup (VTTI track) with varying speeds and artificial rain.

• Correctness: Homomorphic fusion outputs were compared against plaintext fusion. The Intersection-over-Union (IoU) between the two exceeded 0.99997 on average, with bounding box center deviations $\le 0.5$ pixels. This confirms that CKKS-based approximate arithmetic introduces negligible error for perception tasks.
• Computational Complexity:
  • Vendor Payload Construction: Scales linearly with the number of detections ($O(N)$) and occupied bins ($O(B)$). Encryption is the dominant cost.
  • Server-Side Fusion: Scales as $O(B \cdot V)$. The server processes aggregated bins rather than individual detections.
  • Postprocessing: Dominated by decryption and graph-based cluster fusion, scaling as $O(B + \sum B_c^2)$.
• Performance Overhead: Homomorphic encryption introduces a constant-factor overhead (ranging from 300x to 1000x compared to plaintext) and increases payload size (ciphertext expansion factor $\approx 3 \times 10^3$). However, the system maintains linear scaling, making it predictable and feasible for real-time deployment under bounded scene complexity.
• Situational Awareness: Sarus significantly improved scene-level coverage, especially for pedestrians and at long ranges ($\ge 40$m) where LiDAR performance degrades due to sparsity. It successfully aggregated complementary detections from camera and LiDAR modalities.

5. Significance and Claims

The paper claims that privacy-preserving multi-vendor perception fusion is feasible for real-time deployment when statistical compression (moment encoding) and spatial sparsity (binning) are jointly exploited.

• Feasibility: Despite the computational overhead of HE, the linear scaling behavior ensures that the system remains predictable and scalable. The bounded constant-factor overhead does not preclude real-time operation in scenarios with moderate scene complexity.
• Privacy vs. Utility: Sarus successfully decouples utility from privacy leakage. It allows vendors to contribute to a unified scene understanding without exposing proprietary model behaviors or sensitive environmental data.
• Limitations: The authors acknowledge that the Gaussian approximation may degrade precise bounding-box localization at high IoU thresholds. The framework currently assumes trusted admission control and does not handle active adversaries or complex key management scenarios (e.g., threshold decryption), which are left for future work.

In conclusion, Sarus bridges the gap between cooperative perception and privacy preservation, offering a practical pathway for secure, multi-vendor collaboration in autonomous driving ecosystems.