⚡ electrical engineering

Privacy-Preserving Fully Distributed Gaussian Process Regression

This paper proposes a privacy-preserving fully distributed Gaussian process regression protocol based on secure multi-party computation that enables agents to collaboratively learn a global model and optimize hyperparameters while preventing data leakage from semi-honest coalitions.

Original authors: Yeongjun Jang, Kaoru Teranishi, Jihoon Suh, Takashi Tanaka

Published 2026-08-12

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Yeongjun Jang, Kaoru Teranishi, Jihoon Suh, Takashi Tanaka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your smart devices, like your phone or your home thermostat, are constantly learning from your habits to make better predictions. They use a clever mathematical tool called Gaussian Process Regression (GPR). Think of GPR as a super-smart detective that doesn't just guess the answer; it also tells you how confident it is in that guess. This is incredibly useful for things like self-driving cars or medical monitoring, where knowing the "uncertainty" is just as important as the prediction itself.

Usually, to make these detectives super-smart, you'd need to dump all the data from every device into one giant central brain. But that's a privacy nightmare. Your health records or daily routines shouldn't be sitting in a single server where they could be leaked. So, scientists came up with Distributed Learning, where each device keeps its own data and only shares its "conclusions" with neighbors. However, even those conclusions can sometimes accidentally reveal secrets about the original data. This paper tackles the tricky problem of how to let these devices collaborate and learn together without anyone (even a sneaky group of neighbors) figuring out what the others' private data actually looks like.

The Secret Club of Learning Machines

In this paper, the authors, Yeongjun Jang and his team, propose a new way for a group of agents (let's call them "learning robots") to solve a puzzle together without ever showing their private pieces to each other. They call their solution a Privacy-Preserving Fully Distributed Gaussian Process Regression protocol.

Here is the scenario: Imagine a group of hospitals trying to predict how a new patient will respond to a treatment. Each hospital has its own private list of past patients. They want to combine their knowledge to get a better answer, but they can't share their patient lists because of strict privacy laws. They also don't trust a central "super-server" to hold the data, because that server could be hacked or could be a competitor.

The authors' big idea is to use a cryptographic trick called Secure Multi-Party Computation (SMPC). To understand this, imagine the hospitals are trying to calculate the average height of all their patients, but they don't want to tell anyone their specific patient's height.

The Magic of "Secret Sharing"

The paper uses a technique called Additive Secret Sharing. Here's how it works in our story:

The Split: Instead of sending their real number (say, 170 cm), a hospital splits it into random "shards." For example, Hospital A might keep a shard of +50, send a shard of -30 to Hospital B, and a shard of -20 to Hospital C.
The Noise: To the neighbors, these shards look like random noise. Hospital B sees "-30" and has no idea if the original number was 170, 500, or -100. It's mathematically impossible to guess the original number without all the shards.
The Reassembly: The hospitals pass these shards around in a circle. Eventually, they add up all the shards they receive. Because the math is set up perfectly, the random noise cancels out, and the sum reveals the correct total (or average) without anyone ever seeing the individual numbers.

The authors built their entire system on this concept, but they had to solve a few tricky problems to make it work for real-world data.

The "Quantization" Hurdle

Real-world data (like patient heights or temperature) involves decimals. But secret sharing usually works best with whole numbers (integers). To fix this, the authors introduced a "ruler" or a scale factor. They tell the robots to round their numbers to the nearest tick mark on a ruler.

The Trade-off: If the ruler has very fine tick marks (a small scale factor), the math is very precise, but the numbers get huge, making communication slow. If the ruler has coarse tick marks, it's fast but less accurate.
The Finding: The team proved that you can make the error from this rounding as tiny as you want by choosing a fine enough ruler and running the protocol for enough rounds. They showed that even with this rounding, the final result is practically identical to what you would get if everyone had shared their raw data openly.

The "Masking" Dance

There was one more danger: What if two hospitals collude? If Hospital A and Hospital B are neighbors, could they figure out what Hospital C is hiding?
To stop this, the authors added a masking step. Before sending their shards, the robots generate extra "dummy" numbers that cancel each other out perfectly. It's like a group of dancers passing a secret note; they pass it around in a triangle so that no two people can ever see the whole path of the note. The paper proves that as long as the network of robots is connected enough (specifically, if every pair of neighbors shares at least one common friend), a small group of "semi-honest" robots (who follow the rules but try to peek) cannot learn anything more than the final average result.

Optimizing the "Secret Sauce"

The paper also tackles a problem often ignored in privacy research: Hyperparameter Optimization.
In GPR, there are "knobs" (called hyperparameters) that control how the model learns. Turning these knobs correctly is vital for accuracy. Usually, you need to look at all the data to find the best settings. The authors showed how the robots can adjust these knobs together, using the same secret-sharing dance, without ever revealing their local data. They let the robots take small steps toward the best setting, averaging their progress securely at each step.

What They Found

The team tested their method on two real-world datasets:

SARCOS: A dataset about robot arm movements (used for testing large-scale problems).
Diabetes: A dataset about patient health records (where privacy is critical).

They compared their method against other privacy techniques that rely on a central server or heavy encryption.

Speed: Their method was significantly faster than the alternatives, which often timed out or took over 300 seconds. However, the exact speed depends on the network setup. For a 20-agent network with 4 neighbors, their method finished in about 0.59 seconds. But if the network is larger (40 agents) or more densely connected (19 neighbors per agent), the time increases to roughly 0.99 seconds or 6.69 seconds depending on the dataset. While not always under one second, it remains orders of magnitude faster than the competing methods.
Accuracy: The results were very close to the "perfect" non-private version. The difference (measured as Root Mean Square Error) was tiny, often less than 0.02.
Privacy: They mathematically proved that the protocol is secure against groups of up to a certain size of colluding agents. The size of this "safe group" depends on how many neighbors each robot has; more connections mean better privacy.

The Bottom Line

This paper doesn't just suggest a cool idea; it provides a working recipe. The authors demonstrated that you can have your cake and eat it too: a group of agents can learn a powerful, accurate model together while keeping their individual data completely hidden from each other, without needing a trusted central boss. They showed that by using secret sharing and a bit of mathematical "rounding," you can achieve this in a fully distributed way that is both fast and secure.

The results suggest that this approach is ready for real-world use, offering a practical path forward for privacy-sensitive applications like healthcare, finance, and smart cities, where data is too valuable to share but too important to ignore.

Technical Summary: Privacy-Preserving Fully Distributed Gaussian Process Regression

Problem Formulation
Distributed Gaussian Process Regression (GPR) allows multiple agents to collaboratively learn a model of a target function by aggregating local datasets without a central server. While this approach improves predictive accuracy and scalability, it introduces significant privacy risks. In standard distributed GPR, agents exchange local posterior means and variances derived from their private datasets. These exchanges can inadvertently leak sensitive information about the underlying local data (e.g., patient records or proprietary industrial data). Existing solutions relying on Secure Multi-Party Computation (SMPC) or Homomorphic Encryption (HE) often require a trusted central server for aggregation, creating a single point of failure and computational bottleneck. Conversely, Differential Privacy (DP) methods sacrifice accuracy by injecting noise. Furthermore, existing privacy-preserving frameworks largely overlook the optimization of kernel hyperparameters, which is critical for prediction accuracy.

Methodology
The authors propose a fully distributed GPR protocol based on SMPC that eliminates the need for a trusted central server. The core methodology relies on a secure distributed average consensus algorithm utilizing additive secret sharing over a finite field of integers ( $\mathbb{Z}_q$ ).

Quantization and Consensus: Since additive secret sharing operates on integers, the protocol quantizes real-valued local model parameters (posterior means and variances) using a scale factor $L_z$ . Agents perform a diffusive consensus update using Metropolis weights. To ensure privacy, the protocol employs masking terms generated via a fully distributed procedure where neighbors share random "shares" of zero vectors. This ensures that no individual agent can infer the specific contribution of another agent beyond the aggregated result.
Convergence Analysis: The paper establishes that the local models practically converge to the global model obtained by standard distributed GPR. A sufficient condition on the modulus $q$ and scale factors is derived to guarantee that the convergence error is bounded by $L_z$ . By refining parameters, this error can be made arbitrarily small, albeit with increased communication costs.
Privacy Guarantees: Security is formalized using the simulation-based security paradigm. The protocol is proven to be $h$ -private against semi-honest coalitions of bounded size, where $h$ depends on the graph topology (specifically, the number of common neighbors between adjacent agents).
Hyperparameter Optimization: The framework is extended to include privacy-preserving optimization of kernel hyperparameters (length-scale and signal variance). This is achieved by alternating between the secure consensus protocol and local gradient ascent steps, effectively implementing a consensus-based gradient method without exposing local gradients.

Key Contributions

Fully Distributed Architecture: The protocol achieves privacy without a trusted central server, addressing the scalability and trust limitations of existing SMPC and HE-based approaches.
Quantization Analysis: The authors analyze the impact of quantization errors on convergence and provide explicit guidelines for selecting protocol parameters ( $L_z, L_w, q$ ) to balance privacy, convergence accuracy, and communication cost.
Formal Privacy Proof: Formal privacy guarantees are established for semi-honest, potentially colluding agents, showing that individual contributions remain hidden beyond what is implied by the aggregated output.
Hyperparameter Optimization: The work introduces a method for privacy-preserving optimization of kernel hyperparameters, a component often neglected in prior privacy-preserving GPR literature.
Efficiency: The proposed method avoids the heavy computational overhead of matrix inversions over encrypted shares or the decryption/aggregation burden on central servers found in competing methods.

Experimental Results
The protocol was evaluated on two real-world benchmark datasets: the SARCOS dataset (robotic arm control) and the Diabetes patient dataset.

Convergence: Experiments demonstrated that local hyperparameter estimates and model outputs converge to a common value across all agents. The Root Mean Square Error (RMSE) between the proposed protocol's output and the standard distributed GPR output decreased monotonically as the number of iterations increased and the quantization scale factor decreased.
Runtime Performance: The proposed protocol significantly outperformed existing methods in terms of total runtime. On the SARCOS dataset, competing methods ([12], [13]) exceeded 300 seconds, whereas the proposed method completed in under 1 second for smaller networks and approximately 0.5–1.0 seconds for larger networks.
Trade-offs: While the proposed method exhibited slightly higher RMSE compared to centralized or server-assisted methods due to quantization, the authors note this gap can be narrowed by increasing iterations and refining scale factors. The runtime advantage is attributed to avoiding matrix inversions over shares and encryption/decryption overheads.

Significance
The paper claims to provide a practical and effective solution for privacy-preserving collaborative learning in scenarios where agents mutually distrust each other and cannot rely on a central authority. By integrating secure consensus with GPR, the method enables agents to jointly learn accurate models while protecting sensitive local data. The inclusion of hyperparameter optimization within this privacy-preserving framework addresses a critical gap in the literature, ensuring that the resulting models are not only private but also performant. The experimental results validate the protocol's applicability to real-world problems, demonstrating a substantial reduction in computational runtime compared to state-of-the-art alternatives.