🤖 machine learning

Beyond Weights and Gradients: A Taxonomy of Federated Learning Messages

This paper proposes a formal mathematical definition and a three-category taxonomy for federated learning messages—encompassing model structures, statistical summaries, and data-conditioned representations—to address the limitations of existing frameworks and clarify the trade-offs between utility, privacy, and communication costs in modern decentralized training.

Original authors: Alvaro Javier Vargas Guerrero, Xinguang Wang, Quang Manh Doan, Guy Nagels

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alvaro Javier Vargas Guerrero, Xinguang Wang, Quang Manh Doan, Guy Nagels

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of doctors who all have their own private patient records. They want to build a super-smart AI to diagnose diseases, but they can't share their actual patient files because of strict privacy laws.

Traditionally, Federated Learning (FL) was like this: Each doctor trains a small part of the AI on their own patients, then sends back a "recipe update" (mathematical weights or gradients) to a central chef. The chef mixes all the updates to improve the main recipe. The doctors never show their patients; they only show the recipe changes.

However, this paper argues that the field has grown up. Doctors are now sending all kinds of other things besides just recipe updates. Some are sending "fake" patient data, others are sending simple statistics like "average age," and some are sending complex maps of how diseases connect.

The authors say our old definition of what gets sent is too narrow. They propose a new Taxonomy (a classification system) to organize these different "messages" into three main buckets, comparing them like different types of mail.

The Three Types of Messages

The paper categorizes everything sent between the local computers and the central server into three groups:

1. Model Structures & Parameters (The "Recipe Updates")

What it is: This is the classic method. You send the actual math that makes the AI work (the weights) or the directions to change it (gradients).
The Analogy: Imagine sending a massive, detailed cookbook to the chef. It tells the chef exactly how to cook the dish.
Pros/Cons: It's very powerful for teaching the AI complex things, but the book is huge (takes a lot of internet bandwidth) and heavy to carry (requires powerful computers). Also, if someone steals the book, they might be able to reverse-engineer the original ingredients (patient data).

2. Statistical Summaries (The "Headlines")

What it is: Instead of sending a whole cookbook, you just send a few numbers. "The average age is 45," or "There were 100 clicks today."
The Analogy: Instead of sending the whole book, you send a postcard with just the headline: "Most people like pizza."
Pros/Cons: These are tiny, fast to send, and easy to understand. They are great for simple questions. However, they don't teach the AI how to cook a new dish; they just tell you what's popular. Because they are so simple, they are very easy to protect with math (like adding a little "noise" so no one can guess the exact number).

3. Data-Conditioned Representations (The "Fake Samples" & "Middle Layers")

What it is: This is the most creative group.
- Synthetic Data: You train a local AI to generate fake patients that look statistically like your real ones, then send those fake patients to the server.
- Embeddings: You send the "middle thoughts" of the AI (like a summary of a photo) rather than the photo itself.
The Analogy: Instead of sending the real patient, you send a wax figure that looks exactly like them (Synthetic Data). Or, you send a detailed description of the patient's face without sending the photo (Embeddings).
Pros/Cons: These are very flexible and can help the AI learn complex patterns. But they are tricky. If the wax figure is too perfect, it might accidentally reveal the real person's secrets. They also require a lot of computing power to create the fake data in the first place.

The Great Trade-Off

The paper explains that choosing which "mail" to send is a balancing act between three things:

How smart the AI gets (Utility): Recipe updates and fake samples make the AI very smart. Headlines (statistics) make it less smart but good for simple questions.
How fast and cheap it is (Communication/Compute): Headlines are instant and cheap. Recipe updates and fake samples are slow and expensive.
How safe it is (Privacy):
- Recipe updates are risky because hackers can sometimes reverse-engineer them to see the original data.
- Headlines are easy to protect with math (adding noise).
- Fake samples are risky if the generator isn't careful, as they might memorize real secrets.

What the Research Shows

The authors looked at 202 recent research papers to see what people are actually doing.

The Shift: Before 2021, almost everyone was just sending "Recipe Updates" (Model Parameters).
The Change: Since 2021, there has been a huge explosion in sending "Headlines" (Statistics) and "Fake Samples/Middle Layers" (Data-Conditioned Representations).
The Trend: The field is moving away from just "training a deep learning model" toward more specialized tasks, like finding cause-and-effect relationships or analyzing data without training a giant brain.

The Bottom Line

The paper concludes that we need to stop thinking of Federated Learning as just "sending weights." It's now a toolbox. Depending on whether you have a slow phone, a super-fast server, or strict privacy laws, you should choose a different type of "message" to send.

If you have a weak phone, send Headlines (Statistics).
If you have a powerful server and need a super-smart AI, send Recipes (Model Updates) or Fake Samples.
If you need maximum privacy, you might need to mix these messages with special math (like adding noise) to ensure no one can peek inside.

This new framework helps researchers pick the right tool for the job, rather than forcing everyone to use the same heavy, one-size-fits-all method.

Technical Summary: Beyond Weights and Gradients: A Taxonomy of Federated Learning Messages

Problem Statement

Federated Learning (FL) has traditionally been defined by the exchange of deep learning weights or gradients between clients and a central server, a paradigm established by foundational works like [McMahan et al., 2016]. However, the field is rapidly evolving to include diverse payloads such as synthetic data, intermediate embeddings, and federated analytics. Existing definitions and surveys fail to capture this breadth, often treating the federated message merely as a fixed byproduct of optimization or focusing solely on communication efficiency (e.g., compression, client selection) rather than the nature of the payload itself. Consequently, there is a lack of a cohesive framework to categorize these modern exchanges, evaluate their trade-offs regarding utility and privacy, or guide system designers in selecting appropriate payloads for specific hardware and security constraints.

Methodology

The authors propose a three-pronged approach to address this gap:

Formal Mathematical Definition: The paper steps away from restrictive statistical boundaries to propose a functional definition of a federated message. A message $M_k$ is defined as the output of a local mechanism $F_k$ processing a local dataset $D_k$ and a shared state $\theta$ (e.g., global model, query prompt). Crucially, the message is data-derived ( $M_k \neq D_k$ ) but not the raw dataset.
Conceptual Evaluation Axes: Messages are evaluated based on two primary axes:
- Utility: Whether the message improves the estimation of a global objective, metric, or model state when aggregated.
- Privacy: Whether the transmission structurally or cryptographically bounds the leakage of individual records, achieved through intrinsic masking, information-theoretic bounds (e.g., Differential Privacy), or cryptographic wrapping (e.g., Secure Multi-party Computation).
- Multi-Round Composition: The framework explicitly accounts for the iterative nature of FL, noting that while multi-round composition drives utility (convergence), it strictly degrades privacy over time, necessitating composition accounting.
Taxonomy Construction and Literature Survey: Based on the definitions, the authors construct a taxonomy categorizing messages into three groups: Model Structures, Statistical Summaries, and Data-Conditioned Representations. To validate the relevance of this taxonomy, a survey of 202 recent publications was conducted using keyword-driven discovery and citation snowballing, filtering for papers that explicitly introduce or benchmark structural message forms.

Key Contributions

1. A Formal Definition of Federated Messages

The paper establishes a definition that accommodates payloads ranging from deep learning weights to graph structures and synthetic data. It distinguishes between the mechanism (the function $F_k$ ) and the message (the output $M_k$ ), allowing for a unified view of diverse FL paradigms.

2. A Three-Category Taxonomy

The authors organize federated payloads into:

Model Structures and Parameters: Includes deep model updates (weights/gradients), non-deep parametric models (linear coefficients, decision trees, clustering centroids), and reinforcement learning policies. These are high-dimensional abstractions of local data.
Statistical Summaries: Designed for Federated Analytics (FA), these include basic statistics (sums, means), distributional representations (histograms, sketches), and matrix-based summaries. They are typically low-dimensional and focus on global metrics rather than predictive modeling.
Data-Conditioned Representations: Objects explicitly derived from local samples under a model. This includes embeddings/activations (used in Vertical FL), synthetic data (generated via GANs or diffusion models), and distillation targets (soft labels/logits).

3. Comparative Analysis of Trade-offs

The paper provides a detailed comparison of these groups across four dimensions:

Local Computation: Deep models require high computation (backpropagation); statistical summaries require minimal work; data-conditioned representations vary (embeddings are cheap, synthetic data generation is expensive).
Payload Size: Deep parameters are large; statistical summaries are very small; synthetic data can offer order-of-magnitude bandwidth savings compared to full gradient updates.
Global Aggregation: Summaries allow instant aggregation; deep models require coordinate-wise averaging; data-conditioned representations often require the server to run training loops or optimization steps.
Privacy and Defensive Tooling: Deep weights and embeddings are vulnerable to inversion attacks and often require heavy cryptography (Secure Aggregation, Homomorphic Encryption). Statistical summaries are transparent and best suited for Differential Privacy (DP). Synthetic data relies on the generalization limits of generators and requires DP to prevent memorization.

4. Empirical Landscape Analysis

The survey of 202 papers reveals a significant shift in research focus since 2021. While "Model Structures & Params" (specifically deep updates) remains dominant, there is a marked upward trend in "Data-Conditioned Representations" (e.g., embeddings, synthetic data) and "Statistical Summaries." The analysis indicates that approximately 76.9% of the research contributions are novel algorithms, highlighting a field driven by methodological innovation rather than just application deployment.

Results and Findings

Diversification: The FL ecosystem is moving away from a singular focus on weight updates toward specialized information sharing.
Trade-off Reality: No single message type is universally optimal. High-granularity payloads (deep models, synthetic data) maximize utility for complex tasks but demand immense local computation and face severe empirical privacy vulnerabilities. Conversely, statistical summaries offer rapid aggregation and high compatibility with formal privacy bounds (DP) but lack the semantic depth required for deep learning.
Architectural Implications: The choice of payload dictates the architectural limits. Mobile, cross-device networks are naturally constrained toward lightweight summaries and non-deep models, while cross-silo institutional networks with robust infrastructure are better positioned to handle the cryptographic and computational loads of high-dimensional model structures.

Significance and Claims

The paper claims that the traditional definition of a federated message has become a bottleneck for both research and practice. By providing a structured taxonomy and a formal definition, the authors aim to:

Clarify Trade-offs: Offer a clearer understanding of the balance between utility, communication costs, and privacy risks in decentralized training.
Guide Future Research: Provide a structured path for researchers to optimize federated systems for varying hardware and security requirements, moving beyond standard deep learning updates.
Shift the Paradigm: Highlight that the choice of federated payload is no longer just a technical detail but a strategic decision affecting the privacy, speed, and hardware requirements of the entire federation.

The authors maintain a modest stance, noting that their current survey of 202 papers provides a strong signal of shifting trends but that future work will involve expanding the literature search and developing standardized benchmarking frameworks to empirically validate these trade-offs. They do not claim to have solved the privacy-utility dilemma but rather to have provided the necessary framework to navigate it.