Federated Learning Algorithms for Image Privacy

If you want private image training, federated learning alone is not enough. The short answer is this: FedAvg is the starting point, FedProx, SCAFFOLD, and FedDyn help with skewed client data, pFedMe and PGFed focus on per-client tuning, and FedGAN, VQ-FedDiff, and PDFed are the main picks for image synthesis.

I’d sum the article up like this: your choice depends on three things - privacy, image quality, and system cost. For example, SCAFFOLD can need about 2× the per-round communication of FedAvg, ResNet-18 can cost about 42.7 MB per round, and user-level DP with ε < 4 may need millions of users to keep the accuracy drop near 5%. On the generation side, diffusion methods usually give better image quality than GANs, but they need more compute and slower sampling.

If I were scanning this article fast, these are the points I’d want first:

• FedAvg: simple baseline, but weak on non-IID image data
• FedProx: adds a penalty term to limit client drift
• SCAFFOLD: corrects drift more directly, but sends more data
• FedDyn: changes the training objective to reduce drift
• pFedMe: keeps a shared model plus client-specific tuning
• PGFed: shares only parts of the generative model
• FedGAN: trains generator and discriminator across clients
• VQ-FedDiff: diffusion backbone is shared; local conditioning stays private
• PDFed: no central server; uses blockchain, voting, and memorization checks

Privacy versus Robustness in Federated Learning: Limits and Algorithms

sbb-itb-903b5f2

Quick Comparison

Algorithm	Main Idea	Handles Non-IID Data	Privacy Support	Best Use
FedAvg	Average client updates	Low	DP can be added	Baseline FL
FedProx	Add proximal term	Better than FedAvg	DP can be added	Mixed hardware, skewed data
SCAFFOLD	Use control variates	Strong	DP can be added	Drift-heavy setups
FedDyn	Change local objective	Strong	DP can be added	Fewer rounds, more local compute
pFedMe	Shared + personal models	Strong for per-client fit	Shared model can use DP	Personalized tasks
PGFed	Share only global generative parts	Strong for heterogeneous clients	Local parts stay private; DP can be added	Personalized generation
FedGAN	Sync full GAN across clients	Mixed	DP / encryption options	GAN-based synthesis
VQ-FedDiff	Share diffusion backbone only	Strong	Local conditioning reduces exposure	High-quality synthesis
PDFed	Decentralized diffusion with voting	Aims to limit bias	Memorization filtering + audit trail	Auditable training

Bottom line: if you need a simple baseline, start with FedAvg or FedProx. If skewed data is the main issue, look at SCAFFOLD or FedDyn. If image generation is the goal, FedGAN fits GAN pipelines, while VQ-FedDiff and PDFed fit diffusion workflows where privacy controls matter more.

1. FedAvg

FedAvg is the default baseline for federated image training. Here’s the basic idea: a central server sends the current global model to a subset of clients, each client runs several local SGD steps on its own private data, and the server combines those updates with a sample-weighted average using ($n_i/n$).

Update Aggregation

The local-update period ($K$) sets the main trade-off. More local steps mean less communication, which sounds good. But there’s a catch: they also increase client drift.

That trade-off gets expensive fast. Sending the parameters of a ResNet-18 model takes about 42.7 MB per round. So when clients have very different image distributions, every extra round and every extra local step starts to matter more.

Non-IID Robustness

FedAvg has the hardest time when clients store image data with different styles, labels, or class distributions. In that setting, local models move in different directions, and the server’s average can land on a pooled solution that isn’t very good.

The numbers on CIFAR-10 make that pretty plain. In an extreme non-IID setup using a Dirichlet distribution of 0.05, vanilla FedAvg reached 25.78% accuracy. Under the same setup, FedProx scored 18.05% and SCAFFOLD scored 14.14% before data augmentation was used.

This same update behavior also affects privacy-first training. Once clipping and noise enter the picture, the method has even less room to adjust.

Privacy Integration

DP-FedAvg extends FedAvg by clipping client updates and adding Gaussian noise during aggregation.

One 2022 study on The Cancer Genome Atlas (TCGA) dataset showed what that can look like in practice. The method classified lung cancer subtypes with a privacy bound of ($\epsilon = 2.90$) at ($\delta = 0.0001$), while delivering performance comparable to centralized training.

That said, strong user-level privacy guarantees below ($\epsilon = 4$) often require millions of users to avoid too much utility loss. So the math can work, but the scale requirement is no small thing.

Image-Generation Suitability

This baseline shows more strain in GAN-based image generation. FedAvg can coordinate local generators and discriminators in federated GAN settings, but mixed image distributions can make training unstable.

One detail matters a lot here: synchronizing both the generator and discriminator leads to better image quality than syncing only one of them. If only the discriminator is synchronized, it can hit an optimum too early, which stalls the generator’s learning.

So in federated GANs, both sides need to stay in sync to avoid instability. That’s why FedAvg is mostly treated as the starting point that later privacy-first image-generation methods try to beat.

Feature	FedAvg (Standard)	DP-FedAvg
Aggregation	Weighted averaging of local weights	Clipped updates + Gaussian noise
Bandwidth cost	High	High
Non-IID handling	Poor (client drift)	Still vulnerable; noise reduces utility
Built-in privacy	No built-in guarantee	Provable ($\epsilon$)-DP

2. FedProx

Where FedAvg tends to let clients wander off, FedProx pulls them back in. It adds a proximal term, $\frac{\mu}{2}|w - w_t|^2$, to each client’s local objective. In plain English, that term keeps local updates closer to the current global model, which cuts down on client drift in heterogeneous image data.

FedProx also gives clients more freedom in how they train locally. Instead of forcing the same number of local steps for everyone, it allows variable local optimization. That matters in cross-device setups, where one phone or laptop may have far less compute power than another. Slower clients can still send partial updates instead of being left out. The catch is that $\mu$ needs careful tuning: if it’s too low, FedProx acts a lot like FedAvg; if it’s too high, updates get smaller and convergence slows.

Non-IID Robustness

The proximal regularization is built for non-IID data, which shows up all the time in federated image tasks. When client datasets look very different from one another, FedProx has more room to help.

In heterogeneous 3D medical image classification, FedProx-enhanced frameworks showed accuracy gains of up to 18.2% over standard federated learning methods. That’s a big jump, and it matters most when client image distributions differ sharply.

Privacy Integration

FedProx also works with differential privacy. The DP-Prox framework mixes the proximal term with Gaussian noise and adaptive sampling. On MNIST and Fashion-MNIST, it reached nearly 6% better accuracy than standard SGD-based DP methods.

That extra stability is the key point here. Privacy noise can make local training shaky, and the proximal term helps keep things under control. So when training is both noisy and non-IID, FedProx tends to be a better fit than plain DP-SGD.

Image-Generation Suitability

For image generation, the same constraint can help local generators stay closer to the shared model. That can limit overfitting to skewed local data, which in turn lowers mode collapse risk in federated GANs and diffusion models.

The strongest support for FedProx still comes from classification and medical imaging, but the logic carries over well to federated image generation.

Feature	FedAvg	FedProx
Local Objective	Standard loss	Loss + proximal term
Local Iterations	Fixed/uniform	Variable (inexact)
Straggler Handling	Discards slow devices	Accepts partial updates
Non-IID Strategy	Simple averaging	Proximal regularization
Privacy Compatibility	Standard DP-SGD	DP-Prox (stabilized)

3. SCAFFOLD

SCAFFOLD tackles client drift head-on. Instead of only trying to limit how far local training can wander, it uses two control variates: one on the server side and one on the client side. At each local training step, the client uses these variates to estimate drift and correct for it, which helps keep updates pointed in the global direction. In plain terms, SCAFFOLD goes after the main issue that FedAvg leaves open and that FedProx only partly softens.

Update Aggregation

Each local step uses a drift-corrected gradient update. That gives SCAFFOLD a steadier update rule than FedAvg's plain averaging and FedProx's proximal penalty, especially on skewed image data. The catch is communication. Each client has to send both model weights and control variates, which is about 2× the per-round communication of FedAvg. The server and clients also have to store those variates across rounds, so memory use goes up too.

Non-IID Robustness

SCAFFOLD was designed for heterogeneous data, and the results show why that focus matters. On CIFAR-100 with an extreme non-IID split using Dirichlet (\alpha = 0.05), vanilla SCAFFOLD reached a top-1 accuracy of 14.14%.

Privacy Integration

Because the correction term relies on stochastic gradients, DP-SCAFFOLD works with differential privacy without adding extra privacy leakage. DP-SCAFFOLD can deliver user-level privacy guarantees of (\epsilon < 4) while keeping utility drops under 5% when millions of users take part. One common implementation trick is a warm-start phase. In that setup, the first few rounds initialize client control variates before global model updates begin. That step matters most when privacy noise and heterogeneous image data are both pushing on the model at once.

Image-Generation Suitability

Drift correction helps keep federated GANs and diffusion model generators aligned across clients, which makes training steadier. Most validation work is still done on small benchmark datasets like EMNIST, though, and the added per-round communication makes large-scale deployment harder.

Feature	FedAvg	FedProx	SCAFFOLD
Mechanism	Simple model averaging	Proximal regularization	Control variates (variance reduction)
Non-IID Handling	Poor; client drift	Moderate; limits update magnitude	Strong; corrects update direction
Communication	Moderate	Moderate	High (about 2× parameters per round)
Convergence Speed	Slower on skewed data	Moderate	Often fewer rounds needed
Privacy Compatibility	Standard DP-FedAvg	DP-FedProx	DP-SCAFFOLD

FedDyn takes a different route by reshaping the optimization objective rather than correcting gradients.

4. FedDyn

Unlike SCAFFOLD, which uses an explicit drift-correction term, FedDyn changes the objective itself. The idea is simple: reshape the local objective so client updates stay closer together and drift less over time.

In privacy-preserving image generation, FedDyn can also work with differential privacy by adding Gaussian noise to local updates before aggregation.

Smaller ε gives stronger privacy, but it usually hurts image quality. So with FedDyn, the main trade-off is privacy vs. utility.

This objective-shaping method matters most when local image distributions are highly uneven.

5. pFedMe

When the target moves from one global model to better results for each client, pFedMe is a natural next move. It’s a personalized federated learning method that keeps a shared backbone, while each client also maintains and updates its own model around that backbone. In plain terms, clients still learn from one another, but each device can also tune the model to match its own data.

Non-IID Robustness

That setup works well for non-IID image data. If client datasets differ a lot, one global model often falls short for some clients. pFedMe helps by letting each client adjust locally without giving up the gains that come from shared learning.

Privacy Integration

pFedMe can also work with differential privacy. One common setup is to apply DP-SGD to the shared model updates, while client-specific parameters stay local. This matters for diffusion models because they can memorize training data, which may lead them to reproduce private training images. With personalization, that tension becomes a bit clearer: the shared model leans toward broad patterns, while more fine-grained local traits remain on the client.

Image-Generation Suitability

pFedMe is a good match for heterogeneous clients, but there’s a cost. Each client has to train a personalized model alongside the shared updates, which adds local compute. So it works well as a baseline when you want personalization first, before moving to methods that trade some of that client-level tuning for other privacy or image-generation gains.

6. PGFed

PGFed pushes personalization a step further by drawing a clearer line between what gets shared and what stays on the client. PGFed, also called Personalized Global Federated Learning, is a personalized federated learning method for generative models. It shares only transferable generative components, while client-specific parts remain local.

Update Aggregation

PGFed-style methods share only selected parts of the generative pipeline, and keep client-specific modules on each device. That split becomes most useful when image distributions vary a lot from one client to another.

Non-IID Robustness

Selective sharing helps when client image distributions differ in a big way. If one fully aggregated model struggles on outlier clients, sharing only the parts that generalize across clients can handle non-IID variation better. The tradeoff is pretty clear: you get stronger personalization, but you give up some of the simplicity that comes with full-model averaging.

Privacy Integration

PGFed can also work with differential privacy. In that setup, noisy global updates learn broad structure, while sensitive details stay inside local client modules.

Image-Generation Suitability

This split fits GANs and diffusion models well, since global structure and local detail can be trained separately. In practice, that means the model can learn shared patterns across clients without forcing every client to use the exact same generator behavior. It also sets up the more model-specific methods discussed next.

7. FedGAN

FedGAN takes a different path from earlier personalization methods. Instead of only deciding what to share, it tries to keep an entire GAN in sync. Each client trains both the generator and the discriminator on its own device, then syncs both parts at set intervals. That makes FedGAN a GAN-specific federated method. And compared with FedAvg-style setups that often kept the generator on a central server, this setup cuts down on constant back-and-forth during training.

Update Aggregation

After every K local SGD steps, each client sends both generator and discriminator weights to the server. The server then computes a sample-weighted average and sends the updated model back out. In plain terms, clients do more work locally before checking in.

That tradeoff matters. A larger K reduces bandwidth use, but image quality can drop. So the sync schedule helps with communication, but it also leaves FedGAN more exposed when client data is heavily skewed.

Non-IID Robustness

FedGAN does come with a convergence guarantee under non-IID data. Even so, skewed client distributions can still throw training off balance and make mode collapse more likely. When that happens, the hit shows up where it hurts most: output image quality.

Privacy Integration

Since FedGAN exchanges both parts of the GAN, the privacy issue moves away from raw image sharing and toward model parameters. That sounds safer at first glance, but full generator and discriminator weights can still leak information through reconstruction attacks and membership inference attacks.

There are a couple of ways people try to soften that risk:

• DP noise can help limit what an attacker can pull from shared parameters.
• PP-FedGAN adds CKKS encryption plus differential privacy, with about 10 seconds of client-side overhead per round.
• PS-FedGAN shares only the discriminator, which reduces leakage from the generator.

Image-Generation Suitability

FedGAN still showed that it could produce representative images across classes in a 5-agent, 10-class non-IID split. That said, there’s a plain cost here: each local agent has to train both a generator and a discriminator, which doubles local compute cost.

In that sense, FedGAN sits between generic federated optimization and image-generation methods built for a specific model design.

8. VQ-FedDiff

VQ-FedDiff takes a different path from FedGAN. Instead of syncing both parts of a GAN, it shares the diffusion backbone across clients and keeps the vector-quantized conditioning on each client. That setup is aimed at sensitive, non-IID image data, which makes it a strong fit for image use cases where raw data can't leave the local site.

Update Aggregation

Clients aggregate the shared diffusion backbone, while VQ conditioning stays local. In plain English, the core model is learned together, but each client keeps its own conditioning tied to its local data distribution.

That split helps the method hold up better when client data is uneven.

Non-IID Robustness

The local conditioning is meant to deal with heterogeneity. The method reports state-of-the-art FID on IID and non-IID photorealistic and medical benchmarks.

Privacy Integration

Privacy comes from keeping client-specific conditioning local. The authors report low privacy risk in their experiments.

That same design also shapes image quality and sampling cost.

Image-Generation Suitability

Diffusion models avoid common GAN problems like mode collapse, and they tend to generate higher-quality synthetic images. The tradeoff is speed: sampling is slower than with GANs.

Use VQ-FedDiff when image quality matters more than sampling speed. A related federated diffusion method built on a UNet backbone also cuts exchanged parameters by up to 74% compared with naive FedAvg.

9. PDFed

Unlike VQ-FedDiff, which changes the model itself, PDFed changes how training gets managed. It removes the central server and runs diffusion-model training through Ethereum smart contracts.

Update Aggregation

Each client trains locally, stores model weights on IPFS, and submits or votes on candidate models through a smart contract. Other participants then test those submitted models on their own private data with a Quality-Novelty (Q-N) score and use those votes to decide which models should be aggregated or kept in training. In plain terms, trust moves away from one central coordinator and into protocol-level voting.

Non-IID Robustness

PDFed aims to reduce heterogeneity bias by letting nodes accept models only when the Q-N scores support that choice. It also lets validation-only nodes vote, even if they don't have the hardware needed for full training. That opens the door to more participants instead of limiting the process to the few nodes that can do heavy compute.

Privacy Integration

PDFed focuses its main privacy defense on training-data memorization. Its Q-N score uses visual fingerprinting and KNN searches to spot memorized samples and filter them out during data loading. Model provenance is also written on-chain through C2PA manifests, which makes the training history auditable.

There is a clear cost here. Fingerprinting and verification add compute overhead, and on-chain provenance adds more coordination work. The upside is stronger auditability, but it comes with a heavier workflow.

Image-Generation Suitability

PDFed is built for diffusion models, not GAN-based FL. Its Q-N objective tries to balance image fidelity with novelty so privacy leaks are less likely. That makes PDFed a fit for diffusion pipelines where auditability and memorization control matter more than keeping the setup simple.

Privacy, Utility, and Cost Trade-Offs

The main trade-off here is pretty simple: more privacy usually means more compute, more communication, or both. The comparison below boils that down to three things that matter most in practice: privacy, utility, and cost.

On the privacy side, FedAvg exposes the most because it sends full model updates without a formal privacy guarantee. FedProx helps with training stability, but on its own, it doesn't do much to strengthen privacy. DP-based methods give you formal guarantees, but you usually give up some accuracy in return. That trade is real. Still, at scale, it's often manageable.

For image quality, the split is mostly between GANs and diffusion models. Diffusion-based approaches tend to generate better images. The catch is cost: they need more compute and take longer at sampling time than GAN-based methods.

Communication adds another layer. SCAFFOLD cuts variance and often reduces the number of rounds needed, but it also sends about 2× the data per round compared with FedAvg. FedDyn can hit high accuracy in fewer rounds, though client-side compute may climb fast. By contrast, prompt-based federated methods shrink transmission to a small fraction of what model-based FL needs.

The table below shows where each method lands on the privacy-utility-cost curve.

Algorithm	Privacy Strength	Image Quality Potential	Communication Cost	Client Compute	Best-Fit Use Case
FedAvg	Low (vulnerable to gradient inversion)	Not for generation	High	Moderate	General non-sensitive FL
FedProx	Low	Not for generation	Medium	Medium	Heterogeneous non-sensitive data
SCAFFOLD	Low	Not for generation	High (≈2× FedAvg)	Medium	Faster convergence, higher per-round cost
FedDyn	Low	Not for generation	Low	Very High	High-accuracy needs, strong hardware
pFedMe	Low–Medium	Not for generation	Medium	High	Per-client personalization, non-IID data
PGFed	Medium	Medium	Medium	High	Selective sharing, heterogeneous generative tasks
FedGAN	Medium (reconstruction-prone)	Medium	High	High	Synthetic data augmentation
VQ-FedDiff	Medium	High	Medium	Very High	High-fidelity image generation
PDFed	Medium–High (memorization filtering)	High	Medium	High	Auditable, decentralized diffusion training

In U.S. medical imaging, this trade-off usually leans toward formal privacy guarantees. Hospitals and other institutions working under HIPAA can't share raw imaging data across silos, which makes FL with user-level DP the main route for cross-institution collaboration. In many cases, teams aim for stricter guarantees such as ε < 4, and utility can still stay within about 5% of non-private training when millions of users take part.

The next section turns these trade-offs into a direct pros-and-cons breakdown.

Pros and Cons

The table below sums up each method's main upside, downside, and privacy concern. It takes the trade-offs discussed earlier and boils them down into practical use cases and risks.

Algorithm	Pros	Cons	Best For	Key Privacy Caveat
FedAvg	Simple to implement; low communication overhead	Poor non-IID handling; prone to client drift	Homogeneous, non-private image tasks	Updates can expose memorized training data
FedProx	Stable in heterogeneous environments; handles device variance	Requires careful tuning of the proximal term μ	Non-IID networks with mixed hardware	Does not prevent gradient-based data reconstruction
SCAFFOLD	Corrects client drift using control variates	High state and communication overhead	High-variance, non-IID environments	Extra control-state increases the attack surface
FedDyn	Strong convergence through dynamic regularization	More complex to tune and deploy	Large-scale tasks needing fast convergence	Similar to FedAvg, it has no explicit privacy guarantee
pFedMe	Strong personalization for uneven client data	Weaker global consistency	Clients needing user-specific outputs	Local models may still encode sensitive features
PGFed	Shares only transferable generative parts	Weaker global consistency	Selective sharing, heterogeneous generative tasks	Local modules may still encode sensitive features
FedGAN	Established approach for synthetic image generation	Unstable training; prone to mode collapse	Federated synthetic image generation	Discriminator updates can reveal data patterns
VQ-FedDiff	State-of-the-art FID scores; personalized conditioning	Very high compute demands	High-fidelity medical or sensitive imaging	Conditioning reduces but doesn't eliminate memorization
PDFed	No central aggregator; reduces memorization via the Q-N objective	Blockchain latency adds operational complexity	Auditable diffusion training with memorization control	Metadata on a public ledger requires careful management

One point matters a lot here: these four methods improve optimization, not privacy. They still need DP or secure aggregation.

The split is pretty simple. Optimization-focused methods help training run better, but privacy has to be added afterward. Generative methods move the risk instead of removing it, often toward model leakage and memorization. Diffusion methods tend to produce better output quality than GANs, but they also demand more compute. PDFed's Q-N check is lightweight at an average of 41 ms per image, though blockchain latency still adds extra overhead.

Conclusion

No single federated learning algorithm fits every image-privacy setup. The best pick comes down to your goal, your data, and the limits of your system.

Looking across all nine methods, the pattern is pretty clear. Optimization-focused methods help training stay on track. Personalization-focused methods help each client get a better fit. And generative methods move the privacy tradeoff toward what gets shared and how much of it leaves the client.

FedAvg is the baseline. FedProx works better when clients differ a lot. SCAFFOLD and FedDyn focus on drift and convergence. pFedMe and PGFed lean into personalization. FedGAN and VQ-FedDiff support federated image generation. PDFed adds decentralized auditability.

The best option depends on the constraint you care about most.

Goal	Recommended Algorithm	Why
Multi-site baseline	FedAvg / FedProx	Simple, widely used, and better for system heterogeneity
Drift correction	SCAFFOLD / FedDyn	Corrects update direction and improves convergence on non-IID data
Personalization	pFedMe / PGFed	Keeps sensitive details local while sharing global structure
GAN-based synthesis	FedGAN	Established approach for federated synthetic image generation
Diffusion-based synthesis	VQ-FedDiff	Higher image fidelity and more stable training than GAN-based methods
Decentralized workflow	PDFed	Supports asynchronous participation and improves transparency

Federated learning does not guarantee privacy, and generative models can still memorize training data. For U.S. teams, one practical point matters a lot: separate user-level differential privacy from example-level differential privacy. User-level DP gives the stronger guarantee.

In practice, the right choice is a tradeoff between privacy strength, image quality, and system cost. Start with the simplest method that clears your privacy and quality bar. Then add differential privacy or partial sharing only if you need it.

FAQs

Which algorithm should I start with?

FedAvg is the classic starting point for federated learning. It handles model initialization, local training, model aggregation, and global updating. But there’s a catch: it can be exposed to privacy inference attacks.

If your goal is privacy-preserving image generation, PRISM and FedMD-CG are stronger picks. For large-scale embedding work, DP-FedEmb is built to deal with user-level sensitivity. NanoGPT also adds a privacy layer by keeping generated data stored locally on your device.

How much privacy does federated learning really provide?

Federated learning adds a basic layer of privacy by keeping raw data on local devices and sharing only model updates or gradients.

That said, the privacy here isn't absolute. Attackers can still reconstruct parts of the training data from shared updates. So teams often add differential privacy, secure aggregation, or generative models to give sensitive information more protection. NanoGPT also supports privacy-focused use by storing data locally on your device.

Are diffusion models worth the extra cost?

Yes. Even with higher compute demands and longer run times, diffusion models are still seen as worth it when the goal is high-quality image generation.

They’re slower than models like GANs. But they also tend to train more steadily and produce better image quality. That trade-off is a big reason so many teams still use them.

Work is also being done to cut some of that overhead. For example, researchers are testing federated learning frameworks to make training more efficient without hurting image quality.