ScrambleMix: A Privacy-Preserving Image Processing for Edge-Cloud Machine Learning

Koki Madono · Masayuki Tanaka · Masaki Onishi — Waseda University · AIST · Tokyo Institute of Technology

Image and Video Technology — 11th Pacific-Rim Symposium, PSIVT 2023 (Auckland, New Zealand), LNCS 14403, pp. 326–340, Springer, 2024 · doi:10.1007/978-981-97-0376-0_25

• Paper (DOI)
• Slides (PDF)
• Code (GitHub)

Abstract

This paper proposes ScrambleMix, a novel privacy-preserving image processing for edge-cloud machine learning. ScrambleMix combines image scrambling and AugMix to improve visual information hiding. Specifically, to make two scrambled images from a single input image, each copy of the input image is scrambled using a different key every time. Then, the scrambled images are mixed with a randomly sampled mixing ratio. A self-teaching loss is introduced to improve the classification performance of ScrambleMix. In this study, we first evaluate the visual information hiding quantitatively using Learned Perceptual Image Patch Similarity (LPIPS). Then, the experiments with different settings demonstrate the proposed ScrambleMix outperforms the existing approaches for edge-cloud machine learning in terms of both classification accuracy and visual information hiding.

How it is trained and used

• ScrambleMix augmentation. Every training image becomes D scrambled images, each built from its own sampled key pair.
• Self-teaching loss. Different keys should not change the prediction for the same image, so each posterior ŷ_i,d is pulled towards the average posterior ȳ_i = StopGrad((1/D) Σ_d ŷ_i,d): L = L_CE + λ L_ST, L_ST = (1/BD) Σ_i Σ_d KL(ŷ_i,d ‖ ȳ_i).
• Test-time augmentation. At inference the posteriors of T scrambles with keys from the training key set are averaged.

Main results

Classification accuracy (%) without test-time augmentation (T = 1), for two classifiers. ScrambleMix is the best privacy-preserving input on every dataset.

Values from the PSIVT 2023 presentation slides (slide 39). The slides label the WideResNet as 40×10; the released code builds WRN-40-2.

With test-time augmentation, ScrambleMix with only T = 4 keys is compared with InstaHide at T = 10:

Values from slide 40.

Security. After the InstaHide attack of Carlini et al. (2020), a higher Inception Score means the attack recovered a more natural image. InstaHide jumps from 1.394 to 2.777 (+1.383); ScrambleMix only moves from 1.012 to 1.177 (+0.165), so the attack does not recover recognisable content (slide 41).

Use it

ScrambleMix, the self-teaching loss and multi-key test-time augmentation are a pip-installable package in the paper repository, which reproduces the ScrambleMix rows of the tables above.

pip install git+https://github.com/MADONOKOUKI/psivt23_scramblemix

import torch
from scramblemix import ScrambleMix, ScrambleMixViews, scramblemix_loss, predict_tta

sm = ScrambleMix.random(num_pairs=4, key_seed=0)      # secret key set: 4 key pairs, kept on the edge side
views = torch.stack([ScrambleMixViews(sm)(torch.rand(3, 32, 32)) for _ in range(8)])   # (B, D=4, 3, 32, 32)

model = torch.nn.Sequential(torch.nn.Flatten(), torch.nn.Linear(3 * 32 * 32, 10))     # any classifier
logits = torch.stack([model(views[:, d]) for d in range(views.shape[1])])             # (D, B, classes)
loss = scramblemix_loss(logits, torch.randint(0, 10, (8,)), lam=1.0)   # cross-entropy + self-teaching loss
pred = predict_tta(model.eval(), torch.rand(8, 3, 32, 32), sm, num_keys=4).argmax(1)   # multi-key TTA

git clone --depth 1 https://github.com/MADONOKOUKI/psivt23_scramblemix
cd psivt23_scramblemix && pip install -e .
python train.py --dataset cifar10 --model shakedrop   # ScrambleMix, ShakeDrop, CIFAR-10 (slides 39–40)

The repository ships the original key sets and reproduces the ScrambleMix rows of the tables above (scripts/reproduce_accuracy.sh).

Citation

@inproceedings{madono2024scramblemix,
  title     = {{ScrambleMix}: A Privacy-Preserving Image Processing for Edge-Cloud Machine Learning},
  author    = {Madono, Koki and Tanaka, Masayuki and Onishi, Masaki},
  booktitle = {Image and Video Technology (PSIVT 2023)},
  series    = {Lecture Notes in Computer Science},
  volume    = {14403},
  pages     = {326--340},
  publisher = {Springer Nature Singapore},
  year      = {2024},
  doi       = {10.1007/978-981-97-0376-0_25}
}

Related work in this series

• Block-wise Scrambled Image Recognition Using Adaptation Network (AAAI WS 2020)
• Scrambling Parameter Generation to Improve Perceptual Information Hiding (Electronic Imaging 2021)
• SIA-GAN: Scrambling Inversion Attack Using Generative Adversarial Network (IEEE Access 2021)