Researchers create privacy technique that protects sensitive data while maintaining performance

Imagine that a group of researchers has created a machine-learning algorithm that can determine from lung scan images if a patient has cancer. In order for clinicians to start utilising this model in diagnostics, they wish to share it with hospitals all around the world.

However, there is an issue. They trained their model by exposing it to millions of actual lung scan images in order to teach it how to anticipate cancer. It is possible for a hostile agent to extract the sensitive material that has been encoded into the model's internal workings. Scientists can avoid this by making it more difficult for an opponent to guess the original data by adding noise, or more general randomness, to the model. However, perturbation decreases a model's accuracy, therefore it is preferable to add as little noise as possible.

Researchers at MIT have now created a method that allows users to potentially inject the least amount of noise while yet maintaining the security of important data.

The researchers developed a framework based on a novel privacy metric they call Probably Approximately Correct (PAC) Privacy that can automatically estimate the least amount of noise that has to be added. Additionally, this framework is simpler to utilise for many models and applications because it does not require understanding of a model's internal workings or training procedure.

In a number of instances, the researchers demonstrate how much less noise is needed with PAC Privacy than with other strategies to secure sensitive data from adversaries. This might aid engineers in developing machine-learning algorithms that can indubitably conceal training data while retaining accuracy in practical contexts.

"PAC Privacy meaningfully uses the entropy or unpredictability of the sensitive data, allowing us to add, in many circumstances, an order of magnitude less noise. With the help of this framework, we can automatically privatise arbitrary data processing without making any unnatural changes. Although we are still in the early stages and working with simple cases, Srini Devadas, the Edwin Sibley Webster Professor of Electrical Engineering and co-author of a recent publication on PAC Privacy, is enthusiastic about the potential of this technique.

Together with graduate student in electrical engineering and computer science Hanshen Xiao, Devadas co-authored the study. The study will be presented on August 24 at Crypto 2023, an international conference on cryptology.

Understanding privacy

How much sensitive data may an adversary recover from a machine-learning model with noise introduced is a basic concern in data privacy.

According to the differential privacy definition of privacy, privacy is attained when an adversary seeing the model after it has been released cannot determine if any random person's data was utilised for the training process. However, it frequently takes a lot of noise to obfuscate data usage to prove that an adversary cannot differentiate it. The precision of the model is decreased by this noise.

PAC Privacy approaches the issue quite differently. Instead than concentrating just on the distinguishability issue, it characterises how difficult it would be for an adversary to reassemble any portion of randomly selected or created sensitive data once noise has been added.

For instance, differential privacy would concentrate on whether the adversary could discern if a specific person's face was in the dataset if the sensitive data were photos of human faces. On the other hand, PAC Privacy may examine if an opponent could extract a silhouette—an approximate—that someone could identify as a specific person's face.

In order to prevent an adversary from confidently reconstructing a close approximation of the sensitive data, the researchers developed an algorithm that automatically instructs the user how much noise to add to a model. According to Xiao, this technique ensures anonymity even if the adversary has unlimited computational power.

The PAC Privacy algorithm uses the entropy, or uncertainty, in the initial data from the adversary's point of view to determine the ideal quantity of noise.

The user's machine-learning training algorithm is run on the subsampled data using this automatic technique, which randomly selects samples from a data distribution or a huge data pool to create an output learnt model. This is repeated on several subsamples, and the variance of all outputs is compared. The variance determines the amount of noise that must be added; a lower variance indicates a need for less noise.

algorithm benefits

In contrast to other privacy strategies, the PAC Privacy algorithm does not require understanding of a model's internal workings or the training procedure.

A user can initially indicate their preferred level of trust when using PAC Privacy. For example, a user might desire a guarantee that an enemy won't be more than 1% certain that they've successfully rebuilt the sensitive data to within 5% of its true value. In order to accomplish those aims, the PAC Privacy algorithm automatically informs the user of the ideal amount of noise that should be added to the output model before it is shared publicly.

"The noise is ideal in that if you add less than we advise, everything could go wrong. However, the impact of adding noise to neural network parameters is complex, and we cannot guarantee that the model's utility would not decline as a result of the added noise, according to Xiao.

This highlights a drawback of PAC Privacy: the method doesn't inform the user of the accuracy the model will lose whenever noise is introduced. PAC Privacy can be computationally expensive since it requires repeatedly training a machine-learning model on numerous data subsamplings.

One method for enhancing PAC Privacy involves altering a user's machine-learning training procedure so that the output model it generates does not change significantly when the input data is subsampled from a data pool. The PAC Privacy algorithm would need to run fewer times to determine the ideal amount of noise, and it would also need to add less noise as a result of this stability, which would result in less deviations between subsample outputs.

Devadas continues, "An additional advantage of more stable models is that they frequently have less generalisation error, meaning they can make more accurate predictions on data that hasn't been seen before. This is a win-win situation for machine learning and privacy."

We would love to delve a little more deeply into the connections between stability and privacy in the next years, as well as the connections between privacy and generalisation error. Although it is not yet evident where the door goes, we are knocking on it here.