Research has revealed that more than 400 apps that passed App Store and Google Play review activated hidden features after approval. The apps can generate sexual and violent deepfakes from photos uploaded by users; these images can be used to humiliate or threaten victims. According to the research, a significant portion of the apps appear to belong to a single organization, and the same software was offered for sale in both major stores.
This touches the weakest point of app store moderation: software that looks innocent at the moment of review but changes its behavior after approval.
What happened?
The apps entered store review as harmless tools — a photo editor, an entertainment filter. Once approved and published, deepfake and violence-generation modules were switched on from the server side. In other words, the actual functionality wasn't in the code on the device, but behind a remote switch. These features were turned off while the review team tested them; they turned on once the user installed the app.
Why does it matter?
Apple's narrative has been built for years on the idea of a "walled garden": every app passes human and automated review, which is why the App Store is safer than the web. This incident exposes a structural gap in that narrative. Review inspects an app in the state it's in at the moment of approval. For an app that later changes its behavior from the server, that inspection is blind by definition. The store approves the binary it sees — not the remote configuration it doesn't.
This tactic isn't new. In the past, too, apps have shown one face in review and another in the wild: tools that opened gambling features based on geographic location, software that displayed a hidden screen on a certain date, examples that served a fake interface to reviewers via "region control." What makes the deepfake wave different is that generative AI has driven the cost down to near zero. Producing harmful content used to take effort; now a photo and a server call are enough. That's where the scale comes from — a single organization being able to produce hundreds of variants isn't a coincidence but the natural result of automation.
The second critical point: the same apps were also present on Google Play. This takes the problem out of the realm of an Apple-specific flaw. If two different review cultures, with two different automation infrastructures, fell into the same trap, the fault lies not with a single team but with the model itself. No static review that relies on server-side logic can fully see post-approval intent.
Who is affected, and how?
The heaviest price is paid by the people whose images are used without their consent. Next come the stores that have made the promise of safety a pillar of their marketing.
| Party | Impact |
|---|---|
| Victims | Non-consensual sexual/violent imagery, harassment and blackmail risk |
| Apple / Google | Loss of trust in the "reviewed = safe" narrative |
| Honest developers | Harsher, slower review processes |
| Regulators | Grounds for legal pressure on server-side moderation |
The timing matters too. The European Union's Digital Services Act imposes systemic risk auditing obligations on large platforms; debates over age verification and child safety are hardening in many countries. In this environment, hundreds of harmful apps that passed review hand regulators concrete grounds to say "self-regulation is insufficient." For the platforms, this is a matter not just of reputation but of liability.
What's next?
There are a few things to watch. The first is whether the stores will begin auditing post-approval behavior — that is, periodically sampling the server calls of published apps. The second is a tightening of policy toward apps that activate functionality on the server side; this could also affect developers running legitimate A/B tests. The third is mandatory disclosure and restrictions specific to AI generation modules.
The realistic assessment is this: human review is structurally at a disadvantage against an actor determined to hide its intent. The stores will likely be forced to move toward post-approval behavioral monitoring — which in turn creates a new tension with privacy and developer freedom. The "reviewed, safe" badge no longer means as much as it used to; the task now is to accept that and rebuild the model.
Yorumlar (0)
Henüz yorum yok. İlk yorumu sen yaz.
Yorum yapmak için Sinyal'i indir
Yorumlar Sinyal hesabıyla yapılır. Mobil uygulamada giriş yap, yorum bırak.