On-device scam detection and privacy architecture in messaging

As automated scams and social engineering tactics evolve, maintaining user security while preserving the foundational guarantees of end-to-end encryption remains a central challenge for messaging platforms. A newly proposed technical approach introduces an optional, on-device feature designed to alert users to potential scam messages entirely on mobile hardware.

Design Principles

Recent improvements in mobile machine learning make it feasible to run text classification locally without incurring severe battery, performance, or model-size penalties. The architecture of this system rests on three strict principles to maintain privacy:

• On-device only: Both the machine learning model and the message data remain entirely on the user hardware.
• No automatic reporting: Message content never leaves the device automatically. Data is only shared if a user explicitly chooses to report a conversation.
• User control: The feature is optional, allowing users to enable or disable protection at any time.

How Scam Alert Works

When activated, the system downloads a lightweight machine learning model to the device. This model analyzes incoming messages from non-contacts, evaluating conversational structure and linguistic signals against known scam patterns. If a message is flagged as a likely scam, a private warning appears within the chat interface, remaining invisible to the sender.

Users retain full autonomy over the outcome. They can block the sender, report the message, or mark the chat as trusted if the warning is a false positive. Marking a chat as trusted also gives users the option to share the last five received messages to help refine future model accuracy.

Foundational Safeguards

To ensure transparency and prevent systemic abuse, the implementation incorporates robust engineering safeguards. All model weights are published publicly on transparency ledgers prior to deployment, ensuring that neither the platform nor external actors can silently deliver targeted models to specific individuals.

Minimal telemetry, such as aggregate warning counts and user action frequencies, is processed using confidential virtual machines built on trusted execution environments. By applying differential privacy and secure aggregation techniques, the system ensures that population-wide performance metrics can be measured while keeping individual user data strictly inaccessible.