Malevolent users

Malevolent users come in all sizes, be they crackers, advertising mailing list marketers, or disgrunted co- or ex-employees. Any system used for real purposes must be protected against them.

Consider Ringo, which handles relatively harmless information (individual users' musical tastes), but even this information could be a problem if users had to be identified. For example, users could be the subject of flamage from others solely on the basis of their musical tastes; advertisers could specially target users who like groups carried by particular labels.) For reasons like this, Ringo keeps users anonymized, unless they sign their own comments with their own name.

Even worse, if individual agents communicate with each other (as opposed to Ringo's, model of one agent for many users), anonymity alone is not a protection without more sophisticated mechanisms, given malevolent users.

Since we are talking about building, for example, matchmaking agents which are getting personal information by, e.g., scanning people's email, concerns about malevolence and user privacy must be paramount. It is commonly accepted in the operating system and computer security disciplines that adding privacy or security mechanisms after the bulk of the system has been designed is always a mistake---such additions are difficult to make, fragile and likely to break, and usually interfere with legitimate usage of the system to such an extent that they are routinely disabled or circumvented by even nonmalicious users. (Many anecdotes and war stories about such attempts are available by perusing Risks Digest.) It thus behooves us to make sure that adequate protections are designed in from the start, before it's too late.

In addition to the discussion above, of course, it should be noted that it is not only malevolent users which create sociological, and technical reasons for implementing good security, of course. There are also political reasons which argue for such protections, even if malevolence is discounted.