Does it help bad guys to know how knowledge discovery works?

The temptation faced by people who develop new ways to carry out knowledge discovery is to keep it all secret so that the bad guys can’t figure out what’s being done and so evade detection.

From the discussion in the last three posts I hope that you can see the weakness of this intuitively-appealing idea. First of all, security by obscurity — that is keeping the details of a system hidden as a way of keeping it secure — is just a bad idea. The problem is that you, the good guy, don’t necessarily find out when the bad guys figure out your system, and so they can lead you by the nose until you do. And no matter how obscure you try to be, the bad guys have a strong motivation to try and figure out your system and so, in the end, they will.

Security by obscurity does help to find amateurs and incompetents, but there are so few of these in settings where it matters that it’s not a good idea to treat them as the main problem.

In fact it’s a good idea if the bad guys know something about the kind of knowledge discovery that is being used against them. Why?? Because it encourages them to try and use concealment and manipulation; and these, as we have seen, create signatures that often make them easier to discover than if they had not bothered.

Pickpockets used to use this technique. In a crowd, one member of the gang would yell out “Beware of pickpockets” (note the same counterintuition). The result: everyone would put their hands on their wallets, showing the other members of the gang where they were. As good guys, we can use the same technique. Every knowledge discovery system is improved by treating it as a two-stage problem. The first stage is something like a big sign saying “Knowledge discovery in progress”‘ and the second stage is actual knowledge discovery, tuned to watch for concealment and manipulation.

That’s not to say that bad guys need to know everything about how a knowledge-discovery system works. One good way to introduce uncertainty is to include some randomization, so that the results for the same record might vary from time to time. This makes it hard for a bad guy to learn the exact attribute values that will cause problems.

Although randomization has some attractive properties, it is politically difficult because it means that some records escape scrutiny that they might have received on other occasions. If one of these records turned out to be a significant false negative, there would be repercussions.


0 Responses to “Does it help bad guys to know how knowledge discovery works?”

  1. Leave a Comment

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


%d bloggers like this: