‘AI’ performance not what it seems

As I’ve written about before, ‘AI’ tends to be misused to refer to almost any kind of data analytics or derived tool — but let’s, for the time being, go along with this definition.

When you look at the performance of these tools and systems, it’s often quite poor, but I claim we’re getting fooled by our own cognitive biases into thinking that it’s much better than it is.

Here are some examples:

• Netflix’s recommendations for any individual user seem to overlap 90% with the ‘What’s trending’ and ‘What’s new’ categories. In other words, Netflix is recommending to you more or less what it’s recommending to everyone else. Other recommendation systems don’t do much better (see my earlier post on ‘The Sound of Music Problem’ for part of the explanation).
• Google search results are quite good at returning, in the first few links, something relevant to the search query, but we don’t ever get to see what was missed and might have been much more relevant.
• Google News produces what, at first glance, appear to be quite reasonable summaries of recent relevant news, but when you use it for a while you start to see how shallow its selection algorithm is — putting stale stories front and centre, and occasionally producing real howlers, weird stories from some tiny venue treated as if they were breaking and critical news.
• Self driving cars that perform well, but fail completely when they see certain patches on the road surface. Similarly, facial recognition systems that fail when the human is wearing a t-shirt with a particular patch.

The commonality between these examples, and many others, is that the assessment from use is, necessarily, one-sided — we get to see only the successes and not the failures. In other words (HT Donald Rumsfeld), we don’t see the unknown unknowns. As a result, we don’t really know how well these ‘AI’ systems really do, and whether it’s actually safe to deploy them.

Some systems are ‘best efforts’ (Google News) and that’s fair enough.

But many of these systems are beginning to be used in consequential ways and, for that, real testing and real public test results are needed. And not just true positives, but false positives and false negatives as well. There are two main flashpoints where this matters: (1) systems that are starting to do away with the human in the loop (self driving cars, 737 Maxs); and (2) systems where humans are likely to say or think ‘The computer (or worse, the AI) can’t be wrong’; and these are starting to include policing and security tools. Consider, for example, China’s social credit system. The fact that it gives low scores to some identified ‘trouble makers’ does not imply that everyone who gets a low score is a trouble  maker — but this false implication lies behind this and almost all discussion of ‘AI’ systems.

Advertisements

China-Huawei-Canada fail

Huawei has been trying to convince the world that they are a private company with no covert relationships to the Chinese government that might compromise the security of their products and installations.

This attempt has been torpedoed by the Chinese ambassador to Canada who today threatened ‘retaliation’ if Canada joins three of the Five Eyes countries (and a number of others) in banning Huawei from provisioning 5G networks. (The U.K. hasn’t banned Huawei equipment, but BT is uninstalling it, and the unit set up jointly by Huawei and GCHQ to try to alleviate concerns about Huawei’s hardware and software has recently reported that it’s less certain about the security of these systems now than it was when the process started.)

It’s one thing for a government to act as a booster for national industries — it’s another to deploy government force directly.

China seems to have a tin ear for the way that the rest of the world does business; it can’t help but hurt them eventually.

Lessons from Wannacrypt and its cousins

Now that the dust has settled a bit, we can look at the Wannacrypt ransomware, and the other malware  that are exploiting the same vulnerability, more objectively.

First, the reason that this attack vector existed is because Microsoft, a long time ago, made a mistake in a file sharing protocol. It was (apparently) exploited by the NSA, and then by others with less good intentions, but the vulnerability is all down to Microsoft.

There are three pools of vulnerable computers that played a role in spreading the Wannacrypt worm, as well as falling victim to it.

1. Enterprise computers which were not being updated in a timely way because it was too complicated to maintain all of their other software systems at the same time. When Microsoft issues a patch, bad actors immediately try to reverse engineer it to work out what vulnerability it addresses. The last time I heard someone from Microsoft Security talk about this, they estimated it took about 3 days for this to happen. If you hadn’t updated in that time, you were vulnerable to an attack that the patch would have prevented. Many businesses evaluated the risk of updating in a timely way as greater than the risk of disruption because of an interaction of the patch with their running systems — but they may now have to re-evaluate that calculus!
2. Computers running XP for perfectly rational reasons. Microsoft stopped supporting XP because they wanted people to buy new versions of their operating system (and often new hardware to be able to run it), but there are many, many people in the world for whom a computer running XP was a perfectly serviceable product, and who will continue to run it as long as their hardware keeps working. The software industry continues to get away with failing to warrant their products as fit for purpose, but it wouldn’t work in other industries. Imagine the discovery that the locks on a car stopped working after 5 years — could a manufacturer get away with claiming that the car was no longer supported? (Microsoft did, in this instance, release a patch for XP, but well after the fact.)
3. Computers running unregistered versions of Microsoft operating systems (which therefore do not get updates). Here Microsoft is culpable for an opposite reason. People can run an unregistered version for years and years, provided they’re willing to re-install it periodically. It’s technically possible to prevent (or make much more difficult) this kind of serial illegality.

The analogy is with public health. When there’s a large pool of unvaccinated people, the risk to everyone increases. Microsoft’s business decisions make the pool of ‘unvaccinated’ computers much larger than it needs to be. And while this pool is out there, there will always be bad actors who can find a use for the computers it contains.

Advances in Social Network Analysis and Mining Conference — Sydney

This conference will be in Sydney in 2017, from 31st July to 3rd August.

http://asonam.cpsc.ucalgary.ca/2017/

As well as the main conference, there is also a workshop, FOSINT: Foundations of Open Source Intelligence, which may be of even more direct interest for readers of this blog.

Also I will be giving a tutorial on Adversarial Analytics as part of the conference.

Even more security theatre

I happened to visit a consulate to do some routine paperwork. Here’s the security process I encountered:

1. Get identity checked from passport, details entered (laboriously) into online system.
2. Cell phone locked away.
3. Wanded by metal detection wand.
4. Sent by secure elevator to another floor, to a waiting room with staff behind bullet-proof glass.

Here’s the thing: I got to carry my (unexamined) backpack with me through the whole process!

And what’s the threat from a cell phone in this context? Embarrassing pictures of the five year old posters on the wall of the waiting room?

I understand that government departments have difficulty separating serious from trivial risks, because if anything happened they would be blamed, regardless of how low-probability the risk was. But there’s no political reason not to make whatever precautions you decide to take actually helpful to reduce the perceived risks.

“But I don’t have anything to hide” Part III

I haven’t been able to verify it, but Marc Goodman mentions (in an interview with Tim Ferriss) that the Mumbai terrorists searched the online records of hostages when they were deciding who to kill. Another reason not to be profligate about what you post on social media.

Public vs Private/Metadata vs Data/Modelled vs Unmodelled

Governments, intelligence organizations, and law enforcement have traditionally made a distinction between collecting data about what I do in public spaces, and what I do in private spaces. These were originally conceptualized in geographical terms: on the street (public) vs inside my house (private). Even in these terms, issues quickly arise: what about what I am doing in front of a window? What if there is a CCTV camera that points at the space outside my house, but also happens to capture a view into the house? Is the fact that the snow has melted on my roof but not my neighbors grounds for a search warrant?

The beginning of electronic communication changed the venue, but didn’t change the framing of the boundary between public and private: the content of a communication is typically considered to be private, but its envelope, how and when it was sent and to whom, is typically considered to be public. Enter the distinction between data (the contents) and metadata (the envelope).

Note in passing that the multinational internet organizations did not, and do not, acknowledge the existence of any boundary between public and private. Their avowed intent is to collect and organize all the world’s information. And, by and large, people have been willing not only to permit this, but to actively help. (It’s not clear to me how much of this should be attributed to ignorance and how much to foolhardiness.)

What’s changed in the past (say) decade is a growing public awareness of how much metadata about them there is, and how easily it is collected and stored in today’s world. However, in my view, to frame the problem as one of data is to get into a muddle.

The conventional view of privacy is that it’s about information flow: who is allowed to collect and retain my information. Privacy advocates, therefore, want to put barriers in the way of collecting “personal information”. But what is “personal information” and how does it fit into the data/metadata distinction? Much of the discussion today jumps off from the realization that some metadata can naturally be considered personal information. So the proposed solutions are all about limiting information collection practices, and preserving the integrity of data once collected. However, this attack on the problem doesn’t provide a rationale for which data should be treated as personal; and the rationale needs to be widely agreed and self-evidently sensible if it is to affect government choices, and the wider discussion of the balance between security and privacy.

It is more helpful to consider privacy not as a scalar property that I either have none of, some of , or lots of, but rather as a property of a relationship. A has privacy with respect to B is B is unable to build a useful model of A. “Unable” might mean “physically unable”, but it might also mean “legally unable”.

So I might have little privacy with respect to my spouse, and to Google, and to foreign governments; more with respect to my work colleagues; and even more with respect to the Canadian (and other Five Eyes) government. (Yes, that really is the order.)

This definition reveals aspects of privacy that information containment definitions cannot. Yes, if B cannot gather information about me, then B cannot model me usefully, so my definition subsumes existing ones.

But B also cannot build a useful model of me if B lacks the algorithmic and computational resources to actually do the modelling. This was the basis of practical privacy even as recently as 30 years ago — if you wanted to build a decent model of someone you either had to live with them, or you had to physically travel around to various information sources gathering small pieces of information at each one. It was at least partially feasible to model a single individual, but it didn’t scale. (This, of course, is how it was possible to insert agents into other countries under deep cover, a practice that is much, much difficult today than it was during the Cold War.)

B also cannot build a useful model of me if the data about me is massively contradictory. So one approach to preserving my privacy, which does not jump out from conventional information flow views, is to increase the flow of information about me that is available in the real world, but make sure that the vast majority is untrue. Of course, this would be a tedious process if it had to be done manually, but the availability of computation helps individuals as well as modellers. To take a simple example, it would not be hard to create multiple email addresses and have your mail client randomly choose which one to use to send each outgoing message. It wouldn’t be too hard to aggregate this information and still build a model, but it’s a start. Now imagine that your mailer generates multiple versions of each email you send with the contents subtly changed (this is within the state of the art — see the Frankenstein malware) and sent to multiple similar receivers, only one of which is real. The problem of finding out who said what to whom becomes much more difficult very quickly.

People need to be able to operate, at least some of the time, in private: businesses making pricing decisions, inventors creating intellectual property, governments building budgets. It seems to me that there are only really two alternatives for the near future:

1. Build enclaves within the present communication/computation infrastructure that are much harder to penetrate, but for which the true costs are paid (i.e. without the ad subsides that provide much of our “free” infrastructure today). This would require the infrastructure providers to assume liability and charge accordingly. It’s not clear this is doable, but it might be.
2. Learn to do private activities  “out in the open” using techniques such as fake replicants, since it is much easier for an insider to identify the genuine one than it is for an outsider (at least within limited time frames).

In the shorter term, the discourse on privacy would make more sense if we expand the discussion from information flow management to the bigger issue of modelling.