1.  “Intelligence analysts” don’t usually interact with datasets directly, but rather via “data analysts”, who aren’t considered “real” analysts. I’m told that, at least in Canada, you have to have a social science degree to be an intelligence analyst. Unsurprisingly (at least for now) people with this background don’t have much feel for big data and for what can be learned from it. Intelligence analysts tend to treat the aggregate of the datasets and the data analysts as a large black box, and use it as a form of Go Fish. In other words, intelligence analysts ask data analysts “Have we seen one of these?”; the data analysts search the datasets and the models built from them, and writes a report giving the answer. The data analyst doesn’t know why the question was asked and so cannot write a more helpful report that would be possible given some knowledge of the context. Neither side is getting as much benefit from the data as they could, and it’s mostly because of a separation of roles that developed historically, but makes little sense.

2. Intelligence analysts, and many data analysts, don’t understand inductive modelling from data. It’s not that they don’t have the technical knowledge (although they usually don’t) but they don’t have the conceptual mindset to understand that data can push models to analysts: “Here’s something that’s anomalous and may be important”; “Here’s something that only occurs a few times in a dataset where all behavior should be typical and so highly repetitive”; “Here’s something that has changed since yesterday in a way that nothing else has”. Data systems that do inductive modelling don’t have to wait for an analyst to think “Maybe this is happening”. The role of an analyst changes from being the person who has to think up hypotheses, to the person who has to judge hypotheses for plausibility. The first task is something humans aren’t especially good at, and it’s something that requires imagination, which tends to disappear in a crisis or under pressure. The second task is easier, although not something we’re necessarily perfect at.

There simply is no path for inductive models from data to get to intelligence analysts in most organizations today. It’s difficult enough to get data analysts to appreciate the possibilities; getting models across the chasm, unsolicited, to intelligence analysts is (to coin a phrase) a bridge too far.

Addressing both of these problems requires a fairly revolutionary redesign of the way intelligence analysis is done, and an equally large change in the kind of education that analysts receive. And it really is a different kind of education, not just a kind of training, because inductive modelling from data seems to require a mindset change, not the supply of some missing mental information. Until such changes are made, most intelligence organizations are fighting with one and a half arms tied behind their collective backs.

The idea cannot possibly work on technical grounds. More worryingly, it displays a lack of awareness of the realities of cybersecurity that is astounding.

The first peculiarity is that the Centre is supposed to address four problems: email spam, unsolicited phone calls, fake communications a la Facebook, and malware. Although these have a certain superficial similarity — they all annoy individuals — they do not raise the same kinds of technical issues underneath, and no one person could be an expert in detecting, let along prosecuting all of them. It’s a bit like trying to amalgamate the Salvation Army and the police force because they both wear uniforms and help people!

The Centre will rely on reports from individuals: get a spam email and forward it to the Centre, for example. One of the troubles with this idea is that individuals don’t usually have enough information to report such things in a useful way, and they don’t make good starting points for an eventual prosecution. Canada already has a way to report unsolicited phone calls but it only works for people who almost keep the law by announcing who they are at the beginning. The annoying (and illegal) robocalls can’t be reported because the person who gets them doesn’t know where they are coming from and who’s making them. And where there are prosecutions, each person who reports such a call has to sign an affidavit that the purported call did actually happen to provide the legal basis for the incident.

The second, huge, problem with this idea is that, if individuals can report bad incidents, then spammers can also report fake bad incidents! And they can do it in such volume that investigators will have no way to distinguish the real from the fake. Creating fake spam emails and evading mechanisms such as captchas to prevent wholesale reporting  is very easy.

There is also the deeper problem that besets all cybersecurity — attribution. It is always hard to trace cyberexploits back to their origins, and these origins are overwhelmingly likely to be computers taken over by botnets anyway. Working back along such chains to find someone to prosecute is tedious and expert work that depends on starting from as much information as possible.

The right way to address this problem is to set up honeytraps — machines and phones that seem to be ordinary but are instrumented so that, when an exploit happens, as much information as possible is collected at the time. Now there is a foundation for deciding which incidents are worth pursuing and starting out in pursuit with the best possible information. And, who knows, the knowledge that such systems are out there might dampen some of the enthusiasm on the part of the bad guys.

Thinking about this, I realized that it’s another facet of scholarship. One of the (mostly invisible) things that research-active academics do is to take ideas and results from the cutting edge of knowledge and digest, rework, and refactor them to extract the key aspects and get rid of the unavoidable autobiography that goes along with research results. This is mostly why an undergraduate education is better at a research-intensive university. And why undergraduate degrees don’t keep getting longer, although the quantity of stuff we know is growing rapidly.

But there’s another aspect to this which I, at least, had under-rated. That is the relentless, year after year presentation of “the same” ideas to a a new crop of students who don’t already understand them. Of course, the ideas themselves are not the same from year to year — I taught the same course for twenty years, but I was still improving it and seeing implications that had previously escaped me. But the work of presenting the ideas over and over again, by itself, forces me to rethink and say things in different (and better) ways each time. This output side of scholarship was something I had not really appreciated enough. And I think it’s the explanation for why academics are better at communicating to a mixed audience than those who only ever talk to people from the same backgrounds.

I had always felt that working in a research lab rather than a university would be a more sterile experience, and now I think that I understand why. So this post is really a big thank you to all of my students, undergraduates and graduates, for making it possible for me to explain to you. and in the process understand better myself. And to those who got the earlier versions, my apologies, but it simply isn’t possible to polish first and then teach.

This view isn’t accurate, for the following reason: mainstream/ordinary knowledge discovery builds models inductively from data by maximizing the fit between the model and the available data. In an adversarial setting, this creates problems: adversaries can make a good guess about what your model will look like, and so can do better at hiding their records (making them less obvious or detectable); and they can also tell exactly what kinds of data they should try and force you to use if/when you retrain your model.

A simple example. The two leading-edge predictors today are support vector machines and random forests and, in mainstream settings, there’s often little to choose between them in prediction performance. However, in an adversarial setting, the difference is huge: support vector machines are relatively easy to game, because even one record that’s in the wrong place or mislabelled can change the entire orientation of the decision surface. To make it even worse, the way in which the boundary changes is roughly predictable from the kind of manipulation made. (You can see how this works in: J. G. Dutrisac, David B. Skillicorn: Subverting prediction in adversarial settings. ISI 2008: 19-24.). Random forests, on the other hand, are much more robust predictors, partly because their ensemble characteristics makes it hard to force particular behaviour because of the inherent randomness ‘inside’ the algorithm.

The same thing happens with clustering algorithms. Algorithms such as Expectation-Maximization are easily misled by a few carefully chosen records; and no mainstream clustering technique does a good job of finding the kinds of ‘fringe’ clusters that occur when something unusual is present in the data, but adversaries have tried hard to make it as usual as possible.

In fact, adversarial knowledge discovery requires building the entire framework of knowledge discovery over again, taking into account the adversarial nature of the problem from the very beginning. Some parts of mainstream knowledge discovery can be used directly; others with some adaptation; and others can’t safely be used. There are also some areas where we don’t know very much about how to solve problems that aren’t interesting in the mainstream, but are critical in the adversarial domain.

In the political arena, factual deception, saying things that just ain’t so, plays very little role, perhaps because voters have very low expectations of politicians in this area. What we call persona deception, presenting oneself as a better,wiser,  more powerful, more able, more knowledgeable person than one really is is the heart of successful campaigning. It turns out that the deception model captures deception across the whole range from factual to persona deception, so it gives us a lens to look at candidates and campaigns. What’s more, because language generation is almost entirely subconscious, this lens is hard to fool.

The most important skill candidates and their campaigns have is the ability to reach out to potential voters to convince them that they are better than the other possibilities. The language that they use is an important channel, especially in settings where everyone is conservatively dressed, and standing behind a podium that conceals most of their body language, as the Republican presidential field was in Iowa yesterday.

Strong candidates understand, at least instinctively, that they are not making arguments to convince voters, but presenting themselves as more compelling human beings. Our analysis of the speeches of candidates in the 2008 U.S. presidential election showed that candidates use three different kinds of speeches: blue skies speeches that promise generically good things and could be delivered interchangeably by any candidate – they are aimed at a wide audience; track record speeches that use past achievements to imply special qualifications for future achievements – they are aimed at swing voters; and manifesto speeches that describe a candidate’s personal qualities directly – they are aimed at a candidate’s base and reinforce common identity. But in all three cases, it’s not the content of the speech that matters, but what it implies about the speaker.

Our analysis in the last election cycle showed that Obama was by far the best as presenting himself as a wonderful person, and many voters, and certainly many in the media, projected onto the persona  positive qualities that were perhaps not there. Interestingly, yesterday was the first time I have seen open Democratic buyers remorse about electing Obama, something I predicted would happen from the analysis we did.

The Republican candidates’ debate in Ames showed what a shaky grasp many of the candidates have on how to be a convincing candidate. Of course, this venue was a difficult one. Its overt purpose was for candidates to explain themselves to the local Republican base ahead of the Ames Straw Poll,which would have required largely manifesto content; but national television coverage made it an unmissable opportunity to reach out to a wider, but much more diverse audience, suggesting track record content. Blue skies content is always dangerous in the early stages of a campaign because grand but potentially unwise statements can come back to haunt a candidate.

Manifesto content was indeed popular – for example, we learned how many children almost every candidate has – typical content aimed at the base (“I’m a parent just like you”). Several candidates also tried for track record content, but got it quite wrong. The purpose of a track record speech is not for candidates to read their resumes to the audience; it’s to make the argument “I was able to do A, so you can trust me to be able to do similar-but-larger B” and this second part was notably absent.

Voters also want candidates to be sincere — recall the famous quotation “The secret of success is sincerity. Once you can fake that you’ve got it made” (Jean Girardoux). This is not just a cute quotation; this is what good politicians are able to do. In Iowa, this was another area where almost everyone stumbled. It was clear that most of the candidates had not only prepared talking point responses to probable questions, but has also rehearsed actual answers. Delivering from a prepared and memorized script and seeming sincere is a difficult business, and actors who can do it reliably command high rewards.  Most of the candidates failed at seeming sincere. Several managed the worst of both worlds by trying to combine their prepared scripts with some ad libbing and came across as quite incoherent. One of the reasons for Gingrich’s strong showing is that he stayed away from scripts and delivered his answers as if he had just thought of them. Huntsman and Romney, in contrast, were especially wooden.

When humans listen to humans, the content matters. But when character is the issue, other aspects of language matter more. Much language generation is subconscious, and therefore beyond a candidate’s control. This is good for voters because it means we can sometimes see through to the real person no matter how sophisticated their speech writers and spin doctors.

The thing that has gotten attention about this report is that it describes attacks by a single attacker and single vector that have lasted over a long period of time (more than 5 years) and have targeted governments, quasi-government organizations, and businesses in a sophisticated way. The attacks are being attributed to a particular state actor, for obvious reasons, but attribution is always murky in cyberspace so it’s (just) conceivable that someone else is responsible and covering themselves.

It was helpful to get this kind of information out into the public awareness. People on the inside (for several different values of inside) have known about these kinds of attacks, their frequency, and their huge impact for some time; but either haven’t wanted to or haven’t been allowed to reveal them.

The attacks themselves seem to have begun with a spear phishing attack on some mid-level person at each organization, so relatively unsophisticated but requiring substantial preliminary research. I’m not aware of any attempt to measure how easily spear phishing attacks work, but presumably with patience to try with enough spacing that nobody mentions it, not very much personalization is required. Once in, the attacks seem to have been quite sophisticated and long-lasting. Even after the report came out, several of the organizations were denying that they had been hacked. “Hacked and unaware” seems more likely than “not hacked” given that McAfee could see the IP logs.

Of course, since this is only one vector, it would be naive not to suppose that a number of other, broadly similar attacks are going on with other sources and vectors.

I did a fairly large number of media interviews about this report, and the obvious and common question that came up was: what did these organizations do wrong and what can be done to protect against these kinds of attacks? That’s a hard question to answer. Malware detection tools are in their infancy so, while running them is a good idea, they may not protect against sophisticated attacks very well. It doesn’t seem possible to protect completely against spear phishing, given the convenience of attachments. I received a number of emails from companies who claim that their approach/tools would have protected these organizations, but I didn’t see anything that looked like a substantial advance in the state of the art.

It may be that the time has come to do what the military and intelligence organizations do — to run separate networks that do not connect to the internet for anything that needs to be protected. This is, of course, relatively painful; and still not necessarily secure since data and software still need to be walked from one network to the other. But many organizations may need to take partial steps towards this kind of robust physical separation, since virtual separation is not working. In other words, firewalls don’t get the job done.

The problem is that, like many such discussions, it’s based on the autobiographies of a number of people who became extremists — the idea is to look for commonalities in such biographies as hints about the process and/or drivers of extremism.

BUT it ignores the very large number of people from apparently identical backgrounds who didn’t join gangs, or the IRA, or jihadist groups! Such people are counterexamples to almost all explanations of what happens with radicalization, and yet they are often/usually ignored in the discussion.

So Google asks:

“Why does a 13-year old boy in a tough neighborhood in South Central LA join a gang? Why does a high school student in a quiet, Midwestern American town sign on neo-Nazis who preach white supremacy? Why does a young woman in the Middle East abandon her family and future and become a suicide bomber?”

But just as important are questions like: why did the 13-year old boy’s best friend and classmate NOT join a gang, etc.

This summit’s approach is called, in the research community, “sampling on the dependent variable”. Google should know better.

The number of LinkedIn contacts I have is now large enough that I can tell this story. I know someone from one of the more secretive US government organizations. His (or it might be her) public web presence, of course, has nothing at all to do with his day job, and we’ve never exchanged emails using his public email. Yet LinkedIn suggests him as someone I might possibly know.

The reason must be that we have enough mutual connections that the software LinkedIn uses sees that there “should” be some connection between us — it is doing edge prediction. This is exactly the kind of analysis that good intelligence tools can do on relational/graph data. The knowledge is in the links, collectively; in other words, noticing the potential link between us requires knowing both the presence of some links and the absence of others (because the system doesn’t recommend other people whose web presence is as dissimilar from mine as his is).

So, well done LinkedIn, but a cautionary tale for security folks generally, and especially those who believe in anonymization — it can’t be done!

The topic that was missing from all of the discussion was what social media was, and why it is the way it is; and so I came away feeling like the entire discussion, and quite a lot of work, was dancing on clouds. There seem to be a number of things that “everybody knows” about social media, but for which there seems to be little or no evidence. The Arab Spring was driven by social media! Well, maybe, but (a) was it and how much, (b) which parts were important and which were irrelevant?

It seems helpful to divide social media into three categories:

1.  Media that is essentially public access publishing or public access (micro)blogging. Although sites that provide this kind of functionality are often considered “social” there is almost nothing social about them — yes, the audience for posts can be restricted to a particular group, but that’s always been true of any publication. There is an interesting question lurking here though: what are the reasons why individuals read such posts? What kind of bond does it imply between the reader and the author? (Cynically, why would I care what even my closest friend had for breakfast?)

2. Media that start as public access publishing, but where the conversation built on an initial post is more important or interesting than the initial post itself — in other words, there’s something emergent in the conversation that transcends what any of the participants would have said ab initio. This is a kind of social knowledge or opinion construction, and there are lots of interesting questions about who participates, what their roles are, and how the content and tone are affected by the interactions. This is, of course, not a new phenomenon but what’s new is the scope and the detail of what’s recorded, allowing answers to be worked out in ways that were impractical or too expensive before.

3. Media in which explicit relational links are created between one person and another. This is the real heart of social media. Relational links between a pair of people have, of course, always existed, but they could only be constructed in a small number of ways and were (almost always) limited by geography.

The emergent structure of these links is a really interesting artifact that deserves study and from which we will probably learn a lot about what it means to be human in a global society. What does it mean when one person “friends” another? This is one question for which simple answers tend to be assumed, but even a brief consideration of A’s Facebook friends and the rest of A’s relationships in the real world quickly shows that there’s a complex connection between the two sets (and it depends heavily on characteristics of A).

One thing that quickly becomes clear when these questions are addressed computationally is that we aren’t going to get far until relationship links are typed. It’s fairly easy to look at each relationship and give it a numerical weight that reflects (say) closeness — but it’s still true that different kinds of relationships behave differently, and need to be modelled differently to understand them. (Social media sites should also implement this typing — not every piece of data should flow down every link of A’s social network.)

The fundamental question in a world where one person can create a visible relationship, is what does this mean — for the person creating it, for the person at the other end of the relationship, and for the emergent graph structure that a collection of these individual relationships creates. Good, solid answers to this question would be a foundation on which much more useful applications could be built.