Finding Novel Knowledge

In the last post I talked about the Athens system, and how it could be used to discover things that a user didn’t already know about, but which are not too far away — in a sense, Athens imposes directions and distances on the web.

What could this be used for? Here are some of the possibilities:

  • For an organization, finding possibilities that they’re well positioned to handle but haven’t thought of doing yet. In one experiment we did with a large multinational, one participant commented on how much faster it was to use Athens as a kind of brainstorming than doing it the traditional way.
  • For an organization, finding possibilities that a competitor is well positioned to handle but hasn’t thought of doing yet. Athens has the capability to be restricted to a particular web site, so it can be used to analyze a competitor’s web site as a representation of what they already know that they know.
  • For a researcher, finding out what’s just beyond the state of the art.
  • For a learner, discovering the next topic(s) that are appropriate to study.

Many organizations, including those who make a living off discovering new ideas, do not have a structured way of looking for novel knowledge — they seem to rely on individuals — browsing or serendipity. Processes such as brainstorming are used, but it’s hard to tell how successful they are — business books, and Dilbert, are certainly full of counterexamples.

We’ve used Athens for a number of other interesting, less-obvious, kinds of searches. In one we exploited the ability provided by Google to limit searches to particular time ranges, and looked at what could potentially have been known about Osama bin Laden and al Qaeda on September 12th, 2001; and indeed most of what has appeared in books about him since could have been discovered from the web at that time — but not easily using conventional search. We have also discovered that the same search terms restricted to different countries provide interesting metainformation about the countries and how information/knowledge is organized in them. We’ve also found clusters that are obviously plausible clusters, but not ones that any human would ever think to construct.

Athens is available under a GPL; it uses Lingpipe which also imposes some constraints. Email me for details of how to get it.


0 Responses to “Finding Novel Knowledge”

  1. Leave a Comment

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


%d bloggers like this: