Tree of knowledge
In this age of information overload the time we spend searching is growing.
Campbell McCracken asks: what can we do about it? And are taxonomies a possible route to take?
Information overload
Organisations are facing information overload. The amount of information that employees have to
search through to find what they are looking for is continually growing. "It has been estimated
that there are 2.5 billion documents directly accessible on the internet," says Alan Gilchrist,
Senior Associate Consultant at TFPL Ltd. "And there are a further 520 billion documents publicly
accessible in other databases."
The problem is exacerbated by the fact that an estimated 50 per cent of users are not sufficiently
skilled in formulating good search engine queries. It's not surprising, therefore, that one
solution that is finding favour with a number of large corporations is to move away from simple
keyword searches and to use a more organised approach. This is where taxonomies come in.
What is a taxonomy?
A taxonomy can be defined as a classification of elements within a domain. It's a scheme for
creating a set of labeled boxes, into which you can consistently categorise things. In simple
cases, a taxonomy can be visualised as an inverted tree, with the root at the top. Each node at
a given level is connected to a node in the level above it. The nodes at the lowermost levels
are called leaves. The path from the root to the leaf is called a branch and is generally
unique. The category boxes sit at the leaves. In an ideal world, the taxonomy should cover paper
and electronic documents, audio-visual material, people, and all intellectual capital within an
enterprise.
One benefit of the taxonomy approach over a straight search is that you can find accurate
information much more quickly. This is because at each level in the tree you are presented with
a number of choices which help to refine your initial query, guiding you to the category that
most accurately matches the one you want. It's all about helping you to get to the right
information quickly.
"AT&T disclosed last year in the B2B conference in San Diego that using our taxonomies instead
of a basic search saved them $2 million in one year," said Semio's founder and CEO Claude Vogel.
"This equates to a saving of roughly half an hour a day for every employee using this kind of
technology intensively."
Taxonomies can be used in different ways to make the searching of information more efficient.
Wordmap Ltd offers taxonomies as a broker between researchers and the information they are
searching for. "A study of one billion search queries on the AltaVista web search engine showed
that they contained an average of 2.35 words," says Wordmap's Chief Executive Bill Hutchison.
In fact, 70% of the queries submitted used 2 keywords or less. "Obviously using only a small
number of words can lead to ambiguities." For example, if you use the word "Lotus" in a query
you could be referring to a yoga position, flower, car, or software company.
When Wordmap is confronted by an ambiguous word it presents the user with a list of all the
possible meanings of that word. Once the exact meaning has been determined, Wordmap can then
generate a 'super' query by using synonyms and related words joined together in a large
Boolean expression. This new query can then be sent off to as few or as many search engines
as you like. This means that not only is the search query more accurate than if you had only
used simple keywords but it also brings together information from a number of disparate sources.
Specialized thesauri
You can start off a taxonomy using a standard classification scheme, such as the North American
Industry Classification Scheme (NAICS) or the Library of Congress subject headings. Specialist
organisations such as pharmaceutical or law firms can obtain technical taxonomies specific to
their work areas. Other organisations find that they benefit most by setting up a taxonomy
built around their existing processes. (See Case Study - "Taxonomies for Business".)
A taxonomy can be augmented by adding a thesaurus. This could be a plain English thesaurus or
it could be a more specialised technical one. The thesaurus can be used to supply synonyms and
to help determine the next step in navigating the taxonomy, by suggesting a more precise meaning
of the node that you have already reached. "They provide you with a possibility of looking at
a word and saying 'Can I find a better way of describing that?'" says TFPL's Associate Consultant
Barry Mahon.
However although a large taxonomy can help refine a query, it has its downside too. "The biggest
problem that most people seem to have with taxonomies is 'Where am I?'" says Mahon. It is very
important for a good website or portal to show you where you are in relation to where you started.
"Most people get lost quite quickly. They end up in backwaters and wonder how they got there."
Similarly, as the number of options is kept to a minimum to simplify the interface, people
often can't decide where to go next. "A good taxonomy helps you to understand where you are, and
where you should go now, as opposed to where you
can go now," says Mahon. But it's not just a question of
disorientation. "It's also about giving people a sense of their surroundings and a sense of the
context of the subject they're interested in," says Wordmap's Hutchison.
One of the key things in a successful taxonomy is matching its granularity to the application.
The granularity is determined by the techniques used in splitting the target domain into
separate categories.
The right granularity
Two contrasting techniques are neural networks and the use of standard taxonomies, thesauri,
and key concept extraction. "Neural networks try to understand similarities and discrepancies
between the training sets you provide," says Semio's Vogel. "Using training sets you can handle
150 to 200 categories. But for any specific application such as pharmaceutical or legal,
we are talking about 20,000 categories." So, for example, if you are in publishing, with a broad
news access and want to differentiate between politics and the weather, neural networks
will be successful. If you have more specific news, for example chemical news, or the
British Medical Journal, you're going to need a higher
granularity.
TFPL made a study of several world class organisations to see what they were doing about
information overload. Som, the UK Ministry of Defence and Defence Evaluation Research Agency,
for example, did not have a taxonomy as such but were considering compiling one to front end the
knowledge stores that they were building. At the other extreme, Glaxo Wellcome (now GSK) took
advantage of several mergers to reconcile the different taxonomies within the component
organisations to produce a "superthesaurus" (which also carries information) of 53,000 concepts,
200,000 lexical equivalents and 440,000 assertions (concepts joined by relationships).
TFPL's study showed that while a wide range of software was being used to process information, the
building of taxonomies was made "more effective bu human information management experts, working
directly with information users".
To find out how you can obtain a copy of TFPL's report of its study, visit www.tfpl.com
Stanford University Library's HighWire Press has been using Semio Tagger to organise articles
in the 280+ electonic journals it produces online for the journals' publishers. It found that
its search engine, from Verity, helps researchers who know precisely what they were looking for,
but it needed something to help those researchers who didn't know exactly what they were looking
for.
Stanford also needed an automated way of categorising these articles. With almost 12 million
articles online and several hundred more coming in every day, it didn't want to do this manually.
So it installed Semio to help with the classification and taxonomy creation. It also added
some categories that were unique to its organisation by describing the rules and using Semio
to recategorise new documents.
