In the previous post on this topic, we looked at the criteria traditionally applied by dictionary-makers when considering new words for inclusion. The question is as old as lexicography itself. When he wrote his Plan of an English Dictionary in 1747, Dr Johnson noted that it is ‘not easy to determine by what rule of distinction the words of this dictionary were to be chosen’. And having aired his ideas on the subject, he acknowledged that it isn’t always possible to make clear rules and then adhere to them strictly. 

The Oxford dictionary website also has a go at explaining its inclusion principles – this time by means of an elaborate flowchart which takes you through the various decision points. Having cleared numerous hurdles, the successful word is at last included in the dictionary ‘in due course’. I’m not sure I agree with every stage of this. For example, if the question ‘Is its use limited strictly to one group of users?’ is answered with a ‘Yes’, the word is consigned to a sort of purgatory where its behaviour is monitored for possible future inclusion. But dictionaries routinely include vocabulary typical of specific user-groups – the important thing is to apply an appropriate label to indicate that it is not part of the general language. On the whole, though, the Oxford chart gives a good outline of the key criteria: does the evidence come from a range of sources (what we referred to previously as ‘dispersion’), and does it have ‘a decent history of use’(the longevity argument)?

The problem is that the approach applied by both Oxford and Merriam-Webster is rooted in the past. It reflects the realities of print-based dictionary publishing – and those days are gone.

What has changed? First, what we’d call the ‘publishing cycle’. When dictionaries existed mostly as printed books, publishers would produce a new edition every four or five years. They collected new vocabulary as it appeared, but they could take the long view on whether something was worth including. We do things differently now. Consider for example the linguistic fallout of the global financial crisis that began in 2008 – just a year after Macmillan published the second edition of its dictionary. With the dictionary now mainly consulted online, we were able to add important new usages, such as the word credit crunch or the new sense of toxic (when applied to debts) – without having to wait several years. The second big change, which has been gathering pace since the turn of the century, is that the amount of evidence available to us has grown exponentially, thanks to the Web and social media. Thirdly, we’re no longer limited by space constraints. Even the largest printed dictionaries don’t have the infinite amounts of space that online media provide, so they have to be selective. That’s no bad thing: the removal of these limits shouldn’t be a licence to include just anything. But it does allow us to re-think – and broaden – our inclusion policies.

Above all, older notions about ‘what gets into the dictionary’ reflect the idea of the lexicographer as a gatekeeper, the belief that it is up to us to decide (on behalf of everyone else) which facts about language deserve the special status of  being admitted to a dictionary. This notion of the dictionary having special ‘authority’ (which it confers on the words it includes) is well-established, and still has wide appeal. But it may be incompatible with the priorities and expectations of users of the Web – especially digital natives. If a word is in common use, people expect to find it in their online dictionary, and they won’t be impressed by the argument that it first requires ‘a decent history of use’. For many users, in other words, speed and convenience, getting a useful answer now, may be more important than authority.

As in so many other areas, one of the impacts of the Web has been a challenge to the old top-down model of one ‘expert’ provider and many passive recipients. It isn’t simply a case of users expecting dictionaries to respond more rapidly to language change – many of them also want to be involved in the compilation process. (Wikipedia is the obvious analogy.) In the final part of this series, we’ll discuss the implications of ‘crowd-sourced’ dictionary content (already a central feature of Wordnik, for example, and of our own Open Dictionary), and we’ll also look at emerging language technologies which might just change everything.

In Kate Atkinson’s recent novel, Started Early, Took My Dog (2010), there’s an exchange between two of the characters. When one of them mentions a large sum of money, we read that Kelly, the other character, ‘suddenly meerkatted to attention’. Does this mean we have a new verb on our hands, to meerkat? Should it be added to the dictionary? Probably not. Atkinson is doing what most language users do occasionally (and some do quite often): taking a word or phrase the reader already knows, and doing something inventive with it to create a new meaning. In this case, it’s simply a question of making a verb out of a noun – a process we’ve discussed frequently in the blog – so the reader has no difficulty understanding the sentence. But this is what we’d call an ‘exploitation’ (a one-off, imaginative coinage) rather than a ‘norm’ (something which has become ‘settled’ in the language through repeated use), and this is one of several factors we have to take account of when deciding what to put in the dictionary.

There’s a video describing the process by which new words are admitted to Merriam-Webster’s dictionaries. It’s quite a complex procedure, and the editor who explains it says that this question – ‘How does a word get into your dictionary?’ – is the one he gets asked most often. It’s an important question, too, so we’re devoting three posts to it. We’ll look first at the ‘inclusion criteria’ which dictionary editors have traditionally used; then we’ll consider how far these are still relevant in the world of online dictionaries; and in a final post, we’ll look at emerging technologies which could eventually make this whole question irrelevant.

Orin’s recent post on Tebowing highlights the general fascination with new words, and explains why some words never make it into dictionaries. And if the public is engaged with the issue of what goes in the dictionary, journalists and reviewers are even more interested. We’ve discussed before how most dictionary reviews focus almost exclusively on new words and meanings, and this reflects an assumption that ‘getting into the dictionary’ confers a special status on the successful word. As Kerry Maxwell points out:

For many, the perception is that any word which has gained enough currency to be officially recorded is a ‘proper’ word, here to stay for the use of future generations.

Kerry’s article on how words get into a Macmillan dictionary provides a useful summary of the key issues. There are three main criteria. First, is it a ‘real’ word anyway, or is it simply, like meerkatted, an individual writer’s playful use of language? (Of course, ‘exploitations’ like this can turn into norms – and so become dictionary entries – if other people pick up the usage and recycle it often enough.) Second, what does the evidence tell us about our word’s use? Any linguistic feature (be it a word, phrase, collocation, or meaning) which occurs frequently enough over a long enough period will start to look as if it is ‘part of the language’ – and therefore to deserve its place in a dictionary.

The problem of course is deciding what counts as ‘enough’. And frequency alone is not a sufficient condition because a word may be used with great frequency in a single text – but hardly at all outside it. An extreme example is a word like droog, which appears repeatedly in the novel A Clockwork Orange (it’s part of the secret language which the main characters use, and means ‘friend’), but has never entered the general language. So ‘dispersion’ – the extent to which a word occurs in a range of different sources – is almost as important as frequency. Third, there is the question of whether the word is appropriate for the particular dictionary we’re dealing with. There’s a big difference between a major historical dictionary (the Oxford English Dictionary, for example, contains over a million headwords, many of them long obsolete) and a dictionary aimed at schoolchildren. So the needs, expectations, and language skills of the dictionary’s intended users are all factors to be taken into account.

Kerry’s explanation of Macmillan’s inclusion criteria was written in 2006 – which now seems like a bygone era. At that time, ‘the dictionary’ was still (for most people) a printed book of limited dimensions, Facebook and Twitter barely existed, and language technologies in common use today were still in development. In the next post on this topic, we’ll consider how far traditional ideas about what gets into the dictionary remain relevant in 2012.

The Macmillan Dictionary got a mention in The Guardian yesterday, when Jane Martinson pondered the use of the word simper. A fellow journalist (male) had tweeted about a lawyer (female) ‘simpering’ at a witness (male) in the ongoing Leveson Inquiry. (The inquiry was set up in the wake of revelations that News International journalists had obtained stories by hacking into the phones of celebrities, politicians, and crime victims.) ‘Can anyone remember,’ she wondered,  ‘the last time a man was accused of “simpering”?’

She’s right. Corpus evidence suggests that simper is used three or four times as often about girls and women as about boys or men. Not only that, where the word is used about men, there’s sometimes an implication that they are not ‘real’ men (that’s why they simper): we hear from an American writer about ‘Simpering Frenchman Jacques Chirac’ (apologies to our French readers), and there are several cases of gay men described as simpering too. This happens a lot: the only people who flounce in and out of rooms are women (overwhelmingly), and gay men (occasionally) – but never heterosexual men. (I’m just reporting what the data tells us, so don’t shoot the messenger.)

As always, the co-text is instructive: simper appears with adverbs like flirtatiously, seductively, or sweetly, while other verbs found in the vicinity include fawn, pout, blush, and giggle – all words associated (whether we like it or not) with women. This example from the corpus gives a good flavour of how simper is typically used:

She preferred male company … and had no time for giggling, simpering girls who cared for nothing but gossip and the price of hair ribbon.

As Jane Martinson pointed out, the example given in the Macmillan entry has a female subject (She spoke in a simpering tone), and this takes us back to an issue we discussed last year, during Gender English month: should dictionary editors ignore the evidence and show a man in the example (as a way of combating gender stereotypes), or do we record what we find? No easy answers here, though we have to balance our gender-neutral instincts with a description of usage that’s true to the data.

Much has been written about words that blatantly insult women: slut, harpy, bitch and the like. But simper belongs to a more interesting category – words which belittle women, but which do it just subtly enough that (some) men think they can get away with it. Something similar is happening with feisty, another ‘suspect’ word mentioned by Martinson. Again, the data backs her up: feisty is overwhelmingly used about women, and the nouns it frequently modifies include heroine, redhead, tomboy (=honorary male), lady, gal, and even filly. On the surface, it conveys admiration – but this is qualified by the implication that ‘She did well – considering she’s only a woman’.

There is much more to be said on this subject. A man who is quiet and reserved, for example, tends to be described as taciturn – a word rarely applied to women – or even ‘the strong silent type’: both positive descriptions. A woman of the same type is just quiet, and probably also shy or even mousy. Or even simpering … Well, maybe we’ll come back to this another day. Oh, and thanks to Jane Martinson, too, for adding another word (twarrumph) to our growing collection of Twitter-inspired vocabulary.

In a recent post, we saw that the word jargon – while more or less synonymous with terminology – has a much more negative feel. As always, you can tell a lot about a word by the company it keeps, and a comparison of the adjectives that frequently collocate with these two nouns is revealing. Both are frequently used with neutral words like technical, specialized, scientific, and legal. But (unlike terminology) jargon is often modified by adjectives such as incomprehensible, impenetrable, and unintelligible. It is sometimes implied that technical terms are being employed simply in order to make an impression or baffle the listener, when jargon is described as pompous, pretentious and – in over 50 instances in our corpus – unnecessary. So jargon is clearly something to be avoided (in fact avoid is one of its most common verb collocates), as this damning example from the corpus clearly demonstrates:

This new breed [of manager] can be spotted by their willingness to spout incomprehensible jargon.

This technique of examining collocates may help to defuse a row that has erupted this week in Chile, where the new education minister, Harald Beyer, has recommended a controversial change to the history textbooks used in the nation’s schools. From now on, the period when General Pinochet was in power will no longer be referred to as a ‘dictatorship’ but as a ‘regime’. This has, the news story says, ‘provoked outrage among left-wing opposition parties’ – but are they right to be so incensed?

The data suggests that regime and dictatorship are in fact very similar in terms of their ‘emotional charge’. The Sketch Engine – the software package we use for analysing corpus data – includes a nifty tool that allows you to compare the collocates of two words, and the results for this pair are revealing. Both are modified, with more or less equal frequency, by the adjectives repressive, corrupt, tyrannical, and brutal, while authoritarian and oppressive are even more likely to collocate with regime than with dictatorship. It’s a similar story with the verbs. When these nouns are in the subject position, both occur frequently with neutral words like rule or govern. But, if the linguistic evidence is anything to go by, regimes have an even stronger tendency than dictatorships to persecute, murder, oppress, imprison and torture people. So the proposed change in Chile may not after all do much to improve Pinochet’s image – if anything, ‘regimes’ look worse than ‘dictatorships’.

There’s an obvious flaw in this argument: the data here is about the English equivalents, but Chile is a Spanish-speaking country, and the change at the centre of the argument is not from dictatorship to regime, but from dictadura to régimen militar. Fortunately, the Sketch Engine has a large corpus of Spanish too, so we can run the same comparison. The results are very different, and (not surprisingly) reflect the political history of Spain and Latin America – one of the commonest modifiers for both nouns is ‘Francoist‘ (franquista). But here too, there is some evidence to suggest that the connotations of ‘regime’ are no less negative than those of ‘dictatorship’, with régimen often attracting adjectives like authoritarian, dsciplinarian, fascist, and totalitarian.

As always, comments are welcome, and we’d be especially interested to hear what our Chilean readers think about this.

At the recent eLEX 2011 conference in Slovenia (for earlier posts, see here and here), the discussion focussed on the future of dictionaries – or, more broadly, on the various ways in which reference needs might be catered for in years to come. What often happens in this field is that people working in universities and research groups develop software tools or learning materials for their own (local) users, but their ideas and methods are then taken up by more mainstream providers. Serge Verlinde of Leuven University in Belgium gave one of the keynote speeches. Serge is a great example of someone who has done pioneering work over many years to develop online reference tools, in this case for learners of French. His ‘Base lexicale du français’ (BLF) not only supplies information about word meanings, grammar, and collocation, but also guides the user to make the right word choices when writing in French or translating from French to another language. The BLF also includes a ‘reading assistant’ (providing various kinds of help to enable you to understand a text), and a tool for helping users write a text is in development.

This theme of aids for writing (what are sometimes called ‘text remediation’ tools) appeared in several other talks, too. We heard from Magali Paquot about the Louvain English for Academic Purposes Dictionary (LEAD), a web-based resource designed to assist in the production of academic writing, and from another group about resources being developed in South Africa to aid ‘text production’ in a number of languages.

One of the features of resources like this is that they should be ‘dynamic’: that is, the system should learn from what users do, and what information they look for, and adapt itself as it goes along. Of course, much of this is still in the planning stage: we know what we want to do, but haven’t yet fully worked out how to do it. Part of the solution lies in having computational tools that can, for example, identify errors or unnatural features in a text supplied by a user. One research group from Barcelona showed how it had achieved a success rate of almost 90% in automatically detecting ‘bad’ collocations, and this kind of tool could form one component of a writing assistant that really worked. In the long term, devices like this could replace conventional dictionaries – at least for language production – because they would do one of the jobs dictionaries have traditionally done, but do it much better.

Another theme at the conference was ‘UGC’ (user-generated content), already such a big feature of the online world. News programmes, for example, routinely include information supplied by their viewers and listeners in the form of tweets, emails, or comments posted on their websites. In the world of reference, Wikipedia is the obvious example of a resource created entirely by its users, but the trend is spreading to dictionaries too. Wordnik has huge amounts of UGC, with numerous words added by members of the public, example sentences ‘harvested’ from the Twittersphere, and all sorts of lists created by the site’s users. A new translation tool being developed by the Russian company ABBYY will include a facility for users to contribute their own translations. And of course Macmillan has its Open Dictionary – an ever-expanding record of the most up-to-date uses of English around the world.

The conference provided a perfect snapshot of current activity and thinking in this exciting field. We may not be much closer to knowing how things will pan out over the next ten years (or even the next two). Developments in information technology, and in the skills, needs and expectations of its users, are all racing ahead at breakneck speed, so we can’t make predictions with any confidence. It brings to mind the famous remark made by Zhou Enlai, the Prime Minister of China till his death in 1976. When asked what he thought was the long-term significance of the French Revolution of 1789, he replied ‘It’s too soon to tell’.

More news from eLEX2011, the conference on e-lexicography currently taking place in Slovenia.

The conference got off to a rip-roaring start as Simon Krek (one of the organizers) outlined a radical vision for a future in which a range of intelligent language tools would be freely available to make communication easier. The functions Simon mentioned include real-time subtitling (so a person in China could follow the U.S. elections on American TV, since everything would be translated on the spot), automatic summarization of complex documents (in your own language or another one), and (most ambitiously) instant speech-to-speech translation.

Some of this may sound like science fiction. But the same could be said of the ideas that visionaries like John Sinclair were putting forward 25 years ago – some of which are now quite mature technologies that we already take for granted.

Dictionaries – in their familiar form, at least – aren’t necessarily part of this longer-term vision. After all, dictionaries evolved in order to fulfil a number of communicative and informational needs that people have, but there may be more efficient ways of meeting those needs. It’s already the case that many people (especially digital natives) no longer turn to a dictionary to find out what something means or how it is pronounced or spelled. In this new model, the user simply expresses a need to resolve a particular language problem, and the computer does the rest. What matters is getting the answer (quickly and reliably) – and the ‘container’ that holds the answer is of no importance. Adam Kilgarriff (a regular contributor to this blog) put it like this: the dictionary may simply ‘dissolve’ to form one component – albeit an important one – in a much broader operation which can be characterized as ‘search’.

For ‘search’ to work optimally, we need high-quality language resources. Dictionaries are certainly part of this, so are encyclopedias, and corpora. There is interesting work going on in the ‘semantic tagging’ of corpora: the Dutch computational linguist Piek Vossen reported on a project of this type. Essentially, it’s about training computers to do something that human beings are very good at, namely recognising which meaning is meant when several are possible. The challenge is to get the computer to recognise that when it sees the word mouse in a particular context, it refers to a small rodent, or a computer device, or a shy, quiet person. This is extremely difficult work, but Piek’s paper demonstrated that the task was doable, and that the computer’s success rate was gradually improving.

I can’t do justice to the range of excellent papers I’ve attended so far; I feel I’ll need a couple of weeks in a quiet place just to digest it all. Many talks dealt with issues we’ve looked at regularly on this blog. The importance of pragmatics, for example, was highlighted by Mojca Šorli, who had some good proposals for improving the way dictionaries present this information. Collocation, not surprisingly, featured in several sessions: how to extract them automatically from corpora; how to integrate them into reference resources; how to cater for users who need more specialized collocational information (for example when writing academic texts); and how computer systems can be trained to identify ‘miscollocations’ and propose corrections. And as always, there’s a lot to learn from resources being developed for languages other than English.

Erin McKean rounded off the first day with an entertaining talk about the amazing Wordnik site – one example of a resource that goes way beyond what we’d expect in a conventional dictionary, and shows (among other things) how engaging user-generated content can be. As good a demonstration as any of a well-known quote by science fiction writer William Gibson: “The future is already here – it’s just not very evenly distributed”.

There’s no space (or time) to say more, but there will be a final round-up early next week.

(For a final summary of eLEX2011, see this post.)

Today’s post comes from the beautiful Slovenian city of Bled, where I’m attending a conference called ‘eLEX2011’– or ‘Electronic lexicography in the 21st century’.

Regular readers will be aware of how completely the job of producing dictionaries was transformed in the 1980s by the arrival of large language corpora. Those were pioneering times, and the technology struggled to keep up with our appetite for more linguistic data. Now we are awash with it. It has enabled us to produce much better, much more complete descriptions of language. We’d be wary now of making any definitive statement about a word or phrase without first reviewing its use in real communicative situations. A recent post on the use of complete, for example, would have been impossible without a decent corpus and the software to interrogate it. And yet, even 20 years after corpora first came on the scene, the dictionary itself didn’t look radically different. This was a revolution at the ‘producer’ end: dictionary-making would never be the same, but for dictionary users the changes weren’t so obvious.

Now there’s a second revolution – and this time it’s at the ‘consumer’ end. Dictionaries first went digital in the mid-90s, as publishers made their wares available on CD-ROMs as well as in printed books. But the changes this brought were incremental rather than revolutionary, and that technology reached its sell-by date before it had the chance to cause too much disruption to the way dictionaries looked. The really big changes started in the noughties with ‘Web 2.0’, and the pace has been accelerating as digital natives come of age.

The theme of the conference is e-lexicography, but as someone said: what other kind of lexicography is there? This is the second event of its kind. The first, organized by Sylviane Granger and Magali Paquot (who have contributed to our dictionary resources at Macmillan), took place in Belgium in 2009. It was an eye-opener. But the scary thing is that, in the short space of two years, the landscape has already changed significantly. Connectivity rates have advanced so rapidly that, in most parts of the world, high-speed web access is now the norm, or very soon will be. Against this background, the demise of the printed book as a medium for general reference materials – still a topic of debate at ‘eLEX2009′ – now looks inevitable.

Another striking change is the rise and rise of mobile computing. I still have an old-school ‘tower’ computer at home, but this is a dying format. It’s hard to believe that the iPad was only launched in January 2010 (six months after the previous eLEX conference) but this and similar mobile devices are becoming increasingly dominant. Sales of tablet computers are predicted to hit 300 million units per year by 2015, with smartphone sales even higher. Dictionary publishers are already engaging with these new media – the Macmillan Dictionary, like many others, has apps for iPhones and iPads. But the longer-term implications of this game-changing technology are far from clear. We’re right in the middle of a second major revolution for lexicography, and no-one really knows where it is headed. That’s what we’re hoping to find out at this conference, and the Macmillan blog will keep you up to date with developments.

(For more news from eLEX 2011, see here and here.)

This week’s ‘language in new media’ post is one of the fantastic TED Talks.

What we learned from 5 million books uses Google Labs’ Ngram Viewer tool to tell us why exactly a picture is worth so much more than a thousand words!

If you’re new to the Ngram Viewer, you might also like to read Stan Carery’s post on the subject from earlier this year.

This week’s ‘language in new media’ post explores the ‘melody of microblogging’ in ‘The real sound of Twitter’.

21-year-old student Sam Harman, or “evil doctor tweet” as he is sometimes known, has created a programme which turns the global language of Twitter into music. Twinthesis, or ‘Twitter powered synthesis’, harnesses the daily tirade of tweets and transforms them into musical notes, reflecting the tone of the original tweet.

Listen to Sam explaining his project to Jon Kay on BBC Radio 4.

Macmillan runs a series of webinars, which are a bit like interactive lectures that anyone can join in. Coming up in 2011 are speakers such as Lindsay Clanfield and Simon Greenall, and from the same page you can watch sessions from the archive featuring well-known language-teaching experts like Scott Thornbury, Ken Wilson and Sam McCarter. A couple of weeks ago I did a webinar on the subject of language technology and its impact on dictionaries, which you can find here (you can also download the Powerpoint presentation that  the webinar was based around). A word of warning: the first five minutes or so are a little messy (we had a few teething problems with the technology), but it gets better after that.

While computer technology means we can put our dictionary online, hold webinars, and discuss language issues in this blog, language technology is more about the data and software we use in the background, to help us decide what to say about words. Language technology (LT) is big business. It’s what powers search engines like Google or sites offering automatic translation. Some LT tools are pretty simple: sites like Google Fight or Google’s Ngram Viewer work by counting the number of times a particular word or phrase is used – and counting is something that computers are good at. That’s also how we can identify, very reliably, the ‘core’ vocabulary of English, which is shown as the red words in the Macmillan Dictionary. But for more sophisticated tasks like translating, computers have to be trained to understand human language – or at least to perform as if they understand it.

This is hard for computers. Language is full of ambiguities because most words have more than one meaning. Human beings are good at dealing with this, and in most situations misunderstandings are rare. If I say ‘I’m going to the bank’, the person I’m talking to doesn’t need to ask if I mean ‘the financial institution’ or ‘the side of the river’. Context will tell them which sense of bank I’m referring to. But the only thing a machine knows is that bank has two possible meanings (which have different equivalents in other languages), and it has to decide which one fits best. So the main goal of language technology is to enable computers to do what humans do so effortlessly when they communicate with one another. Progress towards this goal is slow but steady. Just this week, an IBM computer has beaten two champion (human) contestants in the popular US quiz show Jeopardy.

Thanks to research in this field, lexicographers now have powerful software that reveals far more about how words behave and work together than we knew just ten or even five years ago – and that’s the main theme of my webinar. I’m afraid I probably talked for too long, and when the session ended there was no time to answer anyone’s questions. So this is another opportunity: if there’s anything you’d like to ask about this subject, use the Comments here and I’ll get back to you.