This article is based on Tika in Action, to be published on Summer, 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code ‘java40beat’ and get 40% discount on eBooks and pBooks ]

also read:

• Java Tutorials
• Java EE Tutorials
• Design Patterns Tutorials
• Java File IO Tutorials

Content Types and Storage

Introduction

The types of content within files vary vastly. We’ve picked two example file format types to examine in this article: the Hierarchical Data Format (HDF), a common file format used to capture scientific information, and Really Simple Syndication (RSS), the most commonly used format to spread news and rapidly changing information.

Content types

For example, consider a scenario in which a science instrument flown on a NASA satellite takes data, which is then downlinked via an overpass of one of any number of existing ground stations, as shown in figure 1. The downlinked data is then transferred via dedicated networks to a science data processing center for data transformation and, ultimately, for dissemination to the public.

In this scenario, the raw data arriving at the science data processing center represents engineering and housekeeping information including raw voltages from the instrument and rudimentary location information (for example, an orbit number). This information is represented as a series of files, each file corresponding to one channel of data from the instrument (three channels in total) and one set of three files per each orbit of the satellite around the Earth.

Data within each channel file is stored initially in a binary data format; for the purposes of this example we’ll assume the widely used Hierarchical Data Format (HDF), version 5 (HDF-5) format. HDF is a binary data format, providing an external user facing API for writing to and reading from HDF-5 files. The HDF-5 API allows users to write data using a small canonical set of data constructs, specifically those shown in table 1 below.

All of the data and metadata from our postulated scenario is represented in a set of three HDF-5 files (corresponding to each channel of the instrument), for each of the orbits that the satellite makes around the Earth. That means that, if the instrument is measuring a set of scientific variables, such as air temperature, wind speed, CO2, or any number of additional interesting other variables, then that information is represented in the HDF-5 files as sets of named scalars, vectors, and matrices in the HDF-5 file.

Of course, that’s just one example of how content is stored inside of a file. Consider another scenario, such as a Really Simple Syndication (RSS) feed file that lists the latest news stories provided by CNN.com, an example of which is provided below in figure 2.

RSS files are based around a simplistic but powerful data model. Each RSS file is an XML file adhering to a prescribed XML schema defining the RSS vocabulary. That vocabulary consists of two main data structures. First, each RSS file typically contains a Channel, which aggregates a set of associated RSS items, each of which typically points to some news story of interest. Every RSS Channel has a set of metadata associated with it, such as a URL and description (for example, http://www.cnn.com/sports/ for the URL and “Latest news stories about sports within the last hour” as the description), as does each RSS item tag.

In the CNN example, CNN publishes set of RSS files, each containing an RSS Channel, one for each CNN news category (such as Top Stories, World, the United States, or any of the other left-hand column categories in figure 2). Each RSS Channel has a corresponding set of latest news stories and links that users can subscribe to via any number of different free RSS readers, including most modern web browsers.

Understanding the types of content is the first step towards automatically extracting information from it. We’ll go into the details of that in the next section, describing how Tika codifies the process of extracting content.

Content extraction

The mechanism by which a file is stored on media may transmit useful information worthy of Tika’s extraction. These mechanisms include the logical representation of files via storage, either through links (like file symbolic links) as well as the notion that files can simply be sets of independent physical files linked together somehow.

Files can be physically stored on a single disk or via the network. Sometimes files may be physically distributed, such as in the case of networked filesystems like Google File System (GFS) or Hadoop Distributed File System (HDFS) but centrally represented via collection of network data blocks or some other higher order structure. We’ll discuss how Tika’s use of the InputStream abstraction hides some of this complexity and uniqueness.

Individual files may be stored on disk as part of a larger whole of logically or physically linked files via some mechanism such as a common collection label, or a unique directory to collect the files together. Tika doesn’t care because it has the ability to exploit information from either case. Madness you say? Read on!

Logical representation

Let’s postulate a simple example of software deployment to illustrate how logical representation of files and directories may convey otherwise-hidden meaning that we’ll want to bring out in the open using Tika. Take for example the software deployment scenario in figure 3.

In our postulated scenario, software is extracted from a configuration management system, let’s say Apache Subversion, and then run through a deployment process that installs the latest and greatest version of the software into the /deploy directory, giving the installed software a unique version number. A symbolic link, titled current is also updated to point to the most recent installed version of the software as a result of this process.

Physical representation

If we expand our focus beyond the logical links between files and consider how those files are actually represented on disk, we arrive at a number of interesting information sources ripe for extraction. For example, considering nowadays that more and more filesystems are moving beyond simple local disks to federations and farms of storage devices, we are faced with an interesting challenge. How do we deal with the extraction of information from a file if we only have available to us a small unit of that file? Even worse, what do we do if that small unit available to us is not a power unit like the file header?

The reality is we need a technology and software approach that can abstract away the mechanism by which the file is actually stored. If the storage mechanism and physical file representation were abstracted away, then the extraction of text and agglomeration of metadata derived from a file could easily be fed into Tika’s traditional extraction processes that we’ve covered so far.

This is precisely the reason that Tika leverages the InputStream as the core data passing interface to its Parser implementations via the parse(…) method. InputStreams obfuscate the underlying storage and protocol used to physically represent file contents (or sets of files). In fact, whether it’s a Google File System URL pointer to a file that’s distributed as “blocks” over the network or a URL pointer to a file that’s locally on disk, Tika still deals with the information as an InputStream via a call to URL.openStream. And, URLs aren’t the only means of getting InputStreams—InputStreams can be generated from Files, byte[] arrays, and all sorts of objects making it the right choice for Tika’s abstraction for the file physical storage interface.

Summary

We started out by showing you how file content organization can affect performance and memory properties and influence how Tika parses out information and metadata. In the case of RSS, its content organization (based on XML) allows for easy streaming and random-access, whereas in the case of HDF5, the entire file had to be read into memory, precluding streaming but supporting random access.

The last important aspect of files is the physical location of a file (or set of associated files) on disk. In many cases, individual files are part of some larger conglomerate, for example, in the case of directories and split files generated by archive/compression utilities.

This article is based on Tika in Action, to be published on Summer, 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code ‘java40beat’ and get 40% discount on eBooks and pBooks ]

also read:

• Java Tutorials
• Java EE Tutorials
• Design Patterns Tutorials
• Java File IO Tutorials

The Standards of Metadata

Introduction

While PDF file properties, and HTML page properties are useful for making decisions such as, “Do I care to read this research paper?” or “Is this web page the one I was looking for?”, the property names themselves don’t tell you everything that you need to know in order to make use of them. For example, is “PDFVersion” an integer, or an alphanumeric? This would be useful to know because it would allow you to compare different “PDFVersion” attributes. What about “Author”? Is it multivalued, meaning that a paper can have multiple authors, or is it only single-valued?

To answer these questions, we usually turn to metadata standards or metadata models. Standards describe all sorts of information about metadata such as cardinality (of fields), relationships between fields, valid values, and ranges, and field definitions, to name a few. Some representative properties of metadata standards are given in table 1.

The International Standards Organization (ISO) has published a reference standard for the description of metadata elements as part of metadata models, numbered ISO-11179. ISO-11179 prescribes a generally accepted mechanism for defining metadata models. There are tons of metadata models out there, and they can be loosely classified as either general models or content-specific models, as depicted in figure 1.

Dublin Core is a general metadata model consisting of fewer than 20 attributes (Creator, Publisher, Format) that are said to describe any electronic resource. On the other side of the coin are Content-specific models, which are unique to a particular file type and only contain metadata elements and descriptions that are relevant to the content type. Examples of these types of models are the Federal Geographic Data Committee (FGDC), a model for describing spatial data files, and Adobe XMP, a metadata standard for media files (like images and videos).

Tika supports both general and content-specific metadata standards. You can get a list of standard metadata models supported by your version of Tika via the option provided by the Tika command –list-met-models line interface.
[code]java -jar tika-app-0.8-SNAPSHOT.jar –list-met-models[/code]
Or, you can print the same list programmatically by calling the TikaCLI from a Java program.

The version of Tika at the time of writing this book (0.8-SNAPSHOT) supports the following metadata models, as output from the DescribeMetadata tool wherein the metadata model name is shown without a left indent and the model’s associated metadata attributes are shown indented one space from the left and capitalized.
[code]ClimateForecast
ACKNOWLEDGEMENT
COMMAND_LINE
COMMENT
CONTACT
CONVENTIONS
EXPERIMENT_ID
HISTORY
INSTITUTION
MODEL_NAME_ENGLISH
PROGRAM_ID
PROJECT_ID
REALIZATION
REFERENCES
SOURCE
TABLE_ID
CreativeCommons
LICENSE_LOCATION
LICENSE_URL
WORK_TYPE
DublinCore
CONTRIBUTOR
COVERAGE
CREATOR
DATE
DESCRIPTION
FORMAT
IDENTIFIER
LANGUAGE
MODIFIED
PUBLISHER
RELATION
RIGHTS
SOURCE
SUBJECT
TITLE
TYPE
Geographic
ALTITUDE
LATITUDE
LONGITUDE
HttpHeaders
CONTENT_DISPOSITION
CONTENT_ENCODING
CONTENT_LANGUAGE
CONTENT_LENGTH
CONTENT_LOCATION
CONTENT_MD5
CONTENT_TYPE
LAST_MODIFIED
LOCATION
MSOffice
APPLICATION_NAME
APPLICATION_VERSION
AUTHOR
CATEGORY
CHARACTER_COUNT
CHARACTER_COUNT_WITH_SPACES
COMMENTS
COMPANY
CONTENT_STATUS
CREATION_DATE
EDIT_TIME
KEYWORDS
LAST_AUTHOR
LAST_PRINTED
LAST_SAVED
LINE_COUNT
MANAGER
NOTES
PAGE_COUNT
PARAGRAPH_COUNT
PRESENTATION_FORMAT
REVISION_NUMBER
SECURITY
SLIDE_COUNT
TEMPLATE
TOTAL_TIME
VERSION
WORD_COUNT
Message
MESSAGE_BCC
MESSAGE_CC
MESSAGE_FROM
MESSAGE_RECIPIENT_ADDRESS
MESSAGE_TO
TIFF
BITS_PER_SAMPLE
EQUIPMENT_MAKE
EQUIPMENT_MODEL
EXPOSURE_TIME
FLASH_FIRED
FOCAL_LENGTH
F_NUMBER
IMAGE_LENGTH
IMAGE_WIDTH
ISO_SPEED_RATINGS
ORIENTATION
ORIGINAL_DATE
RESOLUTION_HORIZONTAL
RESOLUTION_UNIT
RESOLUTION_VERTICAL
SAMPLES_PER_PIXEL
SOFTWARE[/code]
Now that you know what metadata models that Tika supports and that there is a difference between the models (in other words, not all are created equally!), let’s more precisely explore the variations between generic and content-specific metadata models. We’ll use Tika to help us out.

General standards

Most electronic files available via the Internet have a common set of metadata properties, the conglomerate of which are part of what we call general metadata models or general standards for metadata. General models describe electronic resources at a high level as in, “Who authored the content?”, “What format(s) is the content represented in?”, and the like.

To illustrate, let’s take a look at some of the properties of the Dublin Core metadata model attributes, as output by Tika, which supports Dublin Core. Recall the command we showed you above.
[code]java -jar tika-app-0.8-SNAPSHOT.jar –list-met-models[/code]
By exploring the output of the above command a bit, and by using a simple grep command, we can augment the –list-met-models output to isolate only the Dublin Core part.
[code lang=”java”]java -jar tika-app-0.8-SNAPSHOT.jar –list-met-models | grep -A16 DublinCore[/code]
Which produces the output:
[code]DublinCore
CONTRIBUTOR
COVERAGE
CREATOR
DATE
DESCRIPTION
FORMAT
IDENTIFIER
LANGUAGE
MODIFIED
PUBLISHER
RELATION
RIGHTS
SOURCE
SUBJECT
TITLE
TYPE[/code]
Looking at some of these attributes, it’s pretty clear that most or all of them are highly representative of all electronic documents. Think back to table 1. What would the valid values be for something like the FORMAT attribute? Most of the time the metadata field is filled with a valid MIME media type. What would the cardinality be for something like the DATE attribute? Often, electronic documents have a single creation date, but perhaps many last modified dates, so the cardinality is one or more values.

Let’s now focus in on content-specific metadata models.

Content-specific metadata standards

Generic metadata standards and models are great because they address two fundamentally important facets of capturing and using metadata:

• Filling in at least some value per field—Content-specific metadata standards provide at least some value for each field (for example, FORMAT, TITLE in the case of Dublin Core, and their corresponding values captured for, for example, a PDF file, like application/pdf and mypdffile.pdf, respectively) because they’re generic.
• Easily comparable—Mainly due to having some default value, the actual attributes themselves are so general that they are more likely to mean the same thing. (It’s pretty clear what a TITLE is referring to for a document.)

On the other hand, content-specific metadata standards and models are less likely to fulfill either of the above properties. First, they aren’t guaranteed to fill in any values of any of their particular fields. Take MS Office files and their field, COMPANY, derived from the same grep trickery we showed you above:
[code]java -jar target/tika-app-0.8-SNAPSHOT.jar –list-met-models | grep -A28 MSOffice
MSOffice
APPLICATION_NAME
APPLICATION_VERSION
AUTHOR
CATEGORY
CHARACTER_COUNT
CHARACTER_COUNT_WITH_SPACES
COMMENTS
COMPANY
CONTENT_STATUS
CREATION_DATE
EDIT_TIME
KEYWORDS
LAST_AUTHOR
LAST_PRINTED
LAST_SAVED
LINE_COUNT
MANAGER
NOTES
PAGE_COUNT
PARAGRAPH_COUNT
PRESENTATION_FORMAT
REVISION_NUMBER
SECURITY
SLIDE_COUNT
TEMPLATE
TOTAL_TIME
VERSION
WORD_COUNT[/code]
COMPANY is only filled out in an MS Word file’s metadata attributes when the Company name has been entered by the user or owner of the MS Office suite installed on the computer that created the MS Word file. So, in short, if you didn’t fill out the Company field when registering your MS Office, and you begin sharing MS Word files with your other software colleagues, and they want to use Tika to find out what company you work for, they are out of luck. (For privacy-minded people, though, this is a good thing!)

As for being easily comparable, this is another area where content-specific metadata models do not particularly provide a silver bullet. The field in the metadata LAST_MODIFIED HttpHeaders model does not correspond directly to the MODIFIED field in the DublinCore model nor does it correspond to LAST_SAVED field from the MSOffice metadata model. So, content-specific metadata model attributes are not easily comparable across metadata models.

Most document formats have an associated content or file-specific metadata model associated with them (even in the presence of a general model, like Dublin Core). There is the eXtensible Metadata Platform (XMP), pioneered by Adobe for media file formats (images, videos, and so on); there’s a whole slew of MS Office metadata formats, there’s metadata models for JPEG files, metadata for climate related science files in the Climate Forecast metadata model, and there’s corresponding other metadata formats (like FITS) for science files in the astrophysics community. The good news is this: Tika already supports a slew of existing content-specific metadata models and even if it doesn’t, it’s extensible and allows you to add in your own metadata models and attributes/specifications that you can leverage in your own content-specific applications.

We’ll tell you a little bit about metadata quality and how it influences all sorts of things like comparing, understanding, and validating metadata.

Metadata quality

The biggest thing we’ve glossed over while informing you about the wonders of metadata until now is, “So, how does that metadata get populated?” This is a great question. There are plenty of ways. The application program that generates a particular file (for example, MS Office generates Word Documents, PowerPoint files, and so one, or Adobe Photoshop generates PDF files, and so on) can be responsible for annotating a file with metadata.

An alternative is that a user may explicitly fill out metadata about the file on their own when authoring it. Many Software Project Management tools (like MS Project or Fastrack) prompt a user to fill out basic metadata fields (Title, Duration, Start Project Date, End Project Date, and so on) when authoring the file.

Sometimes, downstream software programs author metadata about files. A classic example of this is when a web server returns metadata about the file content it is delivering back to a user request. The web server was not the originator of the file; however, it has the ability to tell a requesting user things like file size, content-type (or MIME type), and other useful properties. This is depicted for the example in figure 2.

During that process, Word annotates the file with basic MsOffice metadata, including AUTHOR, and PAGE_COUNT. After some file is created, later the content creator may publish her file on the Apache HTTPD web server, where it will be available for downstream users to acquire. When a downstream user requests the file from Apache HTTPD, the web server will annotate the file with HttpHeaders metadata, including CONTENT_TYPE and other metadata.

With all of these actors in the system, it’s no wonder that metadata quality, or the examination and assesment of captured metadata for file types, is a big concern. In any one of the steps in figure 2, the metadata for the file could be changed or simply not populated, affecting some downstream user of the file, or some software that must make sense of it later. What’s more, even if the metadata is populated, it’s often difficult to compare metadata captured in different files, even if the metadata captured in fact does represent the same terminology. This is often due to each metadata model’s using its own terms, potentially its own units for those terms, and, ultimately, its own definitions for those terms as well.

Metadata quality is of prime importance, especially in the case of correlating metadata for files of different types and, most often, different metadata models. For a writer of software that must deal with thousands of different file types and metadata models every day, it’s no easy challenge to tackle metadata correlation.

Here comes Tika to save the day again!

Unifying heterogeneous standards

Lucky for us, Tika’s metadata layer is designed with exactly the aforementioned metadata quality challenges in mind. Tika provides a Property class that implements the Adobe XMP standard for capturing metadata attributes. XMP defines a property (called PropertyType in Tika) as some form of metadata captured about an annotated document. XMP also defines property values that are captured for each property of metadata. In Tika we call XMP property values ValueTypes. Let’s take a quick look at a snippet of the Tika Property class.

Listing 1 Tika’s Property class and its support for XMP-like metadata.
[code lang=”java”]public final class Property {
public static enum PropertyType {
SIMPLE, STRUCTURE, BAG, SEQ, ALT
}
public static enum ValueType {
BOOLEAN, OPEN_CHOICE, CLOSED_CHOICE, DATE, INTEGER, LOCALE,
MIME_TYPE, PROPER_NAME, RATIONAL, REAL, TEXT, URI, URL, XPATH
}
// …
}[/code]
The PropertyType and ValueType Java enums allow Tika to define a metadata attribute’s cardinality (for example, is it a SIMPLE value or a sequence of them called SEQ for shorthand), controlled vocabulary (for example, a CLOSED_CHOICE or simple OPEN_CHOICE), and units (for example, a REAL or an INTEGER). Using Tika and its Property class, you are able to decide whether or not LAST_MODIFIED in the HttpHeaders metadata model is roughly equivalent in terms of units, controlled vocabulary and cardinality to that of LAST_SAVED in the MsOffice metadata model.

These capabilities are useful in comparing, validating them, and understanding metadata properties (recall from table 1 that these are important things to capture for each metadata element) and in dealing with heterogeneous metadata models and formats. Tika’s goal is to allow you to curate high-quality metadata in your software application.

Summary

We’ve helped you familiarize yourself with metadata models, Tika’s support for the different properties of metadata models, and most of the important challenges behind dealing with metadata models.

This article is based on Tika in Action, to be published on Summer, 2011. It is being reproduced here by permission from Manning Publications. Manning publishes MEAP (Manning Early Access Program,) eBooks and pBooks. MEAPs are sold exclusively through Manning.com. All pBook purchases include free PDF, mobi and epub. When mobile formats become available all customers will be contacted and upgraded. Visit Manning.com for more information. [ Use promotional code ‘java40beat’ and get 40% discount on eBooks and pBooks ]

also read:

• Java Tutorials
• Java EE Tutorials
• Design Patterns Tutorials
• Java File IO Tutorials

Sounds Greek to Me—Theory of Language Detection

Introduction

The ability to consistently name and classify things is essential for fully understanding them. There are thousands of languages in the world, many with multiple dialects or regional variants. Some of the languages are extinct and some are artificial. Some don’t even have names in English! Others, like Chinese, have names whose specific meaning is highly context-sensitive. A standard taxonomy that can name and classify all languages is needed to make it allow information systems to reliably store and process information about languages.

There are a number of different increasingly detailed systems for categorizing and naming languages, their dialects and other variants. For example, according to the RFC 5646: Tags for Identifying Languages standard, you could use de-CH-1996 to identify the form of German used in Switzerland after the spelling reform of 1996. Luckily there aren’t many practical applications where such detail is necessary or even desirable, so we’ll be focusing on just the de part of this identifier.

The RFC 5646 standard mentioned above leverages ISO 639 just like most of the other formal language taxonomies. ISO 639 is a set of standards defined by the International Organization for Standardization (ISO). The ISO 639 standards define a set of two- and three-letter language codes like the de code for German we encountered above. The two-letter codes that are most commonly used are defined in the ISO 639-1 standard. There are currently 184 registered two-letter language codes, and they represent most of the major languages in the world. The three-letter codes defined in the other ISO 639 standards are used mostly for more detailed representation of language variants and also for minor or even extinct languages.

The full list of ISO 639-1 codes is available from http://www.loc.gov/standards/iso639-2/ along with the larger lists of ISO 639-2 codes. Tika can detect 18 of the 184 currently registered ISO 639-1 languages. The codes of these supported languages are listed below.

• da—Danish
• de—German
• et—Estonian
• el—Greek
• en—English
• es—Spanish
• fi—Finnish
• fr—French
• hu—Hungarian
• is—Icelandic
• it—Italian
• nl—Dutch
• no—Norwegian
• pl—Polish
• pt—Portuguese
• ru—Russian
• sv—Swedish
• th—Thai

After detecting the language of a document, Tika will use the above ISO 639-1 codes to identify the detected language. But how do we get to that point? Let’s find out!

Language profiles

Detecting the language of a document typically involves constructing a language profile of the document text and comparing that profile with those of known languages. The structure and contents of the language profile depend heavily on the detection algorithm but usually consists of a statistic compilation of some relevant features of the text. You’ll learn here about the profiling process and different profiling algorithms.

Usually the profile of a known language is constructed in the same way as that of the text whose language is being detected. The only difference is that the language of this text set, called a corpus, is already known in advance. For example, one could use the combined works of Shakespeare to create a profile for detecting his plays, those of his contemporaries or modern works that mimic the Shakespearean style. Should you come across and old-looking book like the one shown in figure 1, you could use the Shakespearean profile to test whether the contents of the book match its looks. Of course, such a profile would be less efficient at accurately matching the English language as it is used today.

A key question, then, for the developers of language detection or other natural language processing tools often is to find a good corpus that accurately and fairly represents the different ways a language is used. Usually it’s also the better the bigger the corpus is. Common sources of such sets of text are books, magazines and newspapers, official documents, and so on. Some are also based on the transcripts of spoken language from TV and radio programs. And, of course, the Internet is quickly becoming an important source, even though much of the text there is poorly categorized or labeled.

Once you have profiled the corpus of a language, you can use that profile to detect other texts that exhibit similar features. The better your profiling algorithm is the better those features match the features of the language in general instead of those of your corpus. The result of the profile comparison typically is a distance measure that indicates how close or how far the two profiles are from each other.

The language whose profile is closest to that of the candidate text is also most likely the language in which that text is written. The distance can also be a percentage estimate of how likely it is for the text to be written in a given language.

You’re probably already wondering about what these profiling algorithms look like. It’s time to find out!

Profiling algorithms

The most obvious way to detect the language used in a piece of text is to look up the used words in dictionaries of different languages. If the majority of words in a given piece of text can be found in the dictionary of some language, it’s quite likely that the text indeed is written in that language. Even a relatively small dictionary of the most commonly used words of a language is often good enough for such language detection. You could even get reasonably accurate results with just the word “the” for detecting English, the words “le” and “la” for French, and “der”, “die” and “das” for German!

Such a list of common words is probably the simplest reasonably effective language profile. It could be further improved by associating each word with its relative frequency and calculating the distance of two profiles as the sum of differences between the frequencies of matching words. Another advantage2 of this improvement is that it allows the same profiling algorithm to be used to easily generate a language profile from a selected corpus instead of having to use a dictionary or other explicit list of common words.

Alas, the main problem with such an algorithm is that it’s not very efficient at matching short texts like single sentences or even just a few words. It also depends on the way of detecting word boundaries, which may be troublesome for languages like German, with lots of compound words, or Chinese and Japanese, where no whitespace or other extra punctuation is typically used to separate words. Finally, it has big problems with agglutinative languages like Finnish or Korean, where most words are formed by composing smaller units of meaning. For example the Finnish words “kotona” and “kotoa” mean “at home” and “from home” respectively, which makes counting common words like “at”, “from” or even “home” somewhat futile.

Given these difficulties, how about looking at individual characters or character groups instead?

The N-gram algorithm

The profiling algorithm based on word frequencies can just as easily be applied on individual characters. In fact, this even makes the algorithm simpler since, instead of a potentially infinite number of distinct words, you only need to track a finite number of characters. And, it turns out that character frequencies really do depend on the language, as shown in figure 2.

And, obviously this algorithm works even better with many Asian languages with characters that are used in only one or just a handful of languages. This algorithm however has the same problem as the word-based one in that it needs quite a bit of text for an accurate match. Interestingly enough, the problem here is the opposite of that with words. Where a short sentence may not contain any of the most common words of a language, it’s practically guaranteed to contain plenty of the common characters. Instead, the problem is that there simply isn’t enough material to differentiate between the languages with similar character frequencies.

This detail hints at an interesting approach that turns out to be quite useful in language detection. Instead of looking at individual words or characters, we could look at character sequences of a given length. Such sequences are called 2-, 3-, 4-grams or, more generally, N-grams based on the sequence length. For example, the 3-grams of a word like “hello” would be “hel”, “ell” and “llo”, plus “_he” and “lo_” when counting word boundaries as separate characters.

It turns out that N-grams are highly effective at isolating the essential features of at least most European languages. They nicely avoid problems with compound words or the oddities of languages like Finnish. And, they still provide statistically significant matches even for relatively short texts. Tika opts to use 3-grams as that seems to offer the best tradeoff of features in most practical cases.

Advanced profiling algorithms

There are other more advanced language profiling algorithms out there, but few match the simplicity and efficiency of the N-gram method described above. Typically such algorithms target specific features like maximum coverage of different kinds of languages, the ability to accurately detect the language of very short texts or the ability to detect multiple languages within a multi-lingual document.

Tika tracks developments in this area and may incorporate some new algorithms in its language detection features in future releases, but for now N-grams are the main profiling algorithm used by Tika.

Summary

Even though natural language processing is a fiendishly complex subject that has and probably will be a topic of scientific research for decades, there are certain areas that already useful in practical applications. Language detection is one of the simpler tasks of natural language processing and can, for the most part, be implemented with relatively simple statistical tools. Tika’s N-gram based language detection feature is one such implementation.