R Programming/Text Processing

This page includes all the material you need to deal with strings in R. The section on regular expressions may be useful to understand the rest of the page, even if it is not necessary if you only need to perform some simple tasks.

This page may be useful to :
 * perform statistical text analysis.
 * collect data from an unformatted text file.
 * deal with character variables.

In this page, we learn how to read a text file and how to use R functions for characters. There are two kind of function for characters, simple functions and regular expressions. Many functions are part of the standard R base package. However, their name and their syntax is not intuitive to all users. Hadley Wickham has developed the stringr package which defines functions with similar behaviour but their names are easier to retain and their syntax much more systematic.


 * Keywords : text mining, natural language processing
 * See CRAN Task view on Natural Language Processing
 * See also the following packages tm, tau, languageR, scrapeR.

Reading and writing text files
R can read any text file using  or. It is possible to specify the encoding of the imported text file with. The entire contents of the text file can be read into an R object (e.g., a character vector). is more flexible. The kind of data expected can be specified in the second argument (e.g., character(0) for a string).

We can write the content of an R object into a text file using  or. By default  concatenates vectors when writing to the text file. You can change it by adding options  or. The default encoding depends on your computer.

Before reading a text file, you can look at its properties. (parser package) and  (R.utils package) count the number of lines in the file. (parser package) counts the number of bytes and characters in each line of a file. You can also display a text file using.

Character encoding
R provides functions to deal with various set of encoding schemes. This is useful if you deal with text file which have been created with another operating system and especially if the language is not English and has many accents and specific characters. For instance, the standard encoding scheme in Linux is "UTF-8" whereas the standard encoding scheme in Windows is "Latin1". The  functions returns the encoding of a string. is similar to the unix command iconv and converts the encoding.
 * gives the list of available encoding scheme on your computer.
 * ,  and   have also an encoding option.
 * (tau) tests if the encoding is "utf8".
 * (tau) tests if encoding is the same as the default encoding on your computer.
 * (tau) translates the encoding into the current locale.
 * (descr) is less general than.
 * (base)

Example
The following example was run under Windows. Thus, the default encoding is "latin1".

Regular Expressions
A regular expression is a specific pattern in a set of strings. For instance, one could have the following pattern : 2 digits, 2 letters and 4 digits. R provides powerful functions to deal with regular expressions. Two types of regular expressions are used in R There is a also an option called  which can be considered as a literal regular expression. (stringr) is equivalent to  in the standard regex functions. These functions are by default case sensitive. This can be changed by specifying the option.
 * extended regular expressions, used by  (the default),
 * Perl-like regular expressions used by.

If you are not a specialist in regular expression you may find the  useful. This function suggests some regular expression for a specific ("glob" or "wildcard") pattern :

Functions which use regular expressions in R

 * ,,   (stringr) make some substitutions in a string.
 * ,  (stringr) extract some value
 * ,  (stringr) detect the presence of a pattern.
 * see also  (R.utils)
 * See also  in the gsubfn package.

Extended regular expressions (The default)

 * stands for any character.
 * means A,B or C.
 * means any upper letter between A and Z.
 * means any digit between 0 and 9.

Here is the list of metacharacters. If you need to use one of those characters, precede them with a doubled backslash.

Here are some classes of regular expressions : For numbers : For letters : Note that the set of alphabetic characters includes accents such as  which are very common in some languages like French. Therefore, it is more general than  which does not include letters with accent.
 * Digits:.
 * Alphabetic characters:  and.
 * Upper-case letters.
 * Lower-case letters.

For other characters : For combination of other classes :
 * Punctuation characters:.
 * Space characters: tab, newline, vertical tab, form feed, carriage return, and space.
 * Blank characters: space and tab.
 * Control characters.
 * Alphanumeric characters:  and.
 * Graphical characters:  and.
 * Printable characters:,   and space.
 * Hexadecimal digits:.

You can quantify the number of repetition by adding after the regular expression the following characters :
 * The preceding item is optional and will be matched at most once.
 * The preceding item will be matched zero or more times.
 * The preceding item will be matched one or more times.
 * The preceding item is matched exactly ‘n’ times.
 * The preceding item is matched ‘n’ or more times.
 * The preceding item is matched at least ‘n’ times, but not more than ‘m’ times.


 * to force the regular expression to be at the beginning of the string
 * to force the regular expression to be at the end of the string

If you want to know more, have a look at the 2 following help files :

Perl-like regular expressions
It is also possible to use "perl-like" regular expressions. You just need to use the option.

Examples
If you want to remove space characters in a string, you can use the  Perl macro.

Concatenating strings

 * concatenates strings.
 * (stringr) does a similar job.
 * prints and concatenates strings.

Splitting a string

 * : Split the elements of a character vector ‘x’ into substrings according to the matches to substring ‘split’ within them.
 * See also  (stringr).


 * (tau) split a string into tokens.

Counting the number of characters in a string

 * gives the length of a string. Note that that for non-ASCII encodings, there is more one way to measure such a length.
 * See also  (stringr)

Detecting a pattern in a string ?

 * returns a logical expression (TRUE or FALSE).
 * (stringr) does a similar job.

The 1st one is true and the second one is false since there is only one digit in the first number.

Counting the occurrence of each pattern in a string ?

 * (tau) counts the occurrence of each pattern or each term in a text.

Extracting the position of a substring ?

 * (cwhmisc) returns the position of a substring in a string.
 * (cwhmisc) does the same job but returns the first and the last position.

Extracting the position of a pattern in a string ?

 * returns the position of the regular expression.  (stringr) does the same job.   is similar to   but the starting position of every match is returned.   (stringr) does the same job.

Extracting a fixed width substring ?

 * takes a sub string.
 * (stringr) is similar.

Extracting the first word in a string ?

 * First Word in a String or Expression in the Hmisc package

Extracting a pattern in a string ?

 * returns the value of the regular expression if  and its position if.


 * ,,  ,   (stringr) and   (caroline package) are similar to  .   and   return a vector.   and   return a matrix and   a dataframe.


 * Named capture regular expressions can be used to define column names in the regular expression (this also serves to document the regular expression). Install the namedCapture package via  to use  . It uses the base function   to parse a Perl-Compatible Regular Expression, and returns a list of match matrices with column names:

Substituting a pattern in a string

 * makes a substitution.
 * is similar to  but replace all occurrences of the pattern whereas   only replaces the first occurrence.
 * (stringr) is similar to sub,  (stringr) is similar to gsub.

In the following example, we have a French date. The regular pattern is the following : 2 digits, a blank, some letters, a blank, 4 digits. We capture the 2 digits with the  expression, the letters with   and the 4 digits with. Each of these three substrings is surrounded with parenthesis. The first substring is stored in, the second one in   and the 3rd one in.

In the following example, we compare the outcome of  and. The first one removes the first space whereas the second one removes all spaces in the text.

Substituting characters in a string ?

 * substitutes characters in an expression. It stands for "character translation".
 * (cwhmisc) does the same job ...
 * as well as  (stringr).

Converting letters to lower or upper-case

 * converts upper-case characters to lower-case.
 * converts lower-case characters to upper-case.
 * (Hmisc) capitalize the first letter of a string
 * See also,  ,  ,   and   in the cwhmisc package.

Filling a string with some character

 * (cwhmisc) fills a string with some characters to fit a given length. See also  (stringr).

Note that  is very slow. For instance for a vector of length 10,000, we have a very long computing time. does not seem to handle character vectors but the best solution may be to use the  and   functions together.

Removing leading and trailing spaces

 * (memisc package) trim leading and trailing white spaces.
 * (gdata package) does the same job.
 * See also  (stringr)

Assessing if they are identical

 * returns TRUE if both strings are the same and false otherwise.

Computing distance between strings
Few packages implement the Levenshtein distance between two strings:


 * in base package utils
 * in MiscPsycho
 * in stringdist
 * in RecordLinkage

A benchmark comparing the speed of  and   is available here:.

Example with MiscPsycho
(MiscPsycho) computes If   the levenshtein distance is divided by the maximum length of each string.

Approximate matching
search for approximate matches using the Levenshtein distance.
 * If 'value = TRUE', this returns the value of the string
 * If 'value = FALSE' this returns the position of the string
 * max returns the maximal levenshtein distance.

Miscellaneous

 * : Turn unevaluated expressions into character strings.
 * (base) expands a string with respect to a target.
 * (base) and  (base) seek matches for the elements of their first argument among those of their second.
 * makes a character string unique. This is useful if you want to use a string as an identifier in your data.