Ruby Programming/Encoding

Encoding Support
With the advent of Ruby 1.9, Ruby now supports encodings other than US-ASCII for strings, IO, and source code. On an installation of Ruby 1.9.3 MRI on Mac OSX Lion, Ruby has the following encodings:
 * ASCII-8BIT
 * UTF-8
 * US-ASCII
 * Big5
 * Big5-HKSCS
 * Big5-UAO
 * CP949
 * Emacs-Mule
 * EUC-JP
 * EUC-KR
 * EUC-TW
 * GB18030
 * GBK
 * ISO-8859-1
 * ISO-8859-2
 * ISO-8859-3
 * ISO-8859-4
 * ISO-8859-5
 * ISO-8859-6
 * ISO-8859-7
 * ISO-8859-8
 * ISO-8859-9
 * ISO-8859-10
 * ISO-8859-11
 * ISO-8859-13
 * ISO-8859-14
 * ISO-8859-15
 * ISO-8859-16
 * KOI8-R
 * KOI8-U
 * Shift_JIS
 * UTF-16BE
 * UTF-16LE
 * UTF-32BE
 * UTF-32LE
 * Windows-1251
 * IBM437
 * IBM737
 * IBM775
 * CP850
 * IBM852
 * CP852
 * IBM855
 * CP855
 * IBM857
 * IBM860
 * IBM861
 * IBM862
 * IBM863
 * IBM864
 * IBM865
 * IBM866
 * IBM869
 * Windows-1258
 * GB1988
 * macCentEuro
 * macCroatian
 * macCyrillic
 * macGreek
 * macIceland
 * macRoman
 * macRomania
 * macThai
 * macTurkish
 * macUkraine
 * CP950
 * CP951
 * stateless-ISO-2022-JP
 * eucJP-ms
 * CP51932
 * GB2312
 * GB12345
 * ISO-2022-JP
 * ISO-2022-JP-2
 * CP50220
 * CP50221
 * Windows-1252
 * Windows-1250
 * Windows-1256
 * Windows-1253
 * Windows-1255
 * Windows-1254
 * TIS-620
 * Windows-874
 * Windows-1257
 * Windows-31J
 * MacJapanese
 * UTF-7
 * UTF8-MAC
 * UTF-16
 * UTF-32
 * UTF8-DoCoMo
 * SJIS-DoCoMo
 * UTF8-KDDI
 * SJIS-KDDI
 * ISO-2022-JP-KDDI
 * stateless-ISO-2022-JP-KDDI
 * UTF8-SoftBank
 * SJIS-SoftBank

Using Encodings
By default, since Ruby 2.0 all Ruby source files are encoded with UTF-8. The usual way of changing the encoding of a file is to use a so called "magic comment". The magic comment must come directly at the beginning of the file, or directly after a | shebang comment. The syntax of the magic comment requires only one thing: The comment contains the text  followed by the name of the encoding. So the following are all valid magic comments:

The magic comment tells the interpreter that the source code itself, in addition to all of the strings in it, are going to be encoded with the given encoding. So, while this is valid code:

This is not:

Because the first snippet declares the files encoding to be ISO-8859-1 (an extension to US-ASCII that adds accented characters for languages such as French and Spanish), the character "é" is valid. However, in US-ASCII, "é" is an invalid character, and will cause an error.

Encodings and Individual Strings
You can also specify encodings for individual strings in your file (although the characters in literals must still be in the encoding declared by the magic comment, or inserted through escape sequences). This is done by two methods of the String class:  and.

is used for transcoding. Given, say, the ISO-8859-1 string "Olé!", you could use  to convert that to UTF-8, which has all of the same characters. However, you can not transcode the ISO-8859-1 string to US-ASCII, unless it contains only ASCII characters (eg. "Hello"). has many options and can be configured extensively. See its | documentation. Here's the catch of : It is very likely that while the visual display and meaning of the characters remain the same, the underlying bytes most likely will not. is free to change the underlying bytes of a string. Example:

is used to tell Ruby the encoding of a string that already has the correct bytes for that encoding (eg. a UTF-8 string read from a file in an ISO-8859-1 program). will never modify the underlying bytes of a string. Example:

ASCII-8 Bit
Ruby also includes a fake encoding: ASCII-8 Bit, or BINARY. BINARY is the encoding used for binary data.


 * 1) encoding: ISO-8859-1