Text as Numbers

Characters are numbers

A computer can’t store the letter A, only bits. So text is stored as numbers, one (or more) per character, by agreement about which number means which character: a character encoding. Getting that agreement right is why text usually just works, and getting it wrong is why it sometimes turns into Ã¥.

ASCII

ASCII, the American Standard Code for Information Interchange, is the original agreement still underneath everything: 128 codes, 0 to 127, which fit in seven bits ¹. 95 are printable, the digits, upper- and lowercase English letters, and punctuation; 33 are control codes from the days of Teletypes, of which newline and tab are the ones you’ll meet ¹. A is 65, a is 97, and the digit 0 is 48.

xxd shows the bytes in a file or a pipe, in hex, with the text alongside:

me@linuxbox:~$ printf 'A' | xxd
00000000: 41                                       A
me@linuxbox:~$ printf 'Hi!\n' | xxd
00000000: 4869 210a                                Hi!.

41 is 65 in hex; 0a is 10, the newline, shown as . in the text column because it isn’t printable. Python converts between characters and numbers with ord and chr ²:

me@linuxbox:~$ python3 -c 'print(ord("A"), chr(97))'
65 a

Unicode

ASCII has no å, no ö, no €, no Japanese, no emoji. Unicode assigns every character in every writing system a number called a code point, from 0 up to 0x10FFFF, about 1.1 million possible values, written like U+00E5 for å ³. Its first 128 code points are exactly ASCII ¹.

me@linuxbox:~$ python3 -c 'print(ord("€"), hex(ord("€")))'
8364 0x20ac

The euro sign is code point U+20AC. A code point is just a number, though; to store it, it has to be turned into bytes, and that’s what an encoding like UTF-8 does.

UTF-8

UTF-8 is one of the most commonly used encodings ³, and the one everything in this course’s Linux tools expects. Its rules ^{3 4}:

• A code point below 128 (ASCII) is one byte, the same byte as in ASCII. So plain ASCII text is already valid UTF-8 ³.
• Anything higher becomes two, three, or four bytes, each between 128 and 255 ³.

The bytes follow a pattern you can read ⁴:

The x bits hold the code point; the first byte’s leading 1s say how many bytes follow, and continuation bytes start with 10. Here’s å, U+00E5, in bits:

me@linuxbox:~$ printf 'å' | xxd -b
00000000: 11000011 10100101                                      ..

110 then 00011, and 10 then 100101: put the x bits together, 00011100101, and that’s 0xE5. Three- and four-byte characters work the same way:

me@linuxbox:~$ printf '€' | xxd
00000000: e282 ac                                  ...
me@linuxbox:~$ printf '🦊' | xxd
00000000: f09f a68a                                ....

Because every byte of a multi-byte character is 128 or more, UTF-8 never confuses part of a character with an ASCII one, and if bytes are lost, a reader can find the start of the next character again ³.

Characters versus bytes

So the length of text depends on how you count:

me@linuxbox:~$ printf 'Hej då\n' | xxd
00000000: 4865 6a20 64c3 a50a                      Hej d...
me@linuxbox:~$ python3 -c 'print(len("Hej då"), len("Hej då".encode("utf-8")))'
6 7

Six characters, seven bytes, because å takes two. .encode("utf-8") turns text into bytes ³:

me@linuxbox:~$ python3 -c 'print("å".encode("utf-8"))'
b'\xc3\xa5'

When the encoding is wrong

Bytes don’t say which encoding wrote them. Read UTF-8 bytes as if they were an older one-byte encoding, Latin-1, and each byte of å becomes its own character:

me@linuxbox:~$ python3 -c 'print("Hej då".encode("utf-8").decode("latin-1"))'
Hej då

That Ã¥ is the classic sign of UTF-8 text displayed with the wrong encoding, common in old emails and CSV files from older programs. The fix is to read the file as UTF-8, not to change the text. file guesses a file’s encoding ⁵:

me@linuxbox:~$ file utf8.txt
utf8.txt: Unicode text, UTF-8 text

Line endings

One more difference hides at the end of every line. Linux ends lines with a single LF, line feed, byte 0a, written \n. Windows ends them with CR LF, carriage return then line feed, 0d 0a, written \r\n:

me@linuxbox:~$ printf 'line\r\n' | xxd
00000000: 6c69 6e65 0d0a                           line..
me@linuxbox:~$ file crlf.txt
crlf.txt: ASCII text, with CRLF line terminators

A script saved on Windows with CRLF endings can fail on Linux with a confusing error, since the shell sees the invisible \r as part of each line. This is what git’s core.autocrlf setting and its line-ending warnings are about (Git course, module 1).

Your turn

So

Text is numbers. ASCII gives 128 characters a byte each; Unicode gives every character a code point, up to U+10FFFF; UTF-8 stores ASCII as one byte and everything else in two to four, following a pattern of leading bits you can decode by hand. xxd shows the bytes, Python’s ord, chr, and .encode convert, and file guesses an encoding. Lengths differ in characters and bytes, Ã¥ means UTF-8 read the wrong way, and Linux ends lines with LF where Windows uses CR LF.