Computer Security lecture notes

Copyright © 2009 Mark Dermot Ryan
The University of Birmingham
Permission is granted to copy, distribute and/or modify this document
(except where stated) under the terms of the GNU Free Documentation License,

One-way secure hash functions

Secure hash functions

Purpose and usage. Secure one-way hash functions (also known as message digest functions) are intended to provide proof of data integrity, by providing a verifiable fingerprint of the data. A one-way hash function H operates on an arbitrary length input message M, returning h=H(M). The important properties are:

Given M, easy to compute h=H(M)
Given h, hard to compute M such that h=H(M) -- "one-way", or "pre-image resistant"
Given M, hard to find M' (different from M) such that H(M)=H(M') -- "second-preimage resistant"
(Not always satisfied) Hard to find M,M' such that H(M)=H(M') -- "collision resistant"

Note that 4 implies 3 (i.e. if we could solve 3 we could solve 4), but not conversely. Also, 3 implies 2, but not conversely. The strange thing about hash functions is that there are typically billions of collisions, or perhaps infinitely many (if the hash function really does take arbitrary-length input; most have some huge limit). But it is computationally hard to find a single one.

Examples of usage:

unix passwords: when the user selects a password p, the system stores H(p) instead of p itself. When the user later tries to log in with password p, the system again computes H(p) and compares it with the stored value. If they are the same, the user is granted access. The advantage of this scheme is that the system doesn't store the password p. Even if the user breaks into the computer and steals the password file, it is useless to him

Which properties 1-4 does this scheme require?

integrity of downloaded files: when I download a file (say some software), I would like to check that I have received it correctly, i.e.

it wasn't accidentally or maliciously corrupted in transit
it wasn't accidentally or maliciously corrupted on the server.

fingerprints of public keys. Public keys are generally large (e.g., minimum 1024 bits, which is 128 bytes). Checking their integrity with H(k) is easier than checking k itself.
Signing messages. Sign the hash instead.
Commitments. Alice can commit to a value v by sending H(v,r) where r is some new random. To prove she committed to v, she reveals r.
File identification in peer-to-peer networks, video references on youtube, etc.

Typically, a single bit swap in the input swaps about half the bits in the output. Example:

[mdr@hubert]$ md5sum
There is $1500 in the blue box.
05f8cfc03f4e58cbee731aa4a14b3f03 -
[mdr@hubert]$ md5sum
There is $1100 in the blue box.
d6dee11aae89661a45eb9d21e30d34cb -

Brute force attacks on hash functions

The Birthday "paradox" is the surprising answer to this question: how many people must I gather in a room in order to have a probability >0.5 that two of them share the same birthday? (We assume that the birthdays are distributed randomly.) The answer is lower than one might guess: 23. Compare that with this question: how many people must I gather in a room in order to have a probability >0.5 that one of them share the same birthday as me? The answer is much greater: 253.

Suppose there are N possible hash values, and suppose that the output of a hash function is randomly distributed in this space. Now take a set of n strings.

How big does n have to be in order to have a probability >0.5 of some string in that set having a given hash value? The answer is: n must be at least n = ln 2 / ( ln N - ln(N-1) ), which for large N is approximately (ln 2)*N.
Thus, for a 128 bit hash function, you need to test (ln 2) 2^128 inputs (approximately 10^38) to get a 0.5 chance of preimaging the hash, that is to say, of getting a given hash value.

How big does n have to be in order to have a probability >0.5 of two strings in that set having the same hash value? The probability of no duplicate is

(N-1)/N * (N-2)/N * . . . * (N-n+1)/N	=	(1-(1/N)) * (1-(2/N)) * . . . * (1-((n-1)/N)
	<	e^-1/N * e^-2/N * . . . * e^-(n-1)/N
	=	e^-n(n-1)/2N

The middle inequality comes from 1-x<e^-x. Setting this to be 0.5, approximating n(n-1) as n² and solving for n gives n=sqrt(2 * (ln 2) * N ).
Thus, to merely witness a collision for a 128 bit has function, you can do it in much less time: n=10¹⁹ inputs gives us more than 0.5 chance of a clash.

To try to put these numbers into perspective: 10¹⁹ microseconds is 317 000 years, while 10³⁸ microseconds is 10²⁴ years.

The MD5 secure hash algorithm

MD5 was designed in 1991 by Ron Rivest, and is widely used; for example, it is used by Red Hat so that users can verify that packages they download have not been tampered with (e.g., by introducing trapdoors); PGP uses it for message signatures. MD5 takes an input of up to 2⁶⁴ bits (approx 10⁹ Gigabytes), and produces a 128-bit hash (how many collisions do you think there are?). Here is how it works.

Define four bit-munging functions on 32 bit variables:

F( X, Y, Z ) = (X && Y) || (!X && Z)

and similar for functions G, H, I. Define four subroutines like this:

FF( a, b, c, d, M, s, t ) is a := b + ( (a + F(b,c,d) + M + t) <<< s )

where <<< s means left-circular shift of s bits; and similar subroutines GG, HH, II. The algorithm works as follows.

Pad the input so that its length is just 64 bits short of being a multiple of 512 bits. Then, append a 64-bit representation of the input's length (before padding).
Initialise 32-bit variables A, B, C, D to some specific constants.
For each 512-bit block M
{

(M₀, M₁, ..., M₁₅) := M;
(a,b,c,d) := (A,B,C,D);
/* Round 1 */
FF( a, b, c, d, M₀, 7, 0xd76..... );
FF( d, a, b, c, M₁, 12, 0x..... );
/* a further 14 calls to FF like these ones, totalling 16 calls */
/* Rounds 2, 3, 4 are similar, consisting of 16 calls to GG, HH, II */
(A,B,C,D) :+= (a,b,c,d)

}
Output the concatenation (A,B,C,D).

The answer to the question about how many collisions there are in MD5 is this: each input collides with 2^2⁶⁴ / 2¹²⁸ others, on average; this number is approximately 2^2⁶⁴ which is approximately 10^10¹⁸. It seems paradoxical that collisions are so hard to find.

MD5 was invented Rivest in response to partial cryptanalyses of its predecessor MD4 (also by Rivest). The improvements MD5 has over MD4 are just quantity rather than style (four rounds instead of three, etc). SHA is also an MD4 derivative, but outputs 160 bits rather than 128. MD5 was thought to be secure until 2004, even though prior to that its security had been partly eroded; e.g., a paper by Hans Dobbertin in 1996 succeeded in producing a collision to a variant of MD5.

MD5 broken

Collisions for the full MD5 were announced on 17 August 2004 at CRYPTO2004, by Xiaoyun Wang, Dengguo Feng, Xuejia Lai and Hongbo Yu. Their work is a combination of theoretical analysis (based on differential cryptanalysis) which reduces the computational problem, and brute force attack on the reduced computational problem, and they received a standing ovation for their work. Later, on 1 March 2005, Arjen Lenstra, Xiaoyun Wang, and Benne de Weger demonstrated the construction of two X.509 certificates with different public keys and the same MD5 hash, a demonstrably practical collision. The construction included private keys for both public keys.

Do we need collision-resistance?

These are still only collisions, i.e. breaks of the strongest property in the list 1-4 above. One might argue that md5 is still secure for use where only properties 1-3 are required. Many of the applications that use cryptographic hashes, such as password storage or document signing, are in principle only minimally affected by a collision attack. In the case of document signing, for example, an attacker could not simply fake a signature from an existing document -- the attacker would have to fool the private key holder into signing a preselected document. Reversing password hashing (e.g. to obtain a password to try against a user's account elsewhere) does not require collision resistance. Constructing a password that hashes to a given value requires a preimage attack.

Here are two reasons for believing that collision-resistance is important:

Ways of finding collisions can often easily be adapted for finding a useful collision. For example,

Alice prepares two versions of a contract. C1 is favourable to her, C2 favourable to Bob.
She makes several unnoticible changes to each of C1 and C2, and computes the hash of each. The changes are things like extra space at end of line, or double space between words. By making or not making a single change to each of 32 lines, she could easily produce 2^32 versions.
She compares the hash value of each variation of C1 with each variation of C2, until she finds a collision h(C1')=h(C2').
She gets Bob to sign C2', and then later claims he signed C1'.

For some hash functions h, a randomcollision h(m1)=h(m2) can be made into a potentially more useful collision by prepending a fixed message p, and/or appending a fixed message q. In other words,h(m1)=h(m2) may imply h(p.m1)=h(p.m2), or h(m1.q)=h(m2.q), or even h(p.m1.q)=h(p.m2.q). For a dramatic application of this idea, read on.

Colliding postscript files

Because MD5 makes only one pass over the data, if two prefixes with the same hash can be constructed, a common suffix can be added to both to make the collision more reasonable. And because the current collision-finding techniques allow the preceding hash state to be specified arbitrarily, a collision can be found for any desired prefix. All that is required to generate two colliding files is a template file, with a 128-byte block of data aligned on a 64-byte boundary, that can be changed freely by the collision-finding algorithm. Thus, it is possible to construct two colliding Postscript files with the same size with arbitrary contents, e.g.

Letter 1: Joe Bloggs is an excellent student, and I highly recommend him for the position you are offering. He has done brilliant work in his project under my supervision. I have been amazed at what he has managed to achieve, blah bhah blah
Letter 2: I cannot recommend Joe Bloggs for the position you are offering. He registered to do a project with me but he has never attended any meetings. I have seen very little of his work and I suspect he hasn't done anything. Blah blah

Other hash functions

The literature contains a big family of has functions, including MD{2-5}, SHA-{1,256,384,512} and RIPEMD{,-160}. They all work on similar principles to MD5, but they differ in how much munging they do and how many bits they produce.

SHA-1 is a direct competitor with MD5, since both of them were developed at the same time in an effort to strengthen MD4. SHA-1 produces 160 bits; thus its "collision difficulty" is much higher, 2^80 = 10^24 compared to 2^64 = 10^19 for MD5. So it is 10000 times as secure against brute force attacks.

	SHA-1	SHA-256	SHA-384	SHA-512
Message digest size	160	256	384	512
Message size	<2^64	<2^64	<2^128	<2^128
Block size	512	512	1024	1024
Word size	32	32	64	64
Number of steps	80	80	80	80

Cryptanalysis techniques have already begun eroding the security of SHA-1. Although a full collision has not yet been obtained, the same group in China (Xiaoyun Wang, Yiqun Lisa Yin, and Hongbo Yu) announced in February 2005 that collisions of the full version of SHA-1 could be found requiring fewer than 2⁶⁹ operations (a brute-force search would require 2⁸⁰). Later, the reduced this to 2⁶³. Any analysis which reduces the complexity of a brute force attack is considered to be a break of the cryptography. In an interview, Yin states that, "Roughly, we exploit the following two weaknesses: One is that the file preprocessing step is not complicated enough; another is that certain math operations in the first 20 rounds have unexpected security problems." [5]

So what now? Migration to stronger hashes...

Keyed hashes

Keyed hashes (also known as MACs, for "message authentication codes") are hash functions parameterised by a key. They can be used to provide authenticity (since a message accompanied by a keyed hash may be considered signed by a person who has the key).

Ordinary hashes can be used to make keyed hashes. A simple idea is to set HMAC_K(M) = h( K, M ), but this is insecure: (depending on the hash function) the attacker may be able to compute h(K,M') out of h(K,M) where M' is a superstring of M. An alternative is HMAC_K(M) = h(M, K ), but if h is one-way but not collision-free, this could be broken as well. The possibility HMAC_K(M) = h(K, M, K) has been considered, but the preferred one appears to be HMAC_K(M) = h(K1, h(K2, M)) where K1 and K2 are keys derived from K:

Let h be a hash. Let K be the key, padded it with zeros to make it the length of h's block size, b. Let

ipad = 00110110 (=0x36), repeated b/8 times
opad = 01011100 (=0x5C), repeated b/8 times
K1 = K + opad
K2 = K + ipad, where + is XOR

Then HMAC_K(M) = h( K1 , h( K2, M ) )

References

[1] Bruce Schneier, Applied Cryptography. Second Edition, J. Wiley and Sons, 1996.
[2] William Stallings, Cryptography and Network Security, Principles and Practice, Prentice Hall, 1999. Third Edition, 2003.
[3] The University of British Columbia Theoretical Physics department has a web page on cryptography, with lots of interesting remarks and links.
[4] Wikipedia articles on MD5 and SHA1.