..
Copyright (c) 2012-2013 Varnish Software AS
SPDX-License-Identifier: BSD-2-Clause
See LICENSE file for full text of license
.. _phk_varnish_does_not_hash:
=====================
Varnish Does Not Hash
=====================
A spate of security advisories related to hash-collisions have made
a lot of people stare at Varnish and wonder if it is affected.
The answer is no, but the explanation is probably not what most of
you expected:
Varnish does not hash, at least not by default, and
even if it does, it's still as immune to the attacks as can be.
To understand what is going on, I have to introduce a concept from
Shannon's information theory: "entropy."
Entropy is hard to explain, and according to legend, that is exactly
why Shannon recycled that term from thermodynamics.
In this context, we can get away with thinking about entropy as how
much our "keys" differ::
Low entropy (1 bit):
/foo/bar/barf/some/cms/content/article?article=2
/foo/bar/barf/some/cms/content/article?article=3
High entropy (65 bits):
/i?ee30d0770eb460634e9d5dcfb562a2c5.html
/i?bca3633d52607f38a107cb5297fd66e5.html
Hashing consists of calculating a hash-index from the key and
storing the objects in an array indexed by that key.
Typically, but not always, the key is a string and the index is a
(smallish) integer, and the job of the hash-function is to squeeze
the key into the integer, without losing any of the entropy.
Needless to say, the more entropy you have to begin with, the more
of it you can afford to lose, and lose some you almost invariably
will.
There are two families of hash-functions, the fast ones, and the good
ones, and the security advisories are about the fast ones.
The good ones are slower, but probably not so much slower that you
care, and therefore, if you want to fix your web-app:
Change::
foo=somedict[$somekey]
To::
foo=somedict[md5($somekey)]
and forget about the advisories.
Yes, that's right: Cryptographic hash algorithms are the good ones,
they are built to not throw any entropy away, and they are built to
have very hard to predict collisions, which is exactly the problem
with the fast hash-functions in the advisories.
-----------------
What Varnish Does
-----------------
The way to avoid having hash-collisions is to not use a hash: Use a
tree instead. There every object has its own place and there are no
collisions.
Varnish does that, but with a twist.
The "keys" in Varnish can be very long; by default they consist of::
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (hash);
}
But some users will add cookies, user identification and many other
bits and pieces of string in there, and in the end the keys can be
kilobytes in length, and quite often, as in the first example above,
the first difference may not come until pretty far into the keys.
Trees generally need to have a copy of the key around to be able
to tell if they have a match, and more importantly to compare
tree-leaves in order to "re-balance" the tree and other such arcanae
of data structures.
This would add another per-object memory load to Varnish, and it
would feel particularly silly to store 48 identical characters for
each object in the far too common case seen above.
But furthermore, we want the tree to be very fast to do lookups in,
preferably it should be lockless for lookups, and that means that
we cannot (realistically) use any of the "smart" trees which
automatically balance themselves, etc.
You (generally) don't need a "smart" tree if your keys look
like random data in the order they arrive, but we can pretty
much expect the opposite as article number 4, 5, 6 etc are added
to the CMS in the first example.
But we can make the keys look random, and make them small and fixed
size at the same time, and the perfect functions designed for just
that task are the "good" hash-functions, the cryptographic ones.
So what Varnish does is "key-compression": All the strings fed to
hash_data() are pushed through a cryptographic hash algorithm called
SHA256, which, as the name says, always spits out 256 bits (= 32
bytes), no matter how many bits you feed it.
This does not eliminate the key-storage requirement, but now all
the keys are 32 bytes and can be put directly into the data structure::
struct objhead {
[...]
unsigned char digest[DIGEST_LEN];
};
In the example above, the output of SHA256 for the 1 bit difference
in entropy becomes::
/foo/bar/barf/some/cms/content/article?article=2
-> 14f0553caa5c796650ec82256e3f111ae2f20020a4b9029f135a01610932054e
/foo/bar/barf/some/cms/content/article?article=3
-> 4d45b9544077921575c3c5a2a14c779bff6c4830d1fbafe4bd7e03e5dd93ca05
That should be random enough.
But the key-compression does introduce a risk of collisions, since
not even SHA256 can guarantee different outputs for all possible
inputs: Try pushing all the possible 33-byte files through SHA256
and sooner or later you will get collisions.
The risk of collision is very small however, and I can all but
promise you, that you will be fully offset in fame and money for
any inconvenience a collision might cause, because you will
be the first person to find a SHA256 collision.
Poul-Henning, 2012-01-03
Henceforth, whatever our philosopher says about Matter will apply to extension and to extension alone. It cannot be apprehended by sight, nor by hearing, nor by smell, nor by taste, for it is neither colour, nor sound, nor odour, nor juice. Neither can it be touched, for it is not a body, but it becomes corporeal on being blended with sensible qualities. And, in a later essay, he describes it as receiving all things and letting them depart again without retaining the slightest trace of their presence.483 Why then, it may be asked, if Plotinus meant extension, could he not say so at once, and save us all this trouble in hunting out his meaning? There were very good reasons why he should not. In the first place, he wished to express himself, so far as possible, in Aristotelian phraseology, and this was incompatible with the reduction of Matter to extension. In the next place, the idea of an infinite void had been already appropriated by the Epicureans, to whose system he was bitterly opposed. And, finally, the extension of ordinary327 experience had not the absolute generality which was needed in order to bring Matter into relation with that ultimate abstraction whence, like everything else, it has now to be derived. That the millionaire was genuine, ¡°in person and not a caricature,¡± as Dick put it, was evident. Both the nurse, his relative, and his wife, were chatting with him as Jeff delivered the heavy packed ball made up of the gum. 233 "I guess not," said Landor, tolerantly, as he turned[Pg 106] his horse over to his orderly; "but, anyway," he added to Ellton, "we had a picnic¡ªof a sort." Si, unable to think of anything better, went with him. The train had stopped on a switch, and seemed likely to rust fast to the rails, from the way other trains were going by in both directions. The bridge gang, under charge of a burly, red-faced young Englishman, was in the rear car, with their tools, equipments, bedding and cooking utensils. THE DEACON HAS SOME EXPERIENCES WITH THE QUADRUPED. "You are not within a mile of the truth. I know it. Look here: I believe that is Gen. Rosecrans's own cow. She's gone, and I got an order to look around for her. I've never seen her, but from the description given me I believe that's she. Who brought her here?" "Deacon, these brothers and sisters who have come here with me to-night are, like myself, deeply interested in the moral condition of the army, where we all have sons or kinsmen. Now, can't you sit right there and tell us of your observations and experiences, as a Christian man and father, from day to day, of every day that you were down there? Tell us everything, just as it happened each day, that we may be able to judge for ourselves." HAS AN ENCOUNTER WITH THE PROVOST-MARSHAL. "Wonder which one o' them is the 200th Injianny's?" said Si to Shorty. "And your mother, and Harry?" The daughter must be the girl who was talking to him now. She sat on a little stool by the fire, and had brought out some sewing. "Over at Grandturzel¡ªcan't see wot's burning from here. Git buckets and come!" These things, however, gave little concern to the worthy who commanded the Kentish division. Tyler, though an excellent blacksmith, possessed few of the qualities requisite for forming a good general. Provided there was no very sensible diminution in the number of his followers, he cared not a straw for the score or two who, after quarrelling, or perhaps fighting, withdrew in such disgust that they vowed rather to pay the full tax for ever than submit to the insolence of the rebels. One man could fight as well as another, reasoned he; and, provided he was obeyed, what mattered it by whom. Dick went and Tom came¡ªit was sure to be all one in the end. But this burst of indignation soon passed away, and upon the suggestion of the prudent Sir Robert Hailes, he sent an evasive answer, with a command that the Commons should attend him at Windsor on the Sunday following. That it was a stratagem to gain entrance to the Tower, was the opinion of several, but, after much discussion, it was decided that the man should be admitted, and that the monk should be exhibited merely to intimidate the rebels, until the result of this promised communication should be known. HoMEŮͬÐÔÁµcbcb
ENTER NUMBET 0017
yefen1.com.cn
erli0.net.cn
gemi2.com.cn
renwei8.com.cn
wdjp.com.cn
www.canmi2.net.cn
wohua6.com.cn
qujun4.com.cn
houxi8.net.cn
www.8webfind.com.cn