How to use HKDF to derive new keys Mon Jan 30 2023 HKDF is a great tool to derive sub-keys from a master key, however it is often misused! Be careful about the salt parameter and use info correctly! -------------------------------------------------------------------------------- How to use HKDF to derive new keys ================================== Published Jan 30, 2023 - 40 min read /---------- Table of contents ---------\ | Table of contents | | * HKDF misuse | | * Indistinguishable from random | | * Key Expansion | | * Planning for the future | | * HKDF Sub-key derivation on the fly | | * Case Study: AnonCo | | * Private key derivation | | * Last tweaks | | * One more thing | | * Conclusion | | * Small Addendum | \--------------------------------------/ We rely on several cryptographic tools constructed together to secure our lives. Many depend on hidden information with specific properties to provide the security benefits they claim. It is not always convenient to agree upon or distribute a large amount of hidden information like a One-time pad [L1]. Key Derivation Functions (KDFs) reliably create unrelated keys for different cryptographic tools from a single Input Key Material (IKM). In fact, a KDF helped you view this article, specifically the HMAC-based Key Derivation Function (HKDF). While correctly used in your browser, it is often misused as I will show in critiquing an anonymized example I found online in a publication by AnonCo. /[cendyne: good-luck-im-behind-7-proxies]--------------------------------------\ | I will refer to the organization that misused HKDF as AnonCo. Please focus | | on learning from this content rather than finding the source. | \------------------------------------------------------------------------------/ Just about every cryptographic tool out there can be used wrong. Deploying correct cryptography is hard, so hard that you should never do it alone. /------------------------------------------------------------------------------\ | SwiftOnSecurity @SwiftOnSecurity@twitter.com | |------------------------------------------------------------------------------| | Tried explaining danger of homebrew crypto to a journalist in DM | | "Cryptography is nightmare magic math that cares what kind of pen you use." | | [L2] 2/16/2017 | \------------------------------------------------------------------------------/ /[jacobi: teaching]------------------------------------------------------------\ | Cryptography is a science that uses hard mathematical problems to make | | specific threat models practically infeasible to execute. Not all threat | | models are equal. For example, if you are preventing complete theft, you may | | not be preventing partial theft. | \------------------------------------------------------------------------------/ /------------------------------------------------------------------------------\ | Benedict Evans @benedictevans@twitter.com | |------------------------------------------------------------------------------| | @SwiftOnSecurity [L3] @ryfar [L4] | | [I1: Photo included with tweet] | | | | [L5] [L6] 2/16/2017 | \------------------------------------------------------------------------------/ Unlike the watermelons above, AnonCo's misuse does not open them up to any new threats. It happens to meet AnonCo's functional goals more by accident than on purpose. In cryptography, accidents are dangerous and not something to joke about. [I2: An adjustable wrench is a pin between a truck and a trailer. It is being called an adjustable hammer.] /-------------------------------------------------------[kyrmeso: im-in-danger]\ | I would not want to be hit by a loose trailer just because they used a | | wrench as a hitch pin. | \------------------------------------------------------------------------------/ HKDF misuse ----------- In this article, "misuse" has a specific meaning: a cryptographic tool is not delivering all intended security properties because it is not used correctly. /[cendyne: derp]---------------------------------------------------------------\ | If you're like me, you might be stuck on some of the details and not get the | | why the security properties change when misused. After all, it still looks | | like some random data comes out at the end. | \------------------------------------------------------------------------------/ /[cendyne: snake-on-a-plane]---------------------------------------------------\ | This took a lot of vacation time for me to figure out, but it was fun in its | | own way. I hope my words save you some time! | \------------------------------------------------------------------------------/ /[cendyne: reading]------------------------------------------------------------\ | I highly recommend you read Soatok's article: Understanding HKDF [L7] ( | | archived [L8]). It goes into great technical detail which I think any | | aspiring cryptographer should reference. | \------------------------------------------------------------------------------/ Here are a few ways HKDF can be misused: * In a public setting, no salt is given. * The salt input is not indistinguishable from random (IND). * The salt input is used for domain separation. * The same salt is used across multiple transactions in a public setting. * Different salts are used in a private setting. * The inputs to HKDFs are the same for different contexts, resulting in the same keys for different purposes. /[jacobi: studying]------------------------------------------------------------\ | Cryptographic Extraction and Key Derivation: The HKDF Scheme [L9] describes | | several weaknesses observed in KDF use at the time. It contributes HKDF | | which is a robust KDF in that the derived keys are still practically | | unlinkable and unguessable even when HKDF is misused. | \------------------------------------------------------------------------------/ /[jacobi: shrug]---------------------------------------------------------------\ | For example, it is not the end of the world if the salt input is not | | indistinguishable from random. HKDF works well for private deterministic use | | cases where the salt is not provided. When there is no salt, the salt is | | equivalent to 000000000... | \------------------------------------------------------------------------------/ /[jacobi: point-up]------------------------------------------------------------\ | Though, if your public protocol implements unique keys for each session, | | review the above misuses and consider employing a salt which cannot be | | directly manipulated by either party and is otherwise indistinguishable from | | random. | \------------------------------------------------------------------------------/ /[jacobi: peeking]-------------------------------------------------------------\ | That said, the scope of this article is for HKDFs and key-based key | | derivation functions for key expansion in a private setting. | \------------------------------------------------------------------------------/ /[jacobi: bye]-----------------------------------------------------------------\ | Aside, if you're here looking for info on password-based key derivation | | functions (PBKDFs), sorry to say this isn't the right reference! | \------------------------------------------------------------------------------/ Indistinguishable from random ----------------------------- /[jacobi: angel]---------------------------------------------------------------\ | A cryptographically-suitable key is a sequence of bits that are | | indistinguishable from random. In academic literature, it is typically | | contracted to IND. Others may say that it is "uniformly random" as it was | | selected from a uniform distribution of possible configurations. | \------------------------------------------------------------------------------/ /[jacobi: bullshit]------------------------------------------------------------\ | If there is a bias in which states are selected, then it is not uniformly | | random. Likewise, if there is a bias when it is encoded to binary, then it | | is not uniformly random in binary. | \------------------------------------------------------------------------------/ /[jacobi: hooray]--------------------------------------------------------------\ | In practical terms an IND bit string cannot be meaningfully analyzed. It is | | also impractical to guess or brute force when it is sufficiently long. | \------------------------------------------------------------------------------/ /[jacobi: dizzy]---------------------------------------------------------------\ | You may see names like AES-128 or SHA-256 and wonder what that's about. The | | names refer to how many bits there are in their configuration or design. | \------------------------------------------------------------------------------/ /[jacobi: hmm]-----------------------------------------------------------------\ | Asymmetric keys and asymmetric computations have distinctive patterns, which | | makes them unsuitable for use in symmetric cryptography like AES-256. See | | Why some cryptographic keys are much smaller than others [L10] (archived | | [L11]) by CloudFlare. A distinctive pattern makes it not IND! | \------------------------------------------------------------------------------/ /[jacobi: talking]-------------------------------------------------------------\ | That's why we reach for KDFs! It is a tool that produces IND bit strings. | | That said, this article will focus on IND coming in and IND coming out. | \------------------------------------------------------------------------------/ /[cendyne: laptop-thinking]----------------------------------------------------\ | Also: in software, we pass byte strings around in memory to solve problems. | | When I mention bytes or bits, I mean the same thing. | \------------------------------------------------------------------------------/ Key Expansion ------------- /[jacobi: thumbs-up]-----------------------------------------------------------\ | Key expansion is a function that takes a cryptographically-suitable key | | (which is IND) and a context as input and outputs IND sequence of bits of a | | desired length. | \------------------------------------------------------------------------------/ /------------------------------------------------------------------------------\ | How KDFs secure websites | |------------------------------------------------------------------------------| | /[cendyne: hello]----------------------------------------------------------\ | | | When you connected to this web page, your machine and the server created | | | | a shared secret, a shared salt, and then used HKDF to create multiple | | | | IND keys to secure and authenticate data going in each direction. | | | \--------------------------------------------------------------------------/ | | | | /[jacobi: reading]---------------------------------------------------------\ | | | See A Cryptographic Analysis of the TLS 1.3 Handshake Protocol [L12] for | | | | an incredible breakdown of how it works. And for those wanting a less | | | | scientific document, see David Wong's A Readable Specification of TLS | | | | 1.3 [L13] (archived [L14]) section 7.1. | | | \--------------------------------------------------------------------------/ | \------------------------------------------------------------------------------/ /-------------------------------------------------------------[ibzan: thinking]\ | You mentioned desired length. That suggests the length is configurable? | | Could I make a gigabyte of data with it? | \------------------------------------------------------------------------------/ /[jacobi: standard]------------------------------------------------------------\ | It is flexible by design to support all sorts of use cases. It can even | | generate multiple keys at once with a long output length! | \------------------------------------------------------------------------------/ /[jacobi: surprise]------------------------------------------------------------\ | But do not use it as a stream cipher [L15]. It is not meant for that! | \------------------------------------------------------------------------------/ /[cendyne: ych-bonked]---------------------------------------------------------\ | HKDF limits how long its output can be. At most you can get around 8KB (with | | SHA-256) using HKDF. It will only call HMAC up to 255 times in the expand | | phase. See Boring Crypto hkdf.cc line 72 [L16] (archived [L17]) and RFC5869 | | [L18] Section 2.2. | \------------------------------------------------------------------------------/ /-------------------------------------------------------------------[ibzan: oh]\ | Multiple keys? I see only one output. Where are the other keys? | \------------------------------------------------------------------------------/ /[cendyne: teaching]-----------------------------------------------------------\ | The output is just a byte array so you can slice it up as much as you like | | as long as each slice is non-overlapping. | \------------------------------------------------------------------------------/ /[jacobi: thumbs-up]-----------------------------------------------------------\ | Since any slice of an IND bit string is also IND, it is still suitable for | | cryptographic use. See AES-GCM-SIV: Specification and Analysis [L19] section | | 4 for an example on how they expand a key and then slice it up. | \------------------------------------------------------------------------------/ Planning for the future ----------------------- /-------------------------------------------------------------[ibzan: thinking]\ | Every week, my application adds another feature. I cannot make an ever | | growing byte array of secret key data. What should I do about that? | \------------------------------------------------------------------------------/ /[jacobi: laptop-thinking]-----------------------------------------------------\ | This depends on the risks you're willing to accept in the name of | | simplicity. You might have just one key for database encryption, as an | | example. This might be fine if you have a small amount of data. A | | cryptographer can help you understand the risks your implementation has and | | what steps should be taken to improve the security of your data and users at | | scale. | \------------------------------------------------------------------------------/ /-----------------------------------------------------------------[soatok: sip]\ | Tech bloggers, whether or not they are also cryptographers, are not your | | cryptographer. At minimum, they don't know your threat model or systems | | designs. This is an incredibly specialized domain that's easy to get wrong. | | Hire a cryptographer. | \------------------------------------------------------------------------------/ /[jacobi: flails]--------------------------------------------------------------\ | Whatever you do, never use the same key in different cryptographic | | operations! For example, do not use an RSA public key as an HMAC key [L20] ( | | archived [L21]). | \------------------------------------------------------------------------------/ /--------------------------------------------------------------[ibzan: looking]\ | I am looking to encrypt data with different keys without having to provision | | those keys manually. And eventually, I want to encrypt even more different | | things with their own keys without saving the keys that were generated. What | | 's the tool to do that? | \------------------------------------------------------------------------------/ /[jacobi: point-right]---------------------------------------------------------\ | A tool for just-in-time deterministic sub-key derivation is HKDF. | \------------------------------------------------------------------------------/ /------------------------------------------------------------------------------\ | Seriously, a warning: do not do cryptography alone. | |------------------------------------------------------------------------------| | /[jacobi: surprise]--------------------------------------------------------\ | | | Here be dragons, but cryptography! There are still some security | | | | concerns left unaddressed here. What if one of the output keys gets | | | | compromised? How will you rotate that out and prevent new things from | | | | being encrypted with it? | | | \--------------------------------------------------------------------------/ | | | | /[jacobi: glassy-tears]----------------------------------------------------\ | | | This article is not a guide for key life-cycles and key management. You | | | | need a cryptographer to help you with that problem, not a blog post on | | | | the internet and certainly not an answer on Stack Overflow / Stack | | | | Exchange. | | | \--------------------------------------------------------------------------/ | | | | /[jacobi: judges]----------------------------------------------------------\ | | | Do not naively copy what you see recommended here into your application. | | | | What I share and describe is to illustrate the security properties of | | | | the tools we have available. It is up to you and a cryptographer to | | | | build something beautiful and trustworthy together. | | | \--------------------------------------------------------------------------/ | \------------------------------------------------------------------------------/ HKDF Sub-key derivation on the fly ---------------------------------- As a recap, here's where we are in this story: 1. we have Input Key Material (IKM) with enough entropy [L22] 2. we know what cryptographic operation will use this key, and its requirements (e.g. the length of the key) 3. we have a unique label for what it will be used for (e.g. encrypting a certain database table and column) 4. we desire keys on the fly for the operation we are about to do 5. we must be able to create the same keys on the fly again for another operation at a later time And the interface to HKDF looks something like this: interface HKDF { extractAndExpand( key: Uint8Array, salt: Uint8Array | null, info: Uint8Array | null, length: number ) : Uint8Array } /---------------------------------------------------------------[ibzan: shades]\ | Hey, about that application problem, how do I solve it? | \------------------------------------------------------------------------------/ First, we put in our input key material (IKM). Second, our problem does not involve another party executing cryptography. Therefore, we do not put anything into the salt. /-----------------------------------------------------------------[ibzan: gasp]\ | Why not put anything into the salt? | \------------------------------------------------------------------------------/ /[jacobi: teaching]------------------------------------------------------------\ | The salt is for solving a problem between two parties. We want to resist | | analysis when both sides use the same secret key. The salt does not have to | | be secret, but it should have a proportionate amount of IND data. It is | | directly used as an HMAC key in the extract phase on the shared secret. | \------------------------------------------------------------------------------/ /[jacobi: content]-------------------------------------------------------------\ | Since our use case is private and not with another party, the salt provides | | no additional benefit because our context is private. Including a | | satisfactory salt requires us to carefully create, store, recall, and handle | | more data. Without the salt, the extract phase will use an all-zero array as | | the key. This is acceptable for this use case: we are deriving keys for | | ourselves from a suitable IKM. | \------------------------------------------------------------------------------/ /---------------------------------------------------------------[ibzan: wizard]\ | What about on passwords? I see salts on MD5 and bcrypt passwords. | \------------------------------------------------------------------------------/ /[cendyne: ych-slap]-----------------------------------------------------------\ | Don't use MD5. Salt means something slightly different for passwords. It | | exists to increase the computational complexity of creating and maintaining | | a precalculated password database called a Rainbow table [L23]. For key | | exchanges, unique salts eliminate the risks of reusing a shared secret. | \------------------------------------------------------------------------------/ /[cendyne: scrungly]-----------------------------------------------------------\ | Unfortunately bcrypt truncates passwords [L24] (archived [L25]), so consider | | others like Argon2 [L26]. | \------------------------------------------------------------------------------/ Third, for info, we put in a unique label. HKDF could be called twice with the same key, salt, and length, but have different labels like "encryption key" and "authentication key". /[jacobi: talking]-------------------------------------------------------------\ | The info parameter is used to separate keys from one another. It is also a | | good idea to have predictable fixed length info values. You will have to | | exercise caution if variable length data is included. | \------------------------------------------------------------------------------/ And fourth, for length, we provide the key size we want. In this case it is likely bytes, so for 256 bits: length = 32. /[cendyne: fellow-kids]--------------------------------------------------------\ | Many APIs use bytes instead of bits. Be sure to review the documentation of | | the implementation! Note that NIST standards require additional data on info | | , such as the length as bits on the very end. [L27] (archived [L28] line | | 191) | \------------------------------------------------------------------------------/ Finally HKDF can be called and the IND output can be used for encrypting, authenticating, or some other neat thing! /-------------------------------------------------------------[ibzan: surprise]\ | What was that label thing about? | \------------------------------------------------------------------------------/ /[jacobi: wink]----------------------------------------------------------------\ | It is important that we do not reuse keys for different purposes. The label | | is an application-provided string that says what the purpose is. In essence, | | this input is used for domain separation in the expand phase of HKDF. That | | expand phase is what provides KDF security. | \------------------------------------------------------------------------------/ /[jacobi: wtfdidijustread]-----------------------------------------------------\ | If it is possible that the same label is used for two purposes, you have a | | canonicalization problem. | \------------------------------------------------------------------------------/ /-------------------------------------------------------------[ibzan: thinking]\ | What is KDF security? | \------------------------------------------------------------------------------/ /[jacobi: ok]------------------------------------------------------------------\ | In practical terms: if one of the keys made by the KDF is leaked, the other | | keys made by the same KDF are still safe. In addition, all the keys made are | | IND. This second property comes from PRF security. All KDFs are also PRFs. | \------------------------------------------------------------------------------/ /-------------------------------------------------------------[ibzan: thinking]\ | What is canonicalization? | \------------------------------------------------------------------------------/ /[jacobi: typing]--------------------------------------------------------------\ | Canonicalization is the process of taking multiple pieces of data and | | serializing it together in an unambiguous way. If any of the pieces change, | | then the output is also changed. This is especially useful in verifying data | | that can be reorganized in transit. | \------------------------------------------------------------------------------/ /[cendyne: you-got-it]---------------------------------------------------------\ | For example, see Google Cloud Storage - Canonical Requests [L29] (archived | | [L30]) and AWS - Create a signed AWS API request [L31] (archived [L32]). | \------------------------------------------------------------------------------/ /[cendyne: wtf-is-going-on]----------------------------------------------------\ | In practice, canonicalization is an incredibly difficult process to get | | right. See Duo Finds SAML Vulnerabilities Affecting Multiple Implementations | | [L33] (archived [L34]), where Kelby shares a vulnerability that affects | | multiple SAML implementations. | \------------------------------------------------------------------------------/ /[cendyne: uhh]----------------------------------------------------------------\ | Do not blame the users or library authors when the same issue keeps | | appearing. Instead, the specification and technology is prone to misuse. So, | | alternatives should be considered. | \------------------------------------------------------------------------------/ Case Study: AnonCo ------------------ AnonCo's product relies on a technology that abstracts database storage and database operations. Additionally, they use a compatible security dependency which facilitates seamless encryption and decryption when it goes in and out of the database to the application. /-----------------------------------------------------[kyrmeso: do-you-mean-me]\ | Why might we want to use application side encryption? I can encrypt the | | database in AWS. | \------------------------------------------------------------------------------/ /[cendyne: todays-interesting-thing]-------------------------------------------\ | At rest encryption, or encrypting the disk or whatever only matters if your | | attacker can get to the disk. If they can clone the database in AWS, change | | the database's password from AWS, and then sneak inside to dump it, guess | | what? All that data is available for the threat actor to pilfer. | \------------------------------------------------------------------------------/ /[cendyne: not-stonks]---------------------------------------------------------\ | The method described above is literally what happened in the breach I | | experienced. At rest encryption in the cloud is check-box security and | | nothing more. | \------------------------------------------------------------------------------/ However, that security library does not provide per-column encryption keys, which is a feature that AnonCo wants. AnonCo has a lot of customers they need to protect across many integrated products. At their scale, it is a good idea to encrypt each sensitive database field with a different key. Unfortunately, the plumbing to do this requires a product developer to add a new key correctly each time they need to add or migrate an encrypted database field! /[cendyne: dying-at-computer]--------------------------------------------------\ | Not only that, but also the ongoing maintenance of adding new keys to | | mitigate key exhaustion! Holy yak shaving, Batman! | \------------------------------------------------------------------------------/ This not only disincentives secure development, it also introduces the chance of an accidentally reusing keys or introducing weak (not IND) keys through manual process! /---------------------------------------------------------------[kyrmeso: math]\ | How often do I need to rekey? | \------------------------------------------------------------------------------/ /[cendyne: drawing]------------------------------------------------------------\ | It is generally a good idea to swap a new primary key every year, or every | | billion operations for most businesses. To really know: hire a cryptographer | | to give you a recommendation. Like being told to get a lawyer, this advice | | cannot get old if you care about doing the job correctly. | \------------------------------------------------------------------------------/ /-----------------------------------------------------------------[soatok: sip]\ | Tech bloggers, whether or not they are also cryptographers, are not your | | cryptographer. At minimum, they don't know your threat model or systems | | designs. This is an incredibly specialized domain that's easy to get wrong. | | Hire a cryptographer. | \------------------------------------------------------------------------------/ /[cendyne: battery]------------------------------------------------------------\ | I've heard stories of how some big tech teams blindly get told to use a new | | key every day, even if they only store about 1000 records a day. They'd | | store the daily key in DynamoDB encrypted with a KMS master key. When | | reading a record, they'd find the key ID on the record, read the | | corresponding key from Dynamo, and decrypt the record. The proportion of | | data to key period here is in my opinion excessive. | \------------------------------------------------------------------------------/ AnonCo tried to automate key provisioning to eliminate manual process using HKDF. /[cendyne: youtube]------------------------------------------------------------\ | An author at AnonCo shared how their team solved a problem with HKDF. It was | | a genuine team effort that successfully shipped to production at AnonCo. | | There are mistakes and I have responsibly and respectfully communicated my | | critique with constructive recommendations. | \------------------------------------------------------------------------------/ /[cendyne: you-got-it]---------------------------------------------------------\ | I highly respect this team and so I will not be sourcing the example I | | critique. In fact, the example is rewritten in another language to further | | distance the source from this article. | \------------------------------------------------------------------------------/ There are multiple problems with the approach AnonCo used, which I will cover! Here's the important code, which is translated for anonymity. export class ColumnEncrypt { private salt: Uint8Array; private encoder: TextEncoder; private constructor(encoder: TextEncoder, table: string, column: string) { this.encoder = encoder; this.salt = encoder.encode(`${table}_${column}`); } async buildKey(encryptionKey: CryptoKey) : Promise { let data = `${new Date().getFullYear()}`; let key = await crypto.subtle.deriveKey( { name: 'HKDF', salt: this.salt, info: this.encoder.encode(data), hash: 'SHA-256' }, encryptionKey, {name: 'AES-GCM', length: 256}, false, ['encrypt', 'decrypt']); return key; } static async newInstance(table: string, column: string) : Promise { return new ColumnEncrypt(new TextEncoder(), table, column); } } The key they're getting out will look functional, but it will not have the security properties one expects from a KDF. First: salt is being given the label! Again, the salt is meant to resist analysis of a shared secret in a public transaction. /-------------------------------------------------------[kyrmeso: glassy-tears]\ | This salt thing is confusing. | \------------------------------------------------------------------------------/ /[jacobi: notes]---------------------------------------------------------------\ | HKDF was designed to solve an important problem in key agreement which | | produces a shared secret with a mathematical operation in a public setting. | \------------------------------------------------------------------------------/ /[jacobi: excited]-------------------------------------------------------------\ | For example: Finite Field Diffie-Hellman (FFDH) and Elliptic Curve Diffie- | | Hellman (ECDH) produce hard-to-guess secrets which when encoded in binary | | are not IND. | \------------------------------------------------------------------------------/ /[jacobi: happy]---------------------------------------------------------------\ | It was also designed as a KDF that safely handles our use case, when it is | | well understood and applied. | \------------------------------------------------------------------------------/ Private key derivation ^^^^^^^^^^^^^^^^^^^^^^ Most security engineers do not write protocols between peers, servers, or clients. They write solutions to problems within their organization. Distributing secrets is a solved problem. Let's assume that AnonCo can distribute an IND secret IKM to their servers and that the key was correctly created. /[cendyne: heres-johnny]-------------------------------------------------------\ | If you are deriving keys from text like "hunter2" or doing something silly | | like hashing a UUID sha256("9f1bf359-054c-4e3d-8845-ff6cb928c311"), I will | | hunt you down. | \------------------------------------------------------------------------------/ /[jacobi: screaming-inside]----------------------------------------------------\ | In both cases, not enough hidden knowledge is given to the key derivation | | key to maintain its expected security properties in computational | | complexity. HKDF uses HMAC which uses a hash. Since the HMAC key is the same | | size as the hash block size, it should receive an IND key of same size as | | the underlying hash. For example, when using HKDF with SHA-256, aim to | | supply an IND key with 256 bits. | \------------------------------------------------------------------------------/ /[jacobi: idea]----------------------------------------------------------------\ | It may be helpful to think of a Key Derivation Key (KDK) as a combination of | | Input Key Material (IKM) and a Key Derivation Function (KDF). In this case, | | the KDF is HKDF. | \------------------------------------------------------------------------------/ > Deirdre Connolly > > One of the things that you worked on that kind of is touching on tink is > binding properties of key material to the key material itself, as opposed to > it kind of being defined by a standard or out of band or stored somewhere > else. > > Could you tell us a little more about that? > > Tink with Sophie Schmieg - Security. Cryptography. Whatever. [L35] > Sophie Schmieg > > A lot of problems stem from the fact that when we talk about keys, we think > about the 32 or 16 raw bytes of random data that don't include the full > information of how it will be used. The right way, in my opinion, to handle > these things is to always consider the key as the entirety of the function or > functions that it defines. Like, just given the key, I should be able to > encrypt something. > > I shouldn't need any additional context for that. And that means I need to > know: do I use AES-GCM with this key? Do I use as AES-CTR-HMAC or something > with that key? And, this is a fairly simple concept in some aspects, like I > just put everything into the key and then I get like a very straightforward > API where I just have a function called encrypt that takes a plaintext and > some associated data and then just encrypts that. Because the key includes > everything else that you need to know. > > But it also has a lot of security benefits with it. I usually do not want to > use the same key material in two independent contexts. And that means I do not > want to use the same key material with two different algorithms. > > Tink with Sophie Schmieg - Security. Cryptography. Whatever. [L35] The above transcript has minor edits for clarity. You can generate an IND secret key easily with a command like this: $ openssl rand -hex 32 0de81e851cd7995626ad4c3e160ae1c449af4e15c8ceabd44fb75be581adfbaa Then, in your application, decode the hex and now you got 32 bytes or 256 bits of entropy [L22] to use as a master KDK! Assume that from now on, we will be using the binary form, not the hex form, as an Input Key Material (IKM). By definition, the hex form is not IND. If we use IND key material, our application has less computational overhead for the same level of security. /[cendyne: no-bad]-------------------------------------------------------------\ | Hey! Do not copy this key 0de81... and use it in your configuration! Never | | copy what looks like random material in a blog and use it in your code! | | Seriously, don't be like Hyundai: Hyundai Uses Example Keys for Encryption | | System [L36] (archived [L37]). | \------------------------------------------------------------------------------/ /------------------------------------------------------------------------------\ | A short rant on API quality | |------------------------------------------------------------------------------| | /[cendyne: 01010101]-------------------------------------------------------\ | | | Experienced developers see "salt" and set it like a password salt | | | | thinking it will make the construction more secure. | | | \--------------------------------------------------------------------------/ | | | | /[cendyne: this-is-not-how-you-make-friends]-------------------------------\ | | | HKDF is an incredible tool! But, the documentation we have for it (see | | | | Web Crypto API - HkdfParams [L38]) lacks useful examples on how to use | | | | HKDF correctly and for which circumstances. | | | \--------------------------------------------------------------------------/ | | | | /[cendyne: i-have-none]----------------------------------------------------\ | | | In JavaScript, salt is a required parameter! It is supposed to be | | | | optional! This is not C. | | | \--------------------------------------------------------------------------/ | | | | /[cendyne: bad-time]-------------------------------------------------------\ | | | It should not be so hard to do the right thing. | | | \--------------------------------------------------------------------------/ | | | | Here is how to correctly use HKDF in JavaScript to derive new keys, such as | | a signing key, deterministically from an IKM of sufficient entropy [L22]. | | | | // We generated this above as an example IND key derivation key | | let openSSLSecretKey = | | '0de81e851cd7995626ad4c3e160ae1c449af4e15c8ceabd44fb75be581adfbaa'; | | | | // Parse the hex string into a Uint8Array | | let ikm = Uint8Array.from(openSSLSecretKey | | .match(/.{1,2}/g) | | .map((byte) => parseInt(byte, 16))); | | | | // Import the raw key data | | let kdk = await crypto.subtle.importKey( | | 'raw', | | ikm, | | 'HKDF', | | false, // KDF keys cannot be exported | | ['deriveKey', 'deriveBits']); | | | | // We are going to create a signing key from the secret | | // If we create other keys too, they should not have the | | // same label! | | let label = 'signing key'; | | | | // This function works with bytes. | | // Therefore we must encode our label which is text to bytes. | | let encoder = new TextEncoder(); | | let info = encoder.encode(label); | | | | // A salt is a required property, even though it is empty. | | let salt = new Uint8Array(); // Nothing inside! | | | | // Derive a signing key from the key derivation key | | let signingKey = await crypto.subtle.deriveKey( | | // Again, the salt is empty | | // The info will uniquely describe this key | | {name: 'HKDF', salt, info, hash: 'SHA-256'}, | | // The input key material we decoded from hex above | | // and then wrapped in a CryptoKey | | kdk, | | // We're creating an HMAC-SHA-256 key | | {name: 'HMAC', hash: 'SHA-256'}, | | // We do not need to export it, | | // since we can create it deterministically. | | false, | | // it needs to sign and verify | | ['sign', 'verify']); | | | | // Prove that it works | | // Let's sign "Hello world" | | let message = 'Hello world'; | | let encodedMessage = encoder.encode(message); | | let tag = await crypto.subtle.sign( | | {name: 'HMAC'}, | | signingKey, | | encodedMessage); | | | | console.log(`Message ${message} - tag: ${btoa(tag)}`); | | // Message Hello world - tag: W29iamVjdCBBcnJheUJ1ZmZlcl0= | | | | // And prove that it can match its own mac too. | | let verified = await crypto.subtle.verify( | | {name: 'HMAC'}, | | signingKey, | | tag, | | encodedMessage); | | | | console.log(`Verify?: ${verified}`); | | // Verify?: true | | | | /[cendyne: well-yes-but-actually-no]---------------------------------------\ | | | This is a sketch. An example. It is not a complete reference. If you are | | | | creating keys on the fly for different purposes you must do more! I | | | | share some hints below. | | | \--------------------------------------------------------------------------/ | \------------------------------------------------------------------------------/ A brief reminder of what AnonCo's source looks like: async buildKey(encryptionKey: CryptoKey) : Promise { let data = `${new Date().getFullYear()}`; let key = await crypto.subtle.deriveKey( { name: 'HKDF', salt: this.salt, info: this.encoder.encode(data), hash: 'SHA-256' }, // ... ); return key; } Inside HKDF, it is doing something like this: // inputs input_key_material = encryption_key salt = "table_column" info = "2023" // Extract a key derivation key // The goal of extract is to produce an IND KDK // The input key material has enough hidden knowledge // to be an effective input key to the extract process key_derivation_key = HMAC(salt, input_key_material) // Expand the KDK as needed with info output_key = HMAC(key_derivation_key, info + "\x01") AnonCo should have an IND KDK coming in. /[jacobi: angel]---------------------------------------------------------------\ | If a hex string like 0de81e851cd799... happens to be the input key material, | | then HKDF will force it into an IND KDK with the extract phase. | \------------------------------------------------------------------------------/ Logically, transforming an IND IKM to an IND KDK of the same security level provides no benefit and only a minor performance penalty. The extract phase is inappropriately being used for domain separation, when the security goal is only to create an IND KDK. Then the expand phase creates a new unique key using... the year. Uhm, a year is not unique!!! Literally, for this use case, the info parameter must be unique. It would be far better to have the info set to `${table}_${column}_${year}`! /[cendyne: crossmark]----------------------------------------------------------\ | This is one of those sneaky ways where you can use cryptography and it looks | | and behaves the way you want on the outside, but fundamentally misses the | | security guarantees intended. | \------------------------------------------------------------------------------/ This mistake reduces the security guarantees to PRF security. /[jacobi: notes]---------------------------------------------------------------\ | Again, Pseudo-Random Function (PRF) security is that the output of a PRF is | | indistinguishable from random (IND). | \------------------------------------------------------------------------------/ /[cendyne: maybe-you-should-not]-----------------------------------------------\ | This is still good and meets the needs for an encryption operation, but it | | is not what HKDF is supposed to provide and that indicates HKDF is misused | | in this code. | \------------------------------------------------------------------------------/ /----------------------------------------------------------------[kyrmeso: beg]\ | Why does it not give KDF security? | \------------------------------------------------------------------------------/ /[cendyne: cooking]------------------------------------------------------------\ | The salt is only used in the extract phase in HKDF and if present should | | only have a single IND value. | \------------------------------------------------------------------------------/ /[cendyne: kneading-dough]-----------------------------------------------------\ | The extract phases is meant to provide a cryptographically-suitable IND key | | for the expand phase – which is what actually satisfies the security | | property when info is unique. | \------------------------------------------------------------------------------/ /[cendyne: i-have-questions]---------------------------------------------------\ | Instead the info parameter is given "2023" every time this year. I expect | | that multiple keys will be created in the year 2023. Inherently, this | | implementation is not aligned with the design of HKDF. | \------------------------------------------------------------------------------/ A modified version of the code that uses HKDF correctly is: export class ColumnEncrypt { private label: Uint8Array; private salt: Uint8Array; private encoder: TextEncoder; private constructor(encoder: TextEncoder, table: string, column: string) { this.encoder = encoder; this.salt = new Uint8Array(); this.label = encoder.encode(`${table}_${column}`); } async buildKey(keyDerivationKey: CryptoKey) : Promise { // Logically the label above will be prefixed to the data below let data = `_${new Date().getFullYear()}`; let info = new Uint8Array([ ...this.label, ...this.encoder.encode(data) ]) let key = await crypto.subtle.deriveKey( { name: 'HKDF', salt: this.salt, info, hash: 'SHA-256' }, keyDerivationKey, {name: 'AES-GCM', length: 256}, false, ['encrypt', 'decrypt']); return key; } static async newInstance(table: string, column: string) : Promise { return new ColumnEncrypt(new TextEncoder(), table, column); } } /----------------------------------------------------------------[kyrmeso: wut]\ | What happened to encryptionKey? | \------------------------------------------------------------------------------/ /[cendyne: typing]-------------------------------------------------------------\ | I renamed the parameter called encryptionKey to keyDerivationKey. We are not | | encrypting with this key, instead we are deriving keys. | \------------------------------------------------------------------------------/ /[jacobi: facepalm]------------------------------------------------------------\ | Names keep us more honest when there isn't a type system to resist misuse. | | As I said earlier, never use the same key in different cryptographic | | operations! | \------------------------------------------------------------------------------/ Last tweaks ----------- /[cendyne: stick-me-tail-out]--------------------------------------------------\ | Hold on to your tail because there is more to fix! | \------------------------------------------------------------------------------/ There is another problem here, and it is called canonicalization! What if the table were named "customers" and a column "last_order_id", while another table is called "customers_last_order" with a column named "id". And both were written to in the year 2023. Then both will have the info set to "customers_last_order_id_2023"! This problem also applies to the prior version with the salt receiving the label. Ooops! We just got a collision! How can we fix that? An easy and fast way is to prefix the length. private constructor(encoder: TextEncoder, table: string, column: string) { this.encoder = encoder; this.salt = new Uint8Array(); this.label = encoder.encode(`${table.length}:${table}_ ${column.length}:${column}`); } A more robust way is to ensure that the info is a constant length no matter what. An easy solution is to use a hash! However, this will have a performance penalty. export class ColumnEncrypt { private tableHash: Uint8Array; private columnHash: Uint8Array; private salt: Uint8Array; private encoder: TextEncoder; private constructor(encoder: TextEncoder, tableHash: Uint8Array, columnHash: Uint8Array) { this.encoder = encoder; this.salt = new Uint8Array(); this.tableHash = tableHash; this.columnHash = columnHash; } static async newInstance(table: string, column: string) : Promise { let encoder = new TextEncoder(); let tableHash = new Uint8Array(await crypto.subtle.digest( {name: 'SHA-256'}, encoder.encode(table) )); let columnHash = new Uint8Array(await crypto.subtle.digest( {name: 'SHA-256'}, encoder.encode(column) )); return new ColumnEncrypt(encoder, tableHash, columnHash); } async buildKey(keyDerivationKey: CryptoKey) : Promise { let data = `${new Date().getFullYear()}`; let info = new Uint8Array([ ...this.tableHash, ...this.columnHash, ...this.encoder.encode(data) ]) let key = await crypto.subtle.deriveKey( { name: 'HKDF', salt: this.salt, info, hash: 'SHA-256' }, keyDerivationKey, {name: 'AES-GCM', length: 256}, false, ['encrypt', 'decrypt']); return key; } } /[cendyne: wink2]--------------------------------------------------------------\ | The code above will produce constant length info values until the year | | 10000. | \------------------------------------------------------------------------------/ /------------------------------------------------------------[kyrmeso: unicorn]\ | What if I steal the key? You can't change the year! | \------------------------------------------------------------------------------/ /[cendyne: yes]----------------------------------------------------------------\ | You got me! Key management is a hard problem and it is out of scope for this | | article. | \------------------------------------------------------------------------------/ Now, there is no way that info will look the same for different purposes, as long as the key is not compromised. Therefore, the keys will always be unique between different purposes! /[jacobi: teaching]------------------------------------------------------------\ | We expect that hash functions never have collisions. Mathematically, this is | | impossible. For practical security, it is the best we can manage. | \------------------------------------------------------------------------------/ /[cendyne: ych-trashcan]-------------------------------------------------------\ | Once a collision has been found by accident, it is considered weakened. Once | | a collision can be intentionally found, it is broken. See shattered [L39] ( | | archived [L40]). | \------------------------------------------------------------------------------/ One more thing ^^^^^^^^^^^^^^ /[cendyne: hmm]----------------------------------------------------------------\ | The year is a big red flag to me for ignoring the key period. I'd be more | | comfortable with extended-nonce encryption, but I suspect that my | | preferences are incompatible with the API. | \------------------------------------------------------------------------------/ /[cendyne: if-i-fits-i-sits]---------------------------------------------------\ | Here's a sketch of how to do this with HKDF: | | | | In the box that gets encrypted, I'd tuck another nonce inside. This nonce | | would be added to the label in the info value when HKDF is called. And this | | nonce is not overlapping, shared with, or derived from the nonce used for | | encryption. | \------------------------------------------------------------------------------/ /[cendyne: pretzel]------------------------------------------------------------\ | Technically it changes it from every column having a set of encryption keys | | to every single field having its own encryption key, insofar as the | | encrypting primitive is concerned. | \------------------------------------------------------------------------------/ /[cendyne: huff-puff]----------------------------------------------------------\ | However, it means that every time a field is decrypted, there is a | | performance penalty of calling HKDF! We can do better without HKDF, see | | Extending the AES-GCM Nonce Without Nightmare Fuel [L41] (archived [L42]). | \------------------------------------------------------------------------------/ /[cendyne: i-dunno-man-im-just-a]----------------------------------------------\ | I digress. If they want to automate away key periods, they could have a | | feedback system that collects metrics for each encrypted field being written | | over time, create a prediction horizon for its period, and issue a new | | primary key at the period with a similar construction above without saving | | the nonce into the fields. A key ID should already be present on the record | | and is sufficient to identify the key once it is generated. | \------------------------------------------------------------------------------/ /[cendyne: driving-mad]--------------------------------------------------------\ | Honestly this sounds like something that would get patented, but should not. | \------------------------------------------------------------------------------/ Conclusion ---------- Cryptography is hard. There is a lot to consider when you use an existing well- studied construction and not all the information is clear. Much of it requires literally days or weeks of effort to understand the academic writings. It is okay to review what NIST wrote about a cryptographic tool. In fact, their documents are far more accessible than the academic publications out there. Unfortunately, NIST does not incorporate much of the newly contributed cryptography out there, so if you want to learn about the cool things you can do with Blake3, NIST will not satisfy your curiosity. There will be considerations and consequences that are not obvious, even to educated & experienced security engineers. Making security easy is hard. AnonCo believes that making the tools developers already use secure by default will result in better security for their users and customers. And I agree! I have seen product developers avoid the unfamiliar path to deliver changes, improvements, and bugs to the product. Security needs to be built into their tools to enable them to deliver quickly, not to hold them back. I do not have the full context of AnonCo's source code, but this misuse suggests that AnonCo needs to hire a cryptographer. There is likely more to find inside AnonCo. Small Addendum ^^^^^^^^^^^^^^ Database cryptography is hard. The above sketch is not complete and does not address several threats! This article is quite long, so I will not be sharing the fixes. Be aware of the following: * Invisible Salamanders [L43] : ciphertexts exist that can be decrypted successfully with authenticated encryption schemes with distinctly different keys. This demonstrates lack of key commitment. * Confused Deputy [L44] : an attacker swaps data around or presents a ciphertext intended for another party to an authorized decryptor called the "deputy." The deputy is confused and reveals the plaintext to the attacker. This demonstrates insufficient authentication. * And certainly more... Again, hire a cryptographer. -------------------------------------------------------------------------------- Indistinguishable from random (IND): It is computationally impractical to determine if an indistinguishable from random string was truly random or generated through a computational process. HMAC-based key derivation function (HKDF): Hashed Message Authentication Code (HMAC)-based key derivation function or HKDF is a concrete KDF implementation which uses HMACs to derive new key material. HKDF specifically employes two stages: extract and expand. The extract stage produces a temporary key derivation key which is uniformly random for the expand stage. The expand stage produces the requested length and binds the output to the domain provided in the context or info paremeter. Input Key Material (IKM): A secret which may become KDK after it is processed. Key Derivation Function (KDF): A Key Derivation Function or KDF takes a secret and can produce longer secrets from it for use in other applications. Since its output must be uniformly random, you will find a KDF relies upon PRFs to produce the desired output. To make longer outputs than the PRF supports, often a PRF gets a counter as part of its input which is incremented as needed until enough output data is available. Pseudo Random Function (PRF): A Pseudo Random Function or PRF is a keyed function that produces uniformly random data. It takes an input and reliably produces the same output with a fixed length. It sounds a lot like a hash function, and often is made with a hash function with additional mechanics around it. Hash based Message Authentication Code (HMAC): An HMAC is a MAC which uses hashes or message digests to authenticate a message. It is resistant to length extension attacks, which the underlying hash function may be susceptible to. You will find that common instatiations of HMAC are also PRFs. Key Derivation Key (KDK): The secret used to derive more keys through a KDF. [L1]: https://en.wikipedia.org/wiki/One-time_pad [L2]: https://twitter.com/SwiftOnSecurity/status/832055497251487744 [L3]: https://twitter.com/SwiftOnSecurity [L4]: https://twitter.com/ryfar [L5]: https://twitter.com/benedictevans/status/832135049537425408 [L6]: https://archive.vn/JG11Q [L7]: https://soatok.blog/2021/11/17/understanding-hkdf/ [L8]: https://archive.vn/JX612 [L9]: https://eprint.iacr.org/2010/264 [L10]: https://blog.cloudflare.com/why-are-some-keys-small/ [L11]: https://archive.vn/jhjMm [L12]: https://eprint.iacr.org/2020/1044 [L13]: https://www.davidwong.fr/tls13/#section-7.1 [L14]: https://archive.vn/wOQRW [L15]: https://en.wikipedia.org/wiki/Stream_cipher [L16]: https://github.com/google/boringssl/blob/ 254b8e113977aeb84ced559772537cd15bbe6cfc/crypto/hkdf/hkdf.c#L72 [L17]: https://archive.vn/NpKm6 [L18]: https://tools.ietf.org/html/rfc5869 [L19]: https://eprint.iacr.org/2017/168 [L20]: https://infosecwriteups.com/attacks-on-json-web-token-jwt-278a49a1ad2e [L21]: https://archive.vn/XIEbX [L22]: /topics/entropy.html [L23]: https://en.wikipedia.org/wiki/Rainbow_table [L24]: https://github.com/php/doc-en/issues/1328 [L25]: https://archive.vn/ZUNeX [L26]: https://en.wikipedia.org/wiki/Argon2 [L27]: https://github.com/ianmacd/gts6lwifi/blob/master/crypto/kbkdf.c#L191 [L28]: https://archive.vn/S21ue [L29]: https://cloud.google.com/storage/docs/authentication/canonical-requests [L30]: https://archive.vn/J8yHR [L31]: https://docs.aws.amazon.com/general/latest/gr/create-signed-request.html [L32]: https://archive.vn/OivEV [L33]: https://duo.com/blog/duo-finds-saml-vulnerabilities-affecting-multiple- implementations [L34]: https://archive.vn/qoAa1 [L35]: https://securitycryptographywhatever.buzzsprout.com/1822302/10697566-tink -with-sophie-schmieg [L36]: https://www.schneier.com/blog/archives/2022/08/hyundai-uses-example-keys- for-encryption-system.html [L37]: https://archive.vn/AEk93 [L38]: https://developer.mozilla.org/en-US/docs/Web/API/HkdfParams [L39]: https://shattered.io/ [L40]: https://archive.vn/63QrO [L41]: https://soatok.blog/2022/12/21/extending-the-aes-gcm-nonce-without- nightmare-fuel/ [L42]: https://archive.vn/VrI99 [L43]: https://eprint.iacr.org/2019/016 [L44]: https://en.wikipedia.org/wiki/Confused_deputy_problem [I1]: https://c.cdyn.dev/c7nW3nxr [I2]: https://c.cdyn.dev/D9E7gvzt