How Google played with bad cryptography
- 41 min read - Text OnlyWhile investigating one of my side projects, I discovered a bad pattern in Google's Services where service accounts (for machines not humans) authenticate in a way which requires a dangerous dance by Google to verify. One that I should hope others do not copy.
It's not just a big company that makes mistakes. Across the industry and through all levels of experience we see developers slip up and introduce vulnerabilities or risky code that evolves into a vulnerability later. Even senior developers with years of Amazon and Google on their resume will commit these vulnerabilities into production. I've seen it happen.
You might be familiar with one: SQL Injection!
What am I up to?
First an introduction to how I got to this problem.
November 2018, a senior engineer gave one days notice for a confidential integration with Google. I became his replacement on that project. No SDK was available for that confidential API and I had no idea how to authenticate with Google. At the time, Google pointed me to Using OAuth 2.0 for Server to Server Applications with the HTTP/REST documentation.
It took a few weeks, but they finally proposed a way to use their existing library in a way where I could extract the token and use it on HTTP requests to the confidential API.
Fast forward to the summer of 2022 and I'm looking at that OAuth 2.0 page again. My side project is to implement an OAuth 2.0 Server and Single Sign On portal.
Naturally I reviewed other implementations to see what standards they use (or inspired) and then make a judgement on if that standard is fit for my project.
What's wrong?
Remember how I mentioned that Broken Access Control and Cryptographic Failures are at the top of the OWASP security risks?
If you do something like this, you will be playing with fire too. So as a preface, do not follow Google's example in this specific case. You will be at risk of introducing broken access control and cryptographic failures.
As part of RFC7523, the client must create a JWT (see RFC7519) to post to the authorization server as an assertion to acquire an OAuth access token which is accepted at Google Cloud API endpoints.
Here's an example of what that looks like according to their documentation, though the JSON is base64 url encoded in practice (see RFC4648).
{"alg":"RS256","typ":"JWT"}.
{
"iss":"761326798069-r5mljlln1rd4lrbhg75efgigp36m78j5@developer.gserviceaccount.com",
"scope":"https://www.googleapis.com/auth/prediction",
"aud":"https://oauth2.googleapis.com/token",
"exp":1328554385,
"iat":1328550785
}.
[signature bytes]
When it is sent to the server, it looks something like this.
POST /token HTTP/1.1
Host: oauth2.googleapis.com
Content-Type: application/x-www-form-urlencoded
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer
&assertion=[base64 header].[base64 payload].[base64 signature]
Do you see the problem?
How does the authorization server know which digital signature key is to be used to verify the signature from the client?Answer: It is using the "iss"
claim (known as the issuer) which is the value of the OAuth 2.0 Client ID. These can be found on the GCP console under APIs & Services -> Credentials
Okay, but how is using an ID to find a key a bad thing? It isn't! In fact a key id which is known to both the client and authorization server has to be sent in the clear (or through some agreed upon mechanism) for the authorization server to know how to verify it.
Here's the problem: in order for Google's authorization server to determine the key to use to authenticate the data, the JWS payload (which should only be read after it is authenticated) is read and parsed prior to being authenticated. This is literally a tracked common weakness in software: CWE-345: Insufficient Verification of Data Authenticity.
So I bring it up with Sophie who happens to be at Google.
A Brief intermission about JWTs
I believe most software developers find JWTs easy to understand, accessible to implement or use through a third party library.
But developers and cryptographers have a different perspective.
The problem is JWTs are often used in places they were not designed for like browser session cookies. Developers interpret the spec as it was written without regard for its consequences. Surely the spec would have an out of the box secure design, right?
See "alg":"none"
.
If you'd like to learn about all sorts of common vulnerabilities with JWT implementations, see JWT Vulnerabilities (Json Web Tokens)
While we could beat a dead horse of alg none for months, let's focus back on the problem this article is concerned about.
Some libraries try to make it difficult to use them wrong. For example Java JWT: JSON Web Token (JJWT) for Java and Android Issue #86 made it hard to read data which should be authenticated prior to authentication.
Generally it's a good thing to have a library that prevents misuse! The problem is JWT's specification (and all JOSE specifications) has proliferated insecure design patterns and compatibility with these implementations introduce more vulnerabilities.
To say cryptographers dislike JWT is an understatement.
Earlier I mentioned what reduced risk for Injection: better tooling, static analysis technologies, and some education. I believe the same applies to Broken Access Control and Cryptographic Failures.
While the libraries can be improved, JJWT is a case study where safety choices clash with developers. This happens in practically every mainstream JWT library. The specification did not require the safety choices that JJWT implemented. Auth0, JJWT, and any other pragmatic implementation receive issues requesting unsafe features by developers. These issues come either to enable compatibility with something unsafe or personal feelings that this is best for them to get to production.
Luciano is saying the same thing I am. Reading and acting upon the JWT Payload claims is dangerous and should be avoided prior to being authenticated. Yet this is exactly what google suggests in their Google Cloud IAM documentation.
How can this be fixed?
I'll re-state problem: information required to determine the correct public key to authenticate the payload is inside the payload which should only be decoded after being authenticated.
The solution is to therefore move, copy, or reference that information outside the payload.
In this case, I believe that the information should be referenced.
Again, the pertinent key identifying information is the issuer which is set to 761326798069-r5mljlln1rd4lrbhg75efgigp36m78j5@developer.gserviceaccount.com
.
Adding a @code{client_id} to the request
Typically, an OAuth 2.0 request will include a client_id
when it is not in an authorization header.
It would look like this!
POST /token HTTP/1.1
Host: oauth2.googleapis.com
Content-Type: application/x-www-form-urlencoded
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer
&assertion=[base64 header].[base64 payload].[base64 signature]
&client_id=761326798069-r5mljlln1rd4lrbhg75efgigp36m78j5@developer.gserviceaccount.com
But of course, the client_id
must be consistent with the issuer inside. If not, then the client may be attempting a forgery.
OAuth 2.0 was designed to be extensible, and so the JWT method that Google employs has its own specification.
But the example shown in RFC7523 does not include a client_id
...
POST /token.oauth2 HTTP/1.1
Host: as.example.com
Content-Type: application/x-www-form-urlencoded
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer
&assertion=eyJhbGciOiJFUzI1NiIsImtpZCI6IjE2In0.
eyJpc3Mi[...omitted for brevity...].
J9l-ZhwP[...omitted for brevity...]
If you peak at the assertion, you'll see the header looks like the following when decoded.
{
"alg": "ES256",
"kid": "16"
}
Okay, so there's a kid
claim.
Using the JOSE Header
Instead, they hint in examples but not as words in the requirements that including a key identifier in the JOSE header is appropriate.
This is a common pattern. The specification permits a lot of freedom by not specifying the requirements for common cases. Instead they only hint within examples what should be done.
That said, I'll point back to the JOSE specification for a signed payload.
That wiggle room phrase "can be identified"... should be "should be identified" in my opinion for any asymmetric key material.
Anyway, the client_id
can be the kid
value in the JOSE header, or it could be the private key id (which is supplied with the key, see below). But the authorization server must do an equality check with the iss
(issuer) claim in the payload or check that the key id matches the issuer. A string equality check is not expensive.
Given that the key looks like (don't worry it is revoked!):
{
"type": "service_account",
"project_id": "iam-example",
"private_key_id": "0643fc8eee2c2c0360dde21ccd5399d2f1020d08",
"private_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpgIBAAKCAQEA+dk80a7PYILJgNh7/ED1NM9BmHYImnhqj6P2bo/7d2YpD1W5\nXS3GOjwAJ7vEriUiRa1W7NHZRdrgICjfDicE4DlteGQJJBpgz9JWsKIujg87N6fT\nlhX37uW6R7Nt/SOvOLby4xZvzQin/P1m+h5oANzwiuOpB1VNmKVFrNRR7y2x8WOt\nM2WyWdkQHBYnfKVqpij2hgOIedm3rh0Om4IJy350IMoW/R29BgyBXs44iRJbCV8+\nChO3x0VsM51Ck7AYrAdh+z8lvU1u71MbEIVajbzDe/kAJuolIt8bc7D/zVMCVufT\noMl3MBu4oNbwhGclKRF3NktcaZaB+Y/WNom3kwIDAQABAoIBAQDqQJnbZvEcZcOT\nwGWO/0BoASJZVeF/IwOWJX657tkw+2Hn9NHU4UQH+ZWTq2Mee8aEWZ80bxQtgKe+\nv1NTK5ZQvMc8p15CsVCvyWBqP8UygGlfJ0UkZPiOzmk3LK4lNz3kCPP1omW0cTc/\n5j6Up8mPdZc6QXWLYJleUybegjtH6U+8YLhH7YPqs8yM/wR+7Vf950GcCimO6pp4\n3TReaADeIBS/FLZkwrNu0E+6WAPBBR9gGwAtnPpuAT7WwY3jA/ukkpvyoqlptB44\nI6LrNS9CDUwT6mzkX8Qvj8BP5KFzxmKw6SrjLkwF7eDmMEmpPxrvmDrTDSDNRMP4\n5G92UqKhAoGBAP1qK1gDr8SX6Mskm+IMYaXIEMvP5o2CqTJHFbqMAUwyuv7NCywT\n0xikbivkHg3z5Z+pOifc6CdDDZVXVt6GHutd6k7mCUfm5tsthtDtaMf7ELFAZd6V\nzw/y39+0WgFXg8UDA0fBzo5nQQSDhaXw6qBo6lIKi1CmtxaRrjslpC1LAoGBAPxl\nwTd2KbfUHw0gjwm6WY4CZv6X95PL0W7U9f3YeS+gcTAN1pNp6GR/d3XflFGTGKYf\nXcWF/5O0KA3SXa+VvNhPpUz9GhXyeZNdREpGeXHjPKo2e0FNU4qlPoCzVCfDRxNv\nSmmKIirJGKtIIiel7E6PsceodHITF3m/Wreo+RnZAoGBAIkjTXWB+TrAoqBcnWdF\nIArhLAW/6pqmHP4ybdXYMlOUGJIPUH539AMf6Ocjugf+90LiB7DO4Wtt5Anvi/k8\nR7tDxasQ3fDlSgVOq+igsdWXTr89hGNiWv3ch76+EP8s5whUyw+oGCoEQrE4o7jb\nmX1ZiYUAY8gvkGFMUSd9BU3lAoGBAIhVTnDu2sn5QmyM0baneghDM+8BlzG2PoJn\ndhiP/aXEPF+Amg82fdkLITQCeNM3aXESMEypfMwD3D7bCs/1SfRt0RQtAxInz5PS\nJTkZqC/kVrh6hUlYw294orJSK3ru+E1/J+qqOppx1WlvpUNVVLd61sTKMVwNA/k3\na4EZPLTBAoGBAJHbJj5byljN25a96gVzuaK2RULxGY+khfhxoqmHUap8fWrtkTDY\n0kOaCb1OYXc0NP3vrwX63qKtIXWFzO8zduZH3/PdHHk15FN9FYqXRKScO4vq39hE\nLUkAtc/tKDUg6qnAWXF/tWO2oGAhlATSFFCOHf9jG9e+4QtXm7GoFXc/\n-----END RSA PRIVATE KEY-----\n",
"client_email": "761326798069-r5mljlln1rd4lrbhg75efgigp36m78j5@developer.gserviceaccount.com",
"client_id": "100048893883123634702",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/761326798069-r5mljlln1rd4lrbhg75efgigp36m78j5%40developer.gserviceaccount.com"
}
The JWT with the kid
properly populated would look more like...
BASE64URL({
"alg":"RS256",
"typ":"JWT",
"kid":"0643fc8eee2c2c0360dde21ccd5399d2f1020d08"
}).
BASE64URL({
"iss":"761326798069-r5mljlln1rd4lrbhg75efgigp36m78j5@developer.gserviceaccount.com",
"scope":"https://www.googleapis.com/auth/prediction",
"aud":"https://oauth2.googleapis.com/token",
"exp":1328554385,
"iat":1328550785
}).
[signature bytes]
Alternatively...
The iss
value could be in the JOSE header.
BASE64URL({
"alg":"RS256",
"typ":"JWT",
"iss":"761326798069-r5mljlln1rd4lrbhg75efgigp36m78j5@developer.gserviceaccount.com"
}).
BASE64URL({
"iss":"761326798069-r5mljlln1rd4lrbhg75efgigp36m78j5@developer.gserviceaccount.com",
"scope":"https://www.googleapis.com/auth/prediction",
"aud":"https://oauth2.googleapis.com/token",
"exp":1328554385,
"iat":1328550785
}).
[signature bytes]
Like the kid
version above, the iss
value in the JOSE header must be compared and found equal to the iss
claim so forgeries are prevented. At least if this exact example were to be followed.
The JOSE header is intended for reading prior to authentication! Like other request parameters, it is considered untrusted.
In practice though, some experts say ignore the header entirely. I think it still has some utility, just not as much as the specifications suggest is possible.
The specifications have consistently demonstrated design choices that lead to unsafe and dangerous implementations.
There are alternatives like PASETO, Macaroons, Biscuits, and more. Unfortunately, not all language have complete implementations.
So did Google mess up?
See the thing about internet specifications that reach maturity is that they usually come after the implementations work in production.
Google made their OAuth JWT token handshake prior to the standard being completed (literally years later).
In 2012, we see Google documented OAuth2 Service Accounts and today ten years later Preparing to make an authorized API call looks the same in substance.
Here's the timeline:
- April 2010: The first OAuth 2 draft was published, does not include Google authors.
- December 2010: The first JWT draft was published, several Google employees are authors.
- February 2012: Google committed that the
iss
claim is the OAuth 2client_id
to the Google API Java SDK. - May 2012: Google documents a new service account authentication mechanism with JWTs in production.
- May 2012: The JWT draft was forked and no longer includes Google authors.
- May 2012: The OAuth 2 JWT Profile draft was published, does not include Google authors.
- October 2012: The OAuth 2.0 Authorization Framework was finalized as RFC6749
- May 2015: The JWT specification was finalized as RFC7519.
- May 2015: The OAuth 2 JWT profile specification was finalized as RFC7523.
Judging from the timeline, and no other inside information, I think that Google gave up on participating in the standardization process. That said, the standardization still supports their implementation and this pattern in particular.
Since Google left the standardization process and had their work in production, it is hard to say that Google broke the specification when they launched. They made their own implementation, documented it, and launched before the standard was finalized.
It is completely possible to do this check safely and securely. But not by inexperienced hands.
Mimicking this implementation might only be safe if you...
- Extract the issuer and only the issuer from the JWT payload
- Discard the rest of the parsed payload
- Sanitize and reference the issuer to find the key
- Discard the issuer and process the JWT with the key
- And then finally go through with other claim validation
This approach is not aesthetic and is not something software developers would think necessary. But I think it is the only safe way to relpicate Google's specific implementation as an authorization server from the documentation.
Google can add a kid
claim, and in fact in the Addendum: Service account authorization without OAuth, they do include the kid
JOSE header claim. But this method is only for directly connecting to a supported API without getting an access token from the authorization server.
Bonus tomatoes
OpenID Connect which is built upon OAuth 2.0 gives a public warning when implementing with Google.
Where the OpenID Connect Spec requires:
It seems Google, at the time, chose consistency with their Service Account token implementation rather than to deviate to match the OpenID standard which Google definitely did participate in.
Note that the JWT spec permits any string or url (as a string) for the iss
claim.
As I review the OpenID Connect Core specification, it does look like kid
is required when key rotation is expected.
I checked Google's OpenID Implementation and I got a token that actually met my expectations unlike the Google Service Account IAM endpoint.
BASE64URL({
"alg": "RS256",
"kid": "1549e0aef574d1c7bdd136c202b8d290580b165c",
"typ": "JWT"
}).
BASE64URL({
"iss": "https://accounts.google.com",
"nbf": 1659843291,
"aud": "...",
"sub": "...",
"email": "cendyne@cendyne.dev",
"email_verified": true,
"azp": "...",
"name": "Cendyne Naga",
"picture": "https://lh3.googleusercontent.com/a-/AFdZucrTTbC2cy1hl-7TqKGjy2ZgmTgHWDkjgzdllxDG=s96-c",
"given_name": "Cendyne",
"family_name": "Naga",
"iat": 1659843591,
"exp": 1659847191,
"jti": "..."
}).
[SIGNATURE BYTES]
The kid
"1549..."
is publicly available at OAuth2/certs, a JWKs endpoint.
{
"keys": [
{
"use": "sig",
"n": "stD2wMn0t...",
"kty": "RSA",
"kid": "1549e0aef574d1c7bdd136c202b8d290580b165c",
"e": "AQAB",
"alg": "RS256"
},
{
"e": "AQAB",
"alg": "RS256",
"kid": "fda1066453dc9dc3dd933a41ea57da3ef242b0f7",
"n": "4DauU23AE...",
"kty": "RSA",
"use": "sig"
}
]
}
Unlike Google's Service Account IAM endpoint, the human accounts OpenID implementation does not force recipients to decode the JWT payload prior to authentication. Instead, the header is treated as an acceptable place to reference a key ID so that the complete JWT can be authenticated.
Final checks
Sometimes documentation is wrong and I want to get all the facts straight.
I checked the Java SDK for how it handles the private key id. Well, it turns out they set it now!
protected TokenResponse executeRefreshToken() throws IOException {
if (serviceAccountPrivateKey == null) {
return super.executeRefreshToken();
}
// service accounts: no refresh token; instead use private key to request new access token
JsonWebSignature.Header header = new JsonWebSignature.Header();
header.setAlgorithm("RS256");
header.setType("JWT");
header.setKeyId(serviceAccountPrivateKeyId); // <========
JsonWebToken.Payload payload = new JsonWebToken.Payload();
long currentTime = getClock().currentTimeMillis();
payload.setIssuer(serviceAccountId);
payload.setAudience(getTokenServerEncodedUrl());
payload.setIssuedAtTimeSeconds(currentTime / 1000);
payload.setExpirationTimeSeconds(currentTime / 1000 + 3600);
payload.setSubject(serviceAccountUser);
payload.put("scope", Joiner.on(' ').join(serviceAccountScopes));
try {
String assertion =
JsonWebSignature.signUsingRsaSha256(
serviceAccountPrivateKey, getJsonFactory(), header, payload);
TokenRequest request =
new TokenRequest(
getTransport(),
getJsonFactory(),
new GenericUrl(getTokenServerEncodedUrl()),
"urn:ietf:params:oauth:grant-type:jwt-bearer");
request.put("assertion", assertion);
return request.execute();
} catch (GeneralSecurityException exception) {
// ...
}
}
On June 4, 2014 Anthony Moore committed this fix to the Java SDK. Now kid
in the JWT header is populated with the private key ID!
It took Google two years in realize that there was a better way. And the specification isn't even finalized yet...
Conclusion
Google's IAM Authorization Server documentation suggests an unsafe design pattern where by they must process the JWT claims prior to authenticating the claims. This is dangerous and you should not follow that example. Existing specifications permit Google's previous and current implementations. I see these specifications as dangerous to naively implement. New implementations of these specifications will contain cryptographic failures, broken access control, and several other categories of the top ten OWASP risks.
Google improved their clients by adding a key identifier (kid
) to the JOSE header. By putting this in the header, they removed the need to read authenticated data prior to authentication. Hopefully Google's authorization server also verifies the key identifier and issuer match.
As a final reply to Sarai Rosenberg: