Relying Upon Open Source and Log4j2

- 13 min read - Text Only

In private, public, commercial, personal, you-name-it projects, nearly every one relies on open source software embedded in the application somewhere.

There are some benefits to this:

  • The development's sponsor does not have to pay for the research and development of features provided by open source software.
  • Open source software gets updates, while stackoverflow posts do not, and some contain vulnerabilities. To be fair it really is on the human to understand what they are contributing to their own code and I agree.
  • Generated code from products like Github Copilot are insecure.
  • Updates to open source often come with new features that may be helpful or useful. Some features come through sponsored work so more can benefit.

Notably there are some negatives, but these negatives are often outweighed by the positives.

  • Supply chain attacks are a thing. Bumping a version does not in any way entail that a developer read all the changes involved and can sign off on them.
  • Legal risk that comes with the license attached to open source software.
  • The sponsor's use case may not be covered by or compatible with available open source software.

But there's one negative omitted from the above.

Software from any source may come with vulnerabilities and those using the software might never know.

Apache Log4j2 has had several undiscovered vulnerabilities for years. Only upon literally global attention after the first did more come out in the matter of days. There are ongoing Nation-State attacks targeting this vulnerability.

  • CVE-2021-44228 - Apache Log4j2 JNDI features do not protect against attacker controlled LDAP and other JNDI related endpoints.
  • CVE-2021-45046 - Apache Log4j2 Thread Context Lookup Pattern vulnerable to remote code execution in certain non-default configurations.
  • CVE-2021-45105 - Apache Log4j2 does not always protect from infinite recursion in lookup evaluation.

Practically everyone is impacted. Not using Log4j2 in your app? Maybe an agent that monitors your app is using log4j2. Or quite literally for me, my networking infrastructure:

For your privacy, this youtube video was not automatically loaded.
Click this area to load an embedded youtube video.
Oh what a bother. Sure, I'll update my unifi system, and a few lambdas at work that rely upon Log4j2. Not too hard.
For your privacy, this youtube video was not automatically loaded.
Click this area to load an embedded youtube video.
Oh come on! Just think of all the poor highly paid FAANG employees that can't do their job because the CI queue is backlogged?
For your privacy, this youtube video was not automatically loaded.
Click this area to load an embedded youtube video.
There is no escape. It is the weekend. Give me a break. No one is spared.
I wonder if equipment on mars will be updated...
Log4j on mars
Archived Tweet

An event like this however does not mean that open source software is inherently bad and vulnerable. Several argue that Open Source Software is less vulnerable because the developers actually want to release something of high quality, whereas closed source environments are constrained by cost and time.

Personally, I agree, I don't think that open source is inherently vulnerable. I have had to resolve vulnerable code added at work by several different people. All of them are under pressure and frankly do not have the mindset to think of the entire context of their changes.
And I'm coming to believe that I should expect most developers do not consider the full context of their changes. I take the time to examine and act. I synthesize everything together in my head and then finally act. I read the documentation and example code before I act. And I don't blindly copy from stack overflow.
However, features requested by enterprises on both open source and closed source projects can be terribly thought out. Who thought user injectable class files in logging code was acceptably safe?? It is up to a developer, not a project manager, to push back on saying Hey this isn't safe to do.
But it seems most developers don't have a voice or are not so socially equipped to speak? I know I'm still learning how to do this but now more in the manager space.

So what can be done to make open source software safer? Frankly the biggest players need to step up. The small ones will not. I know for certain that a company with only 10 developers will not spare a moment contributing back.

Microsoft sometimes seems to be a role model here. They have a significant presence in the open source world. Whether they are doing enough? I don't know.

But there are other big tech players out there, notably Apple.

Apple being terrible
Apple tweet
Free Apple Support: Imagine running a trillion dollar company that bundles various open source components into your products, making billions of dollars of profit annually. When one of your users reach out and ask for help, with the product you ship to your customers, you instead refer the user to the open source project. The project which is run by volunteers which you never sponsored with a cent.

Frankly I think the best way forward is for Big Tech to do what's right for them by actually investing time and resources into the assets they use, instead of shooting themselves in the foot.

Apple's Servers are vulnerable
Archived tweet

But it won't happen.

Log4j fun
Archived tweet

So what other alternatives might there be?

Maybe nation states / governments? It's clear that other nation states are willingly exploiting this.

Log4j fun
Archived tweet

So it seems reasonable that other nation states protect the interests of their citizens by putting efforts towards securing shared software.

Would this count as national defense spending?

There are proposals elsewhere to do this.

Government Support
Archived tweet

But doing tech work in the public sector is... I hear it's unfulfilling, underpaid, and that peers do not care enough about the quality of their work to produce something good. Everyone there is so sure they'll still be there tomorrow that they do not care to learn or to change anything. And that in fact, changes have to come by contracting out work because the contractors will actually do something new instead of deal with beurocracy and internal politics. This opinion comes from a developer for a state institution, not the federal institution.

So if Big Tech won't stand up, and Government employment won't produce anything of use, what is the solution?

I really do not know. It is really easy to take. It is really hard to give.
It is easier for company leadership to justify and nearly fall victim to a useless sales deal with an incompetent vendor like Sh*pe S*curity than to think about something critically and put the resources behind making, contributing, or an adapting an effective solution.
And if you do manage to convince them to hire someone, and their compensation be cheaper than the sales deal, guess what? They'll be tasked and overburdened so much that they can't do what they were hired for. What's a security engineer? Someone who sets up wordpress sites I guess.
Oh and sometimes they update dependencies in project.json, pom.xml, and whatever.gradle files. But do they ever get time to learn the codebase and address the security concerns found by an audit? No.
Instead it gets dispatched to a junior developer who is not security conscious, no security conscious senior developer (e.g. me) was consulted in reviewing the change, and ultimately that contribution has to be reverted on a friday night because it nearly took the site down.

Shared open source software can have an incredible impact when done right or wrong, log4j2 has proven that. I'm not convinced that shared open source software will receive attention proportional to its impact unless a significant culture shift occurs. This is a Tragedy of the commons problem. How do you fix that? How do you convince the people with money to allocate resources into something that will specifically be maintenance? These people have masters in business administration, they are educated to cut out costs on activities that don't produce immediate measurable value. The only exception might be for compliance requirements.

So unless there is a disaster so bad which affects life and property like that of the chemical industry, I do not see change happening.

For your privacy, this youtube video was not automatically loaded.
Click this area to load an embedded youtube video.

Log4j2 isn't enough of a disaster. The SolarWinds cyber attack was not enough.

These security vulnerabilities do not have significant lasting impacts on life and property to qualify as a disaster for a group of people to compel others to get their act together.

Seems like the status quo is here to stay. The culture is conducive to not investing in open source software despite relying upon it. Do you disagree?