Linux Kernel’s Stable Branch Maintainer Greg Kroah-Hartman on Security and the Development Process

The Linux kernel is the largest collaborative software project in existence and also one of the most complex and widely deployed at the same time. It supports the work and entertainment of billions of people, hiding under the hood of Android smartphones, internet servers, research supercomputers, desktops, and laptops running Linux distributions, and more. As such, it is a crucially important project that affects a very wide spectrum of computing and technology in general.

We've reached out to Greg Kroah-Hartman, the lead maintainer of the Linux kernel's stable branch, the staging subsystem, the driver core, USB, and the sysfs subsystems, to discuss the current state of the development process, address some security concerns raised recently, and get a snapshot of the status of the project as seen from the eyes of a deeply-involved and highly-influential insider.

Although we’re taking you quite a while back, can you give us the ‘short version’ of why you were drawn into the Linux kernel development and decided to join the project?

I had used Linux at a previous job as a server for a system we had created, so I was familiar with it and how well it worked compared to other Unix-like systems. The current job I was working on consisted of me writing firmware for a USB device. This was right when Linux was first getting support for USB, and I wanted to test my firmware on all different operating systems to ensure that I had it correct, so I installed Linux on a machine, added the external USB changes, and tested my device out. In doing so, I found a few issues that could be fixed up in the Linux USB codebase and submitted a few changes that were accepted and merged into the next release.

I had a number of USB devices that did not have driver support on Linux, so I took a long weekend and wrote a driver for one of them and submitted it to the developer community. I was instantly rewarded by a long list of "you should do this better here, and that better there" and so on. It was great, other people were willing to help out and find the bugs in the code you wrote, I was instantly hooked. After that driver was merged into the kernel tree, I wrote a number of other ones over the next year or so. With the experience I got doing the Linux kernel work in my own time, it led to a job in a neighboring town doing Linux kernel work full-time, and I gladly took it and have been being paid to do kernel development ever since.

How does a major Linux kernel developer and lead maintainer handle the pressure that arises from the responsibility of making decisions on the most widely deployed piece of open-source software?

What pressure? 🙂

Really, it's all just making sure we review code properly and fix issues that are reported to us correctly and as quickly as possible. It's no different from working with any other codebase at any other company, other than we do all of our work in the open and have many more developers and users contributing, making it a very fast-moving and lively community.

What is the most challenging part of your daily job as a Linux hacker?

Learning to stop reading email late at night. It can wait until the morning. 🙂

Drivers are sometimes a pain on Linux, especially for GPUs, in the sense that almost all hardware out there is closed-source, and vendors don’t like to share much around how things work. What is the status as far as open-source drivers are concerned in 2021?

I disagree, drivers have NOT been a pain on Linux, it is one of our strongest things. It is the reason that people use Linux, as their hardware is supported on it and not on other operating systems. We have supported more different devices and platforms than any other operating system ever has for well over a decade now, if not much, much longer.

Many users of Linux have done so only because it supported the device they wanted to use. Android famously first used Linux because the platform they had was not supported by anything else, and now that is the reason there are billions of Linux users out in the world today.

That being said, the only set of drivers that matters to a user is the one for the hardware that they have, and many users are still stuck with GPUs that are not well supported by Linux or really any other operating system. I always recommend just buying a device that you know works well with Linux if you are going to want to use Linux. No one forces you to use Linux, so pick your operating system based on what you want to do and what hardware you have to support.

And hardware is always "proprietary," we do not have specs for almost any hardware and how it works. What we do have is even better, working open-source drivers for the hardware. That describes how the hardware works better than any spec ever could because oftentimes, specs are wrong or incomplete in many places.

With such a wide scope of hardware to cover, does the Linux kernel have a particular focus or direction, even short term?

Sure, same focus and direction that we have always had, “work on all hardware for all users.” 🙂

A Google engineer has recently posted a blog presenting arguments on Linux kernel security, claiming that not enough attention on upstream security is paid. The final conclusion is that the project is missing at least 100 engineers. What is your comment on these claims, and does the post raise valid arguments?

The blog post was pointing out that companies waste energy by trying to maintain kernel forks and cherry-pick random fixes to them to try to keep an up-to-date and secure kernel. That energy could be much better spent by just using the stable and long-term-supported kernels that the community provides to everyone instead, which consist of all known bug fixes at the moment. I have seen many companies make this mistake, trying to keep a kernel fork working with only minor changes to it, ignoring the hundreds of fixes that they are missing that their users and devices could benefit from both from a security standpoint, as well as a general "this resolves issues we are having" standpoint.

Android and ChromeOS are a good example of this being done well. Those devices now mandate that they be updated to the latest stable kernel constantly, as Google's security engineers have proven that the stable kernel releases fix bugs and security issues way before anyone else ever even realizes there was a problem. They have great numbers on how many issues are resolved and what was resolved when. They know that having up-to-date systems is a requirement for having a secure system.

And yes, we can always use more engineers, we will never turn that down. The huge majority of contributions to the kernel are on new features, as companies want to sell new hardware or have new usage models that they want to see Linux support, and that's great. But also, we need constant maintenance and code review happening as well, so if companies could allow their developers to help out with that effort, that would make everyone's lives easier over time, as well as making the kernel much more secure.

In your opinion, is the current model of development for the Linux kernel future-proof, considering factors like the increasing scope of deployment, the rising requirements in quality and security, and also the ever-increasing size and complexity of the Linux kernel itself.

We always constantly evaluate how our development model is working and what needs to be changed where. It is different today than it was 5 years ago, and vastly different from what it was 20 years ago, and it will be different 5 years from now, and probably much different 20 years from now. Constant evolution is the key to keeping this project working and running well, so we will just have to see what needs to be done in the future to ensure it remains a viable and successful project.

Can tech giants that contribute to the Linux kernel or support the Linux Foundation financially steer things in the direction they wish, and up to what point?

The Linux Foundation is there to help with the legal and marketing support of the Linux kernel, it does not steer or affect how kernel development works at all, except as asked for by the Linux kernel developer community. The kernel community elects a group of their developers to be on the LF Technical Advisory board (TAB), and the chair of the board sits on the board of the Linux Foundation itself. The TAB is there to help with issues that come up in the community, as well as help address issues that companies might have with Linux, helping to be the "bridge" between the two groups when needed.

One example in the past of this was when the TAB helped educate the LF member companies about the issue that Secure Boot was going to cause when it was implemented in UEFI. That effort allowed companies to help steer the UEFI standards such that Linux and all other open source operating systems would be allowed to work properly with Secure Boot.

To answer the other part of your question, yes, "tech giants" can contribute to the Linux kernel, just like any other company does and should do. Everyone contributes to Linux in a "selfish" manner, they want to solve a specific problem that they have, and they know that by working with everyone else, their problem can be fixed better than it would have if they went and tried to do it on their own. It is in their selfish best interest to work with the community to help create the best solution for their problem. It just turns out that almost everyone has the same "problems," so that when one company helps solve an issue, everyone benefits.

Examples of this would be the initial multi-process support added to the kernel. Only companies dealing with "big systems" cared about that at the end of the 1990s, but now everyone has an 8 processor ARM machine in their pocket with more power than those big systems had. Linux works well because the companies at the time worked to solve their specific problem in the community, and in the end, it was good for everyone.

The license of the kernel, GPL v2, puts everyone on a level playing field because if a company contributes code under it, everyone has to publish any changes that they make and distribute based on it so that everyone can see the changes and accept them if they turn out to be good ones.

So yes, "tech giants" contribute, and they are on the same playing field as the "tiny" companies, all of which benefit from working together. And those "giant" companies once were not giant, but the fact that they used Linux helped get them there.

Linus Torvalds, the leading developer of the Linux kernel, has previously expressed his rejection towards treating security bugs any special, or with more urgency than regular, functionality flaws. Has this changed today, considering the raised needs in cybersecurity and the ever-increasing amount in damages victims suffer as a result of breaches?

No, nothing has changed with regards to treating them "special," but it doesn’t mean we’re not prioritizing security or that we don’t take it seriously. On the contrary, we work hard to resolve all the reports we receive on a weekly basis.

To get a better idea on how exactly we treat security bugs, I would suggest reading the ‘Security’ section of this webpage.

If anyone’s looking for more details about how security reports are handled or instructions on how to submit one, they should check out this resource.

What is your message to young coders interested in learning more about how they can contribute to the Linux kernel?

Look at kernelnewbies.org and take a look at the basic "how to write a first patch" tutorial there, as well as thinking about why you would want to contribute to the kernel and what you want to do there. We have a lot of "beginner" tasks to be done and always welcome new people to help us out.