Fedora package delivery security

A few days ago, PackageCloud published a blog post on "attacks against GPG signed APT repositories", and since I am involved in both the Fedora Project Infrastructure and Release Engineering teams, I figured I'd give some insight in how the Fedora Project secures our update delivery.

Note that my remarks are all about the default Fedora Project repositories as shipped with Fedora.

Signed packages

First of all, all RPM packages shipped by the Fedora Project are GPG-signed. As long as you use DNF/YUM to install these packages from a repository where gpgcheck=1 is set (which is true for all repositories in a default Fedora installation), they will fully verify the signatures on the packages. If you install a package outside of a repository (manually downloading it for example), neither will verify the signature automatically, but you can ask them to verify against the current set of imported RPM GPG signing keys with rpm -K <filename.rpm>.

As such, just signing a package prevents against that package being modified or an entirely different package being provided, but not against "freeze" attacks or malicious repository data.

Signed repositories

The Fedora Project does not sign our package repositories (yet), even though this has been requested a lot of times. One of the main reasons for this is that DNF/YUM repodata has no "ValidUntil" or sortalike attribute, and at least librepo (which DNF uses to download and verify repodata), regards expired GPG signatures as fully valid. This means that there is no way at all to mark repository data as valid for a limited amount of time.

As such, while it can protect against modification of the repository data, it does not protect agains attackers (or outdated mirrors) shipping ancient repositories with known-vulnerable software.

So, how does the Fedora Project protect our users? We use a data format called "Metalink" to provide YUM/DNF not only with a list of mirrors that they can download all our software from, but also a list of reasonably fresh repomd.xml checksums to expect. Given that repomd.xml then contain the checksums of all other referred files, this means that if a mirror is serving outdated or modified repodata, DNF/YUM will detect this, and just skip over this mirror. All our repository configurations are set to request the metalink over TLS from the Fedora Project servers.

Now, an important part here is the "reasonably fresh" part. By default, we will send the checksums for at most the last 3 days (we regenerate our repositories every day), to make sure that mirrors that are slightly outdated aren't immediatelly regarded as invalid.

However, when a significant security issue is announced and we have repositories that include fixes for this issue, we have an "Emergency" button. When we press that button, we tell our servers to immediately regard every older repomd.xml checksum as outdated.

This means that when we press this button, every mirror that does not have the very latest repository data will be regarded as outdated, so that our users get the security patches as soon as possible. This does mean that for a period of time only the master mirrors are trusted until other mirrors sync their data, but we prefer this solution over delaying getting important fixes out to our users and making them vulnerable to attackers in the meantime.

This means that as long as our users do not modify their repository configurations, they will be guaranteed (save for any issues in the code surrounding DNF/YUM) to get fresh repositories, and get critical security patches as soon as they are available.

Conclusion

With the metalink system that Fedora is using, I think we are doing a reasonable job at keeping our users secure and up-to-date. If you have any further questions about any of this, feel free to pop by on #fedora-releng on FreeNode, or ping me (puiterwijk) on FreeNode.

Infra Hackfest

Single Sign-On at Fedora Infrastructure