Pushing kernels more aggressively to updates-testing
On thing struck me tonight about the recent fiasco relating to the stable marking of a kernel that just happened to also kill wifi for a great number of users. We did the correct thing, to a degree naturally, the update was in relation to a security update something Fedora takes very seriously. As such our users should always feel safe knowing that we will push such updates fast, keeping their systems secure through multiple means including proactive security and rapid updates.
However the problem is that we don’t apply the update to the existing stable kernel, the patch is always applied on top of the progressing kernel, meaning we also end up shipping a lot of other things such as bugfixes, updates to the latest upstream STABLE tree and various other things. This however is confronted with one problem, the kernels in between the current stable and next update are not all being pushed to updates-testing – only selected kernel updates are. In cases where we then have to release a security fix we are forced to ship a bunch of stuff additionally which is not likely to have been tested extensively.
It occures to me that catching these bugs before they become a problem for average users could be accompliced by making better use of updates-testing, testers are normally willing to experience a degree of breakage and are qualified to file bugs for the most part. Then at least when an urgent update is required we will not likely be surprised by massive unrelated breakage – it might still occure but we can warn people if avoiding massive breakage is impossible and reverting the offending patches is impossible prior to release.
An additional problem caused by this is that when an urgent release contains bugs we will be urged to ship another update straight afterwards. Opening us to even more bugs from another untested delta (since other development is likely to have gone on along side the bugfix) and having our users suck down a second kernel package shortly after the original update.
The other option would be applying the security update to the current stable kernel and not carrying the current delta in the update, but this is expensive in terms of man power and time, it also goes counter to the rapidly developing nature of Fedora in general. This is the realm of the enterprise distros, if people want this approach something like RHEL/CentOS is likely a better fit for them.