The OpenWrt upgrade process

How to upgrade OpenWrt?

An upgrade of OpenWrt can be done in many ways - Upgrade with LuCi, Upgrade with sysupgrade or Upgrade OpenWrt with the new Attended SysUpgrade functionality asu through the OpenWrt Firmware picker or using the OpenWrt Attended Upgrade Client, acu. We will talk about the OpenWrt Imagebuilder in a following episode.

Watch the video on YouTube

Click to view the entire transcript

(Why make a video about upgrade?) Why would I make a video about upgrading OpenWrt? I mean, it’s simple right? You log into the OpenWrt web interface which is called LuCi, you select “System” in the menu, then “Backup/Flash Firmware”, you scroll all the way down to this button, click on it and select the new firmware image. Done. Yes – that’s true. Upgrading OpenWrt is quite simple and can be done in a few clicks. However, there are some things that might come as a surprise especially to new OpenWrt users. First – all software packages that you have installed since the last time you upgraded your OpenWrt device are gone. Why? That’s because of the way the OpenWrt upgrade process is designed. I’ll explain that in the 3rd and 4th chapter. And guys – there’s a great solution provided by the OpenWrt team called “Attended System Upgrade” that I show in the last chapter. So – you might say “Well, I am happy with my OpenWrt version – I won’t upgrade at all”. Let me tell you – this is not necessarily a good idea. Why? (Why upgrade?) The main reason for upgrading or updating OpenWrt is the same like for any other device - security. I’ll talk about the differences between upgrade and update in a minute. Like any other software, OpenWrt has bugs. Again – like any other software. Your router’s stock firmware also has many of them. Trust me. Those bugs may lead to vulnerabilities. That means that they can be exploited in order to do bad things. Such as a malicious attacker could use them to gain access to your network. You can find a list of “Common Vulnerabilities and Exposures” – the so called CVEs on the web site cvedetails.com. What’s concerning here, is the number of high severity vulnerabilities – roughly 30% of the entries for various software products are levels 7 and above. This alone is reason enough to update or upgrade. There might be a couple of other reasons why you would want to upgrade, such as added hardware support for a device you’re using or additional features that you would want, such as Bridge VLAN filtering that has been added in OpenWrt version 21 or nftables firewall which has been added in OpenWrt 22. (Upgrade process at a glance) So – what does the OpenWrt upgrade do actually and what’s the difference between upgrade and update? OpenWrt is a linux distribution really and has a built in package management system. That means that you can install additional software on it. If you go to the “System” tab in LuCi and then Software, you can see a list of packages that are available for OpenWrt. If the list is empty, click on “Update List”. On this “Installed” tab, you can see the packages that are currently installed on your OpenWrt router. Now for updates. This “Updates” tab here shows you packages where an updated version is available. As long as you use a supported version of OpenWrt, that means the last two major versions or so, then you will get updates here frequently and clicking on that “Upgrade” button would install the newer version of the OpenWrt software package. Now what’s the difference between this process and the system upgrade? The system upgrade which you do in System-Backup/Flash firmware basically installs a whole new image. Including the initial set of software packages that come with that version of OpenWrt which you are flashing. So it’s more kind of a re-imaging rather than a guided upgrade really. You can however keep your configuration. In a kind of best effort approach. As I said, additional software packages will be lost here. I’ll talk about this – there are great and easy ways to address this. But why is that Upgrade process designed that way? (why is the Upgrade process designed that way?) You might argue – wouldn’t it be better to have a “rolling update” system in OpenWrt like we do have it on other Linux distributions? So – rather than re-image the device every now and then – just launch opkg upgrade and replace all changed OpenWrt packages with new ones? During my research for this episode I came across this article here where someone does this. I reproduced it and it does work. But guys, please don’t do it that way. There is no guarantee that you won’t end up with a bricked router or a bunch of old binearies here. NOT RECOMMENDED. The reality is – I am sure that the OpenWrt developers would love to have a rolling update. Just – keep in mind that OpenWrt supports more than 1,500 devices! Unlike other router operating systems which support a handful of architectures, that’s really vast! And a huge part of those supported devices are consumer devices. That means – low price. Cheap devices. Let’s face it. And one of the things that would make a device expensive is memory. RAM as well as flash. As a consequence, these devices do not have a lot of memory. And that means that if we would just let’s say create a writable file system such as ext4 on the Router’s flash memory, then the firmware would just not fit on the device – it would be too big. Therefore, these small consumer routers are provisioned with a squashfs file system. That’s a compressed file system that consumes much much less space. But this comes at a cost. The sqashfs file system is read only. So – in other words – you can’t change anything. Or can you? Let’s answer that question right after this call to action! I really need you to get involved please! First off – I would kindly ask you for a “Like” on YouTube if you like the video so far and if you like my OpenWrt videos. Please do leave me a comment – I’ll answer everyone!!!! If you want to become a Patron on Patreon - you’ll hold a very special place in my heart if you do. But most importantly - If you like a video, then please copy the link of the video and post about it in forums. I can’t do that. It’s considered to be self promotion on Reddit or Facebook and the like if I do. But you can. Please do talk about my videos and about my channel to others. That would be really amazing – Also, join the Discord server – the invitation link is in the description – if you join the server then you can talk to me and the folks there every Sunday 9 AM and 6 PM Berlin / Paris time. Thank you so much! Back to the OpenWrt upgrade. I’m on Twitter as well ;-) Now – you will ask – how about my config files and installed software? I can change the OpenWrt config – the router must have read/write flash somewhere? Yes – it has. Let’s look at a simplified typical block device architecture of a consumer device. There is one block that is reserved for the boot loader. Usually that’s u-boot. That boot loader then loads the OpenWrt squashfs from another read-only partition. The way OpenWrt stores additional packages and config files is by making another partition read- and writeable. And that partition is mounted as a so called Overlay file system. You can see this if you type “mount” in a shell. Here on my Belkin RT3200 router I have the read only squashfs on /rom and the overlay file system on the ubi0_5 device. The root file system is of type overlay. All initial files can be found in /rom and the changed files in /overlay/upper. What I see is the rom with the read-write overlay file system masking it from above. Think of it like a pane of glas on top of the rom file system. I always look through the upper overlay file system. If a file does not exist there but in the rom, then I see the file from the rom. If I change or add a file, then it will sit in the upper overlay file system. To me as a user it is transparent. The operating system keeps track of all this. The implication of all this is that if we keep installing or updating packages, then the overlay file system, so the read-write partition, would gradually fill up. Also – deleting something does not free up any space if it is in the rom. In a nutshell – the upgrade procedure is done the way it is out of necessity and limitations rather than anything else. Architectures with writable file systems, such as x86 with ext4 file systems for example are an exception. You could do something like a rolling upgrade on those. But first let’s re-iterate what is lost and what is kept between upgrades? (What is lost and what is kept?) We know that the installed packages are lost between upgrades but the config of the OpenWrt router can be kept. In order to get a better understanding for this, let us do an upgrade on this OpenWrt virtual machine but let’s not use LuCi but rather the command line. In order to upgrade on the command line, we first need to download the image to the ram which is mounted in /tmp. So I do a cd into /tmp. Upgrading x86 is kind of a special case because there is no dedicated sysupgrade image. You just use the same image for installation and upgrade. So I go to the OpenWrt firmware selector, I select Generic x86/64 as target and then – rather than downloading the image I just copy the image url from the “Combined (Ext4)” button. Back to the terminal. Here I type wget and then paste the url. I will now download the image to the /tmp directory. The x86 image is a gzip file which needs to be unzipped first. Gunzip and then the name of the image does the trick. Now I can launch the upgrade by typing sysupgrade, then -i for interactive and -v for more verbose output and then the name of the image. Here we can nicely see what it does. It asks me if I want to keep the config. That defaults to yes. So enter. Do I want to edit the config file list? Normally you wouldn’t. But just so we see what it keeps let’s do that. And here you can see what will actually be preserved between OpenWrt upgrades. The list contains the dhcp and network config, the firewall and potentially wifi config as well as profiles, users and keys that had been generated. Once I finish editing the list, it then goes off, flashes the image and reboots the router. After a short moment, I can then log back into luci and check if everything was successful. Network, DNS, firewall – that should all still be there. So how could we now keep software packages between upgrades? (How to keep software packages) The OpenWrt software developer would tell you – just make a custom build of OpenWrt and include all the packages you need. In other words – compile it from the source code. Well yes – we could do that – but I would assume that most people do not fancy doing that. There are better alternatives. Let’s first look at the opkg package management system. If you want to go straight to the ultimate solution, skip that chapter. (How opkg keeps track of packages) Luckily, opkg keeps track of the software packages that have been installed on the system. You can find that list and additional info in the /usr/lib/opkg subdirectory. There is a status file that contains an overview of all the installed packages. It gives you the package name, version, dependencies on other packages, which config files it uses, and when it has been installed. Pretty much the same you get when you type opkg info or opkg list-installed. You remember that I had spoken about the overlay file system? Now – this vm here has an ext4 file system, so there is no overlay. But on my Belkin Router I have the original file list in /rom/usr/lib/opkg and the modified list in the overlay file system. And even better in /overlay/upper/usr/lib/opkg/info I have files for each package that I had installed after the upgrade. So I could leverage that information to actually get a list of packages that I need to install after an upgrade in order to get my system back into the state it was in before. But how would I go about that on my x86 system where I don’t have an overlay file system? Well, there is still info in the status file that I can use. The installation time stamp. If I would be looking only at the first let’s say four or five digits then I could nicely group the packages by installation time and hence figure out which ones had come with the build and which ones I installed later. But I get even more information out of those files. Packages depend on each others. That means that in order to install some package on OpenWrt, other packages need to be present. When I install the top level package, then opkg will automatically install all dependencies. So what I would be after are all packages which are at the top of the dependency tree and which are either in the list in the overlay file system or which have a time stamp newer than the build time of the image. So that should be quite easy to do with a script, right? All that I’d have to do is find the top level packages and launch opkg to install them. I had started writing a script that does all this – it’s unfinished work – you can still find it on my github repository. Why did I not finish it? Because there is a much better solution to the problem. I run live video sessions open to everyone on my discord server twice every Sunday where people can join and ask questions, talk about technology or just listen in. The invitation link is in the description. In one of the last sessions I spoke about that script and then one of the admins said “Marc, you can download a custom built OpenWrt image from the firmware selector” and the other admin showed us his solution to the problem which is called attended sysupgrade. So first off – many thanks to chaot and thinker on the discord server for bringing me up to speed – I was totally unaware of that. Here’s how that works: (The OpenWrt Firmware picker) Usually when I want to flash a device with OpenWrt, I go to the OpenWrt.org web site and use the firmware selector to chose my device and the platform and then download the right image. What I had totally overseen is that little “Customize installed packages” option here. Let me click on it. A text field opens that contains OpenWrt package names. Wait – could we customize that? Yes we can! I can just type in additional package names like for example packages that my Wi-fi mesh points need in order to come back on line after an upgrade such as wpad-mesh or batman or the like. Or I could type in python3, sudo and the like if I wanted to manage that router with ansible. Or – if my router would not have that much memory or if it was a dumb access point then I can actually remove packages from the build and save some memory. When I now click on “Request build” then the unbelievable magic happens. My request goes off to a build system that builds my personalized image and delivers it to me. No need to compile from source if I want to have a personalized build. It takes a minute or two and I can then download the custom image and upgrade my OpenWrt device with it. It will contain all software packages that I have specified. It will still be the official OpenWrt version, so everything around installing or removing software from the official feeds will still work. Awesome. A big “Thank you” to the OpenWrt team for having put this in place. This is automation at its best. (Attended Sysupgrade) Still – if you use the firmware selector, then you would need to know which OpenWrt packages have been installed on your system. But I told you that the opkg system keeps track of it – so can this be leveraged in order to take the upgrade process to an even more amazing next level? Again – yes, it can. There are a bunch of small software packages that you can install on your OpenWrt router in order to make this custom image upgrade even more transparent. Let’s install them. I go to the System-Software menu in LuCI and search for auc – that’s a command line interface client for attended-sysupgrade. There is also a package available with a nice LuCI interface – it’s called luci-app-attendedsysupgrade. Let me install this as well and refresh the page in my browser. Now I have an additional menu up here under System that reads Attended Sysuprade. If I click on it, then I have this new button “Search for firmware upgrade” here. If I click on that again, then my OpenWrt router goes off to the OpenWrt.org build system and searches for new packages. And here it will deliver a custom built image to me that contains all updated versions of the packages that I had on my system. And – I can install that straight away from the menu. Takes a minute, my router comes back and has everything upgraded – the OpenWrt image, the software packages – everything. Totally transparent. Awesome. On the configuration tab I can set more options. Such as the server where it submits the build request to. If I tick the “Advanced Mode” box here, then Sysupgrade lets me chose the packages to include and additional versions such as release candidates or snapshots before it submits the request. The same thing can be done on the command line, so you don’t have to use LuCI in order to have that functionality. The command is called auc. If I just type auc, then it identifies the running version, it shows me the best available version and also I get a list of installed packages that have an upgrade available. Here again I can just launch the process, takes a while, a custom image is delivered to me and I can upgrade on the fly. Incredible. How does that work behind the scenes? There is a repository called asu on the OpenWrt github that contains the project. Here you can find URLs of available servers as well as instructions on how to build your own build server. Maybe we will do that in one of the next episodes. It should be really simple as the OpenWrt team have included a docker-compose file that lets us build our own sysupgrade server. Just before we close – I was curious to see what type of hardware people are actually requesting builds for. And if I look at the stats of the servers for the last couple of days, it turns out that the Belkin RT3200 and its Linksys labeled counterpart, the Linksys e8450, seem to be everybody’s darling at the moment. Followed closely by the usual suspects such as the Archer C7, the Xiaomi 4A gigabit edition, as well as a big chunk of Fritz boxes here – quite unexpected - and an increasing amount of requests for x86 hardware and the Raspberry Pi4. That’s it for today guys, hope you liked the episode. Thank you so much for watching. Stay safe, stay healthy, bye for now.