How to (not) fix a security flaw

A pair of flaws in the web interface for two small-business Cisco routers make for a prime example of the wrong way to go about security fixes. These kinds of flaws are, sadly, fairly common, but the comedy of errors that resulted here is, thankfully, rather rare. Among other things, it shows that vendors may wish to await a real fix rather than to release a small, ineffective band-aid to try to close a gaping hole.

RedTeam Pentesting GmbH found the flaws in September 2018 and notified Cisco shortly thereafter. The original disclosure date was planned for January 9, but that was postponed until January 23 at Cisco's request. On the latter date, Cisco issued advisories for CVE-2019-1652 and CVE-2019-1653; RedTeam Pentesting released its own advisories, with lots more detail, for CVE-2019-1652 and CVE-2019-1653.

The flaws are bog standard web-application vulnerabilities. CVE-2019-1652 is a command injection that allows authenticated users (in the web interface) to execute arbitrary Linux commands as root. CVE-2019-1653 allows anyone to request the configuration page from the router, which contains all sorts of interesting information, including user names with hashed passwords, VPN and IPsec secrets, and more. In addition, password hashes are all that is needed to log into the web interface—no cracking required.

Beyond that, an additional information disclosure flaw, related to CVE-2019-1653, was reported; it uses a debug interface to retrieve a .tgz (gzipped tar) file, encrypted with a known, hard-coded password, from the device. That file contained even more configuration and debugging information as well as etc.tgz and var.tgz with the contents of those directories from the router. In all of the RedTeam Pentesting advisories, curl was used for the proof of concept, though there are lots of other ways to perform the same actions, of course.

The flaws were found, reported, and fixed; so far, so good—or so it would seem. But on February 7, RedTeam Pentesting found that the fixes shipped by Cisco were, at a minimum, insufficient. Once again, the problems were reported to Cisco, with a disclosure date of March 27. Despite a last-minute request to postpone the disclosure, three new advisories (command injection, configuration information disclosure, and even more configuration information disclosure) were released by RedTeam Pentesting on March 27.

It turns out that in the four months between the report and the "fix", Cisco decided that simply blocking curl was sufficient. In addition, the blocking was done based on the HTTP User-Agent header that curl sends. It can hardly have escaped the notice of someone (or, likely, multiple someones) that curl -A (or --user-agent) will helpfully change that header to anything the user (or attacker) wants. There are a lot of options to curl, which can be seen in its man page, but User-Agent strings are nothing magic—and curl is hardly the only way to perform an HTTP GET or POST.

Another two months on, nearly six months after the original disclosure, Cisco still does not have a real fix for the problems. Not enabling internet (WAN) access to the web interface (which is the default state), or disabling it if it has been enabled, is a workaround for the flaws, but obviously impedes the remote management feature that some customers may have been relying on.

The original disclosures in January set off a flurry of exploit attempts. By chaining the two vulnerabilities together, root access is easily available to any attackers. Some detailed proof-of-concept exploit code is available—and has been since the disclosures. It does not use curl, so it presumably always worked, even on updated routers. There are some 20,000 affected routers that can be found using the Shodan tool and roughly half of them were exploitable at the end of January; most of those were still exploitable at the end of March.

Given how close the second set of disclosures was to April 1, and the almost comical nature of the "fix", one might be forgiven for thinking it is all some elaborate hoax. Of course, the comedic effect is much diminished for those who have the RV320 and RV325 routers installed. At least from the outside, the problems do not seem so difficult to solve that it would make sense to try to slip in a broken hackaround instead of making a real fix. One suspects that Cisco has de-prioritized those router models, so there is no one to work on them. But did anyone with a technical clue at Cisco really think this kind of thing would fly underneath the radar?

One can only imagine how quickly a fix would have been mooted had the web interface in question been open source. There is probably not much in the way of proprietary "secret sauce" in the web interface, but an open-source release might be problematic for other reasons; Cisco would also have to provide ways for users to upload changes to the router, which may have its own set of challenges.

One amusing outcome is a suggestion from Florian Obser to apply the Cisco "fix" to the OpenBSD HTTP server. His "httpd(8): Adapt to industry wide current best security practices" proposal is unlikely to be acted upon, however.

This episode is pretty much a textbook example of the perils of being at the mercy of a vendor when a security flaw is found. It is also yet another example of a device that is meant to be on the internet, but has apparently had little or no thought to security baked in. Debugging interfaces are useful, for debugging; they should be removed from shipping products. Any vendor shipping a web interface should either internally do penetration tests (pentests) on it, hire that job out, or both. It is rather amazing to see this kind of flaw—and response—from a major vendor in 2019—but there are surely others out there that we will hear about in due course.