Handling the Kubernetes symbolic link vulnerability

Posted Dec 20, 2018 19:49 UTC (Thu) by jerojasro (guest, #98169)
In reply to: Handling the Kubernetes symbolic link vulnerability by rweikusat2
Parent article: Handling the Kubernetes symbolic link vulnerability

It might be useful, but if it is so easily misused, time and time and again, maybe it's not worth keeping.

Your approach of demanding "complete avoidance of mistakes" from application authors does not work in the real world, as recurring security issues stemming from the same "feature" show. It's way easier to fix the environment and eradicate the issue in a single place, instead of educating tons of developers and expect them to remember and never make the same mistake again...

(FWIW, I'm mostly paraphrasing/matching what I read at: https://rachelbythebay.com/w/2018/05/13/dates/ )

to post comments

Handling the Kubernetes symbolic link vulnerability

Posted Dec 20, 2018 19:59 UTC (Thu) by rweikusat2 (subscriber, #117920) [Link] (7 responses)

This is not about "misuse" of anything, the underlying problem is called "a TOCTOU race", the pattern of that being

application A verifies that condition X holds
application B changes the state of the system such that X isn't true anymore
application A acts on the presumption that X is still true

And I would indeed "demand" complete avoidance of that if I was in the position to demand anything here, as there's nothing particularly difficult about understanding that this approach is broken.

Handling the Kubernetes symbolic link vulnerability

Posted Dec 20, 2018 22:23 UTC (Thu) by jerojasro (guest, #98169) [Link] (1 responses)

you are correct, we should not let TOCTOU races slip past. Still, some of us do, for whatever reason, and it's better to just make that error possibility disappear, whenever possible, than struggle with every new person who doesn't have yet the instinct/skill to identify TOCTOU issues, wait to be bitten by yet-another instance of the same security issue, and say: "why did this happen, this is a TOCTOU race, there is nothing particularly difficult about understanding it".

Yes, we should pay attention, learn our craft, be mindful. But even so, better to, as Rachel (the author of the article I posted) says, design the potential for error out of the system.

Handling the Kubernetes symbolic link vulnerability

Posted Dec 21, 2018 21:22 UTC (Fri) by rweikusat2 (subscriber, #117920) [Link]

There's no possibility of such a thing "slipping past": The broken code has to be intentionally written in the broken way. And please note that I do mean intentionally here. More often than not, the reason why the broken code got written is probably "It'll work almost all of the time!".

Handling the Kubernetes symbolic link vulnerability

Posted Dec 21, 2018 2:52 UTC (Fri) by dvdeug (subscriber, #10998) [Link] (4 responses)

Functional programming goes so far as to make changing the state of the system impossible to avoid such problems in multithreaded programs. In an environment without memory protection, every single <b>if</b> statement offers a chance for a such a race. In an multiprocess environment, every single use of fstat offers a chance for such a race. If you want to demand complete avoidance for that, show me how you'd write du so that it accurately offers the size of a directory's contents at some snapshot in time. The only way I know how to write du in a way that would avoid that problem would be to lock the system while du is running, which is not something that an ordinary program can do.

Handling the Kubernetes symbolic link vulnerability

Posted Dec 21, 2018 23:09 UTC (Fri) by rweikusat2 (subscriber, #117920) [Link] (3 responses)

du does not suffer from this problem: It provides approximate disk usage information[*]. But it doesn't act on this information.

BTW, the actual Kubernetes problem was even sillier than a TOCTOU race as the faulty code didn't even attempt to verify that the untrustworthy input it acted upon was valid.

[*] Assuming the whole system was locked while it ran, the information would still only be approximate as it could (and likely would) change as soon as the system was again unlocked. Hence, there's no point in even trying as the result is unreliable either way.

Handling the Kubernetes symbolic link vulnerability

Posted Dec 22, 2018 6:32 UTC (Sat) by dvdeug (subscriber, #10998) [Link] (2 responses)

In the context with which we are talking, du does not provide approximate disk usage information. An attacker could get it to return almost any value by shuffling files around behind its back. At least a locked version would not allow an attacker to move a file around behind du's back.

If one doesn't act on provided information, it was useless to provide it. If one does act on it, in any way, there's a potential for a race condition. You said "And I would indeed "demand" complete avoidance of that if I was in the position to demand anything here, as there's nothing particularly difficult about understanding that this approach is broken." If the approach is broken and should be completely avoided, show me how to avoid it here, or argue that du, as well as virtually every other file tool* for any multiprocessing system, is inherently broken. Otherwise you've got to accept that complete avoidance is impractical.

* For example, rm; an attacker could potentially watch for rm to be run, move the file out of the way, and move it back after rm ran. The timing would be hard, but it's at least theoretically possible. Should rm be avoided?

Handling the Kubernetes symbolic link vulnerability

Posted Dec 27, 2018 19:41 UTC (Thu) by rweikusat2 (subscriber, #117920) [Link] (1 responses)

du does provide approximate disk usage information. It (the program) doesn't act on this information, hence, it's not susceptible to a TOCTOU-based attack. What some kind of operator might or might not do based on approximate information provided by du is a different conversation but as operators aren't usually programmed, worrying about how to program them securely is ... eh ... a bit off.
(aka "a red herring"). Dito for rm.

Handling the Kubernetes symbolic link vulnerability

Posted Dec 27, 2018 23:32 UTC (Thu) by dvdeug (subscriber, #10998) [Link]

Saying a filesystem is empty except for a few directories when it in fact had several GB of data in it before and during du's run is not "approximate".

Operators are often programs, frequently written in shell. If they use rm or du, they are subject to these types of attack. Even when operators are human, the fact that they literally can not operate these programs without getting into these race conditions is a concern if this approach is clearly broken.

Handling the Kubernetes symbolic link vulnerability

Posted Dec 20, 2018 21:50 UTC (Thu) by joncb (guest, #128491) [Link] (1 responses)

But by that logic it's even more important that we remove malloc/free as soon as possible. One of the biggest categories of security vulnerabilities is variants on buffer overruns and we've been unsuccessfully trying to get programmers to deal with them correctly for well over 30 years.

Symlinks are such an important part of operating system functionality that i'm not sure how much of linux software would still run if you removed it... including systemd.

Having said that, improving the ergonomics of certain calls to turn them into pits of success instead of pits of failure seems like a valuable endeavor but i don't entirely know what that means from a practical standpoint.

Handling the Kubernetes symbolic link vulnerability

Posted Dec 20, 2018 22:07 UTC (Thu) by sheepgoesbaaa (guest, #98005) [Link]

"pits of success" is a keeper