Insecurity and Python pickles

Posted Mar 12, 2024 19:38 UTC (Tue) by atnot (guest, #124910)
In reply to: Insecurity and Python pickles by pwfxq
Parent article: Insecurity and Python pickles

My lesson would rather be that if you provide something useful in a way that can not be used safely, it will inevitably expand in usage until it becomes a problem.

Pickle is used a lot by machine learning folks because it's an easy way to checkpoint long-running jobs. I've used it for that myself. You *could* hook up custom json serialization or something, but it's a pretty huge pain to do in python. And remember most of these people are researchers first and programmers second. And yes, sure, the pickles might be insecure, but they're you're pulling down megabytes of python code you don't understand to actually run the model anyway, so does it really matter?

And so, lacking better alternatives, the usage expands into new use cases until suddenly the theoretical issue becomes a practical one.

See also: PyYaml, which had all of these convenient functions for writing python inline in your local configuration files. Until people started using it for data interchange. Or the naive file format parsers of a nice convenient tool to resize your images. That then accidentally became the standard library people hooked up to their php sites.

to post comments

Insecurity and Python pickles

Posted Mar 12, 2024 21:59 UTC (Tue) by Karellen (subscriber, #67644) [Link] (8 responses)

And yes, sure, the pickles might be insecure, but they're you're pulling down megabytes of python code you don't understand to actually run the model anyway, so does it really matter?

The thing is, a lot of non-computer people understand the difference between "code" and "data". They might understand it with varying levels of sophistication, but they often understand that running random programs from arbitrary websites can be inherently dangerous, in a way that viewing random pictures, or random videos, or listening to random mp3s, or reading random web pages, "shouldn't" be (genuine bugs aside).

Sure, they don't understand the code that runs a model, in the same way they don't understand the code that makes up a media player. But if they trust the entity that wrote the code, they don't need to. And they probably can (and should) trust Huggingface, or Microsoft.

And with that trusted code, they should be able to try out data files from anywhere, with relative safety, right? Right?

Because, as we've known from auto-executing macros in Office documents for over a quarter of a century now, making data executable is a bad idea. So no-one would be stupid enough to make that mistake again, in the mid-2020s, surely? You, as a regular user who's not a software dev, shouldn't have to check if data is safe to load, should you?

Insecurity and Python pickles

Posted Mar 13, 2024 17:43 UTC (Wed) by NYKevin (subscriber, #129325) [Link]

I'm not sure it's that easy. Most people know the difference between a .exe and all other file formats (assuming a Windows context, because let's face it, that's where the users are). They may or may not be able to recognize any of the following as potentially dangerous:

*.dll (dynamic linked library, analogous to *.so on Linux) - Windows automatically loads DLLs in the same directory as an executable before searching other directories, on the theory that C:\Program Files\ is supposed to be root-owned and you're not supposed to have random *.exe files lying around your homedir.
*.reg (Windows Registry entries, exported as files) - to be fair, Windows does display a warning message on these when you double-click, but if clicked through, it will import them back into your registry.
*.vbs (VBScript, a scripting language supported by ~all versions of Windows) and various other extensions associated with VBScript.
*.bat (MS-DOS batch files, another scripting language supported by ~all versions of Windows)
*.ps1 (PowerShell script, supported by modern versions of Windows) - to be fair, Windows defaults to the "edit" verb on double-click, and unsigned scripts won't run unless a system-wide setting is adjusted.
*.hlp (WinHelp file, supported by everything older than Windows 10) - deprecated primarily because it is hideously insecure, but when I Google it, different websites seem to identify different specific problems (macros, embedded DLLs, possibly others?).

And probably other formats that I don't know about.

Insecurity and Python pickles

Posted Mar 14, 2024 15:41 UTC (Thu) by smoogen (subscriber, #97) [Link] (6 responses)

We have been making this mistake for a lot longer than Office macros, and we will keep making the mistake because people's brains are wired to consciously and subconsciously to choose convenience. Think about how everyone is supposed to drive 100% of the time following all the rules exactly.. and think about the number of times something happens where you drive over a limit, forget to do a turn signal, don't check your mirrors every 5-10 seconds, etc. We all know we should be doing these all the time, but there is some part in most people's brains to say "Oh I can get away with it for a short bit." and 99.999% of the time, that part is right. The same goes with choosing formats which are used to build things.
You start off with a 'this is a fast checkpoint method' and then at the next deadline/problem cycle you have 'oh lets just take that checkpoint and see if we can restart this' to 'oh we can send you the checkpoint' to 'everyone is sending everyone pickles so its a standard.'

In the end, this is where standards get forced into being. People blow up enough towns around factories and eventually someone makes an ISO about valve safety.

Insecurity and Python pickles

Posted Mar 14, 2024 16:30 UTC (Thu) by farnz (subscriber, #17727) [Link]

You start off with a 'this is a fast checkpoint method' and then at the next deadline/problem cycle you have 'oh lets just take that checkpoint and see if we can restart this' to 'oh we can send you the checkpoint' to 'everyone is sending everyone pickles so its a standard.'

This sort of creep in scope is also problematic by itself, even without "choosing convenience", because you change the context around.

The first developer, writing the "fast checkpoint" method, probably had a context of "and anyone who can tamper with the checkpoint can attach a debugger to the running code and replace it".

The second set were aware of that context, but didn't see it as a problem if you're using it as a suspend work + resume later mechanism - after all, if you could suspend work and resume it later on the same system, you can attach the debugger instead of suspending, changing the checkpoint, and resuming.

And the third set, who send checkpoints, aren't aware that the first developer made their choices in the context of "checkpoints are equivalent in threat to a debugger attached to the process loading them, and that's OK", so they simply adopt the existing checkpoint format, because surely it's secure enough, right?

Insecurity and Python pickles

Posted Mar 14, 2024 17:18 UTC (Thu) by adobriyan (subscriber, #30858) [Link] (4 responses)

This is easily fixable by casting spell called "Criminal Negligence" readily available in most countries.

Whoever enables eval() equivalent by default goes to prison.

All it would take is 1 sacrificial cow.

I'd suggest Microsoft PM who authorized Outlook executing attachments by default which clearly caused so much damage in 90s-2000s.
But status of limitations may have expired on the man.

Developers would probably revolt (they won't), both proprietary and open source.
It is important to absorb all the feedback from the constituents but ignore all their whining in the end
which politicians can be very good at.

Insecurity and Python pickles

Posted Mar 14, 2024 18:28 UTC (Thu) by pizza (subscriber, #46) [Link] (3 responses)

> This is easily fixable by casting spell called "Criminal Negligence" readily available in most countries.

Except that criminal laws are not retroactive.

> Whoever enables eval() equivalent by default goes to prison.

Ok, so users have to change the default setting to achieve common legitimate use cases.

Or they're prompted "do <potentially dangerous thing> Y/N?" so often that they automatically say "Yes" without thinking about it any more.

....That which makes computers useful also makes them dangerous. And the definition of each varies on an individual and/or situational basis.

Insecurity and Python pickles

Posted Mar 14, 2024 19:01 UTC (Thu) by adobriyan (subscriber, #30858) [Link] (2 responses)

> > This is easily fixable by casting spell called "Criminal Negligence" readily available in most countries.
> Except that criminal laws are not retroactive.

I'm not proposing new law. I believe all the laws are in place already, it is just that governments chose to not exercise them.

I remember Nimda pandemic while at the university. Us, Linux users were laughing at Windows suckers.
But our machine was dual booting so we were laughing at ourselves too.

It is unthinkable how Microsoft was not crucified for those stunts. It was so easy politically.

> > Whoever enables eval() equivalent by default goes to prison.
> Ok, so users have to change the default setting to achieve common legitimate use cases.

Yes, and then it is not on the manufacturer.

> Or they're prompted "do <potentially dangerous thing> Y/N?" so often that they automatically say "Yes" without thinking about it any more.

Police officers don't even get prompted by their gun's safety lock, they just disable it and shoot the criminal if necessary.
Somehow, the society lives with it and this situation is considered OK by general public, gun manufacturers, police and soldiers.
Nobody is saying "hey, police officer would have override safety lock so many times in his career that they would do it without thinking".
Maybe it is time to stop saying that when talking about software.

Insecurity and Python pickles

Posted Mar 14, 2024 19:28 UTC (Thu) by pizza (subscriber, #46) [Link]

> Police officers don't even get prompted by their gun's safety lock, they just disable it and shoot the criminal if necessary.

So? Police weaponry is specifically designed to kill or otherwise incapacitate; they have no other legitimate uses. [1]

Whereas these "dangerous" software features are used as intended by untold millions of office drones every single day.

[1] They are also used to project the implicit and explicit threat of potentially lethal force should you not submit to their authority.

Insecurity and Python pickles

Posted Mar 14, 2024 20:34 UTC (Thu) by dvdeug (subscriber, #10998) [Link]

> Maybe it is time to stop saying that when talking about software.

Do you want actual security, or are you just trying to shift the blame?

Everyone of us has run into an interface where it demands "are you sure you want to do this?" and you hit yes without thinking because you've been asked that question over and over when you're trying to do exactly that. There is some user blame where the user understands the question (e.g. "are you sure you want to delete this file?"). However, "do I want this spreadsheet to work?" -- when 99% don't have a clue what that means and the remaining 1% _could_ spend the next hour doing a forensic examination of the guts of the spreadsheet but aren't going to waste their time without reason -- is just blame-shifting.

Even a question of "this spreadsheet is trying to write to an external file. This is a warning sign of a malicious spreadsheet; do you want it to continue?" is going to auto-yessed by 50% of the users, and it's going to be a pain using the office spreadsheet that does do that for non-malicious purposes, because 15% of the users are going to auto-no that, even when they're prewarned. That's not completely security theater, but the fail cases on both sides makes it pretty ineffective.