PyTorch and the PyPI supply chain

Posted Jan 12, 2023 6:55 UTC (Thu) by bof (subscriber, #110741)
Parent article: PyTorch and the PyPI supply chain

What I do not understand, from a sysadmin perspective used to the distro model, is this:

All these language package repo things, run wide open to everyone uploading stuff. With the obvious downsides. So why isn't there trusted language "distros" with trusted groups of maintainers curating that into trustable, separate repos meant for the "consumers" out there? And why the frell does everybody consuming the packaging, accept that as God given (adding in a snide remark about the Dino distros)?

to post comments

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 7:12 UTC (Thu) by maniax (subscriber, #4509) [Link] (3 responses)

I have pretty much the same question... In my case, anything that's not maintained by the Linux distribution (which I tend to trust) is installed from external sources only if really really really needed, and mostly set to a very specific version with the idea that as soon as it reaches the distro, it'll be updated. OR, a copy is maintained internally in separate, internal repositories.

And this is not only a security question. Stuff "out there" is usually too bleeding edge to be reliable enough, and just fetching "the latest and greatest" is bound to break stuff.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 8:28 UTC (Thu) by taladar (subscriber, #68407) [Link] (2 responses)

But stuff in the distros is just as bleeding edge, just bleeding edge backports that a person who is comparatively much less familiar with the code base developed and virtually nobody tested, usually with a lying version number on top of it.

Stability in a changing world is an illusion or in many cases even a deception sold to the gullible companies who desire it but don't understand how fast the world really moves in terms of software compatibility with the rest of the world (both in terms of protocols, data formats,... and in terms of legal and regulatory frameworks,...) and security issues.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 8:58 UTC (Thu) by ms (subscriber, #41272) [Link]

Exactly this. I choose to use NixOS, both at home, at work, and on some of my servers. I have zero belief anyone in that project is reviewing upstream code changes. The prevailing attitude is very much "if it compiles, that'll do". Tbqh, I wouldn't be at all surprised if that's pretty much the same right across most distros, with the exception of some of the bigger commercial distros. And even then, I fully expect focus would be on the most critical packages - the kernel, libc, security libraries, xorg/wayland, mutt... - for obvious economic reasons.

I think everything really does just boil down to "you just have to trust other people". Yep, checksums, and version numbers, and all that goodness is great for verifying things don't change that you don't want to. I wouldn't want to be without that. But when I'm looking for a library to solve a particular problem, I look at the number of stars and forks, the rate of commits and who they're from, and the issue tracker, and that's my starting point for establishing trust. And I think it's a good thing: a society where the default behaviour is not to trust, not to give the benefit of the doubt, not to assume good, is not worth having.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 10:12 UTC (Thu) by ballombe (subscriber, #9523) [Link]

> But stuff in the distros is just as bleeding edge, just bleeding edge backports that a person who is comparatively much less familiar with the code base developed and virtually nobody tested, usually with a lying version number on top of it.

The code does not run in a vacuum. Distributions are much more familiar with the environment where the code will run,
and most distribution developers are also part of upstream. they also tend to have more user-aligned view than upstream. user-hostile upstream do exist.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 8:22 UTC (Thu) by LtWorf (subscriber, #124958) [Link]

Well I guess because nobody is doing this job, and because developers always want to use today's new thing, so they think that waiting for vetting is a waste of time.

And since it generally doesn't end up in malware being downloaded, the current system is good enough… until the next malware happens.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 12:45 UTC (Thu) by khim (subscriber, #9252) [Link]

> So why isn't there trusted language "distros" with trusted groups of maintainers curating that into trustable, separate repos meant for the "consumers" out there?

Because there are no “consumers”?

Developers want two things which can not be, obviously, satisfied simultaneouly:

They want to be able to quickly get updates and bugfixes.
They want to be able to be sure there are malicious code.

Distributions solve problem #2 well but entirely fail to handle #1. Language repos and AppStores solve #1 well, but suck at #2.

Since half a loaf is better than no loaf developers stick to what solves one problem and can half-ass the 2nd one rather than use something that fails entirely to solve half of the problem.

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 13:24 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

> So why isn't there trusted language "distros" with trusted groups of maintainers curating that into trustable, separate repos meant for the "consumers" out there?

Haskell has Stackage.

https://www.stackage.org/

https://www.stackage.org/package/stackage

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 15:05 UTC (Thu) by kpfleming (subscriber, #23250) [Link] (1 responses)

Similar things exist... for example the Conda repositories are a curated set of Python (and non-Python) packages which are tested together. It is extremely time-consuming (and thus expensive) to do this sort of thing, and when subjective decisions have to be made about which version of a package to include it gets even more difficult.

Said 'consumers' will need to be willing to compensate the people who do this work; it's definitely not an effort which can be funded with volunteer time (and we can already see how that works in other areas).

PyTorch and the PyPI supply chain

Posted Jan 13, 2023 6:02 UTC (Fri) by bof (subscriber, #110741) [Link]

> Said 'consumers' will need to be willing to compensate the people who do this work; it's definitely not an effort which can be funded with volunteer time (and we can already see how that works in other areas).

Absolutely. Where you have distributions now with significant parts of the important packages somewhat current in their latest and/or rolling releases, they are backed by enough manpower to have dedicated paid people take care of a certain subject area. And they have built their base of enterprise customer subscriptions to fund that in a sustainable fashion.

Seeing Python at the top of the yearly language popularity lists, I feel that something like that should work in the dynamic languages field, too.

So, Conda, right? Is it the only "player" right now doing something like that?

PyTorch and the PyPI supply chain

Posted Jan 12, 2023 22:06 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> So why isn't there trusted language "distros"

Because that's a lot of work. There are companies that are selling this as a service, but they all kinda suck.