Debian discusses vendoring—again

By Jake Edge
January 13, 2021

The problems with "vendoring" in packages—bundling dependencies rather than getting them from other packages—seems to crop up frequently these days. We looked at Debian's concerns about packaging Kubernetes and its myriad of Go dependencies back in October. A more recent discussion in that distribution's community looks at another famously dependency-heavy ecosystem: JavaScript libraries from the npm repository. Even C-based ecosystems are not immune to the problem, as we saw with iproute2 and libbpf back in November; the discussion of vendoring seems likely to recur over the coming years.

Many application projects, particularly those written in languages like JavaScript, PHP, and Go, tend to have a rather large pile of dependencies. These projects typically simply download specific versions of the needed dependencies at build time. This works well for fast-moving projects using collections of fast-moving libraries and frameworks, but it works rather less well for traditional Linux distributions. So distribution projects have been trying to figure out how best to incorporate these types of applications.

This time around, Raphaël Hertzog raised the issue with regard to the Greenbone Security Assistant (gsa), which provides a web front-end to the OpenVAS vulnerability scanner (which is now known as Greenbone Vulnerability Management or gvm).

[...] the version currently in Debian no longer works with the latest gvm so we have to update it to the latest upstream release... but the latest upstream release has significant changes, in particular it now relies on yarn or npm from the node ecosystem to download all the node modules that it needs (and there are many of them, and there's no way that we will package them individually).

The Debian policy forbids download during the build so we can't run the upstream build system as is.

Hertzog suggested three possible solutions: collecting all of the dependencies into the Debian source package (though there would be problems creating the copyright file), moving the package to the contrib repository and adding a post-install step to download the dependencies, or removing gsa from Debian entirely. He is working on updating gsa as part of his work on Kali Linux, which is a Debian derivative that is focused on penetration testing and security auditing. Kali Linux does not have the same restrictions on downloading during builds that Debian has, so the Kali gsa package can simply use the upstream build process.

He would prefer to keep gsa in Debian, "but there's only so much busy-work that I'm willing to do to achieve this goal". He wondered if it made more sense for Debian to consider relaxing its requirements. But Jonas Smedegaard offered another possible approach: analyzing what packages are needed by gsa and then either using existing Debian packages for those dependencies or creating new ones for those that are not available. Hertzog was convinced that wouldn't be done, but Smedegaard said that the JavaScript team is already working on that process for multiple projects.

Hertzog ran the analysis script described on that page and pointed to the output from the package.json file of gsa. He said that it confirmed his belief that there are too many dependencies to package; "Even if you package everything, you will never ever have the right combination of version of the various packages."

To many, that list looks daunting at best, impossible at worst, but Smedegaard seemed unfazed, noting several reasons to believe that those dependencies can be handled. But Hertzog pointed out that the work is not of any real benefit, at least in his mind. He cannot justify spending lots of time packaging those npm modules (then maintaining them) for a single package "when said package was updated in Kali in a matter of hours". He thinks the distribution should focus its efforts elsewhere:

By trying to shoehorn node/go modules into Debian packages we are creating busy work with almost no value. We must go back to what is the value added by Debian and find ways to continue to provide this value while accepting the changed paradigm that some applications/ecosystems have embraced.

He said that Debian is failing to keep up with the paradigm change in these other ecosystems, which means that "many useful things" are not being packaged. Pirate Praveen agreed that there are useful things going unpackaged, but disagreed with Hertzog's approach of simply using the upstream download-and-build process. Praveen thinks that a mix of vendoring (bundling) for ultra-specific dependencies and creating packages for more generally useful modules is the right way forward. It comes down to distributions continuing to provide a particular service for their users:

All the current trends are making it easy for developers to ship code directly to users. Which encourages more isolation instead of collaboration between projects. It also makes it easy for shipping more proprietary code, duplication of security tracking or lack of it. Debian and other distributions have provided an important buffer between developers and users as we did not necessarily follow the priorities or choices of upstream developers exactly always.

One of the reasons Smedegaard felt that the dependencies for gsa could be handled via Debian packages is that gsa (and other large projects) tend to overspecify the versions required; in many cases, other versions (which might already be packaged for Debian) work just fine. But figuring that out is "a substantial amount of work", Josh Triplett said in a lengthy message. He cautioned against the "standard tangent" where complaints about the number of dependencies for these types of projects are aired.

[...] people will still use fine-grained packages and dependencies per the standard best-practices of those communities, no matter the number or content of mails in this thread suggesting otherwise. The extremes of "package for a one-line function" are not the primary issue here; not every fine-grained dependency is that small, and the issues raised in this mail still apply whether you have 200 dependencies or 600. So let's take it as a given that packages *will* have hundreds of library dependencies, and try to make that more feasible.

He said that disregarding a project's guidance on the versions for its dependencies is fraught, especially for dynamically typed languages where problems may only be detected at run time. For those ecosystems, the normal Debian practice of having only one version of a given library available may be getting in the way. Relaxing that requirement somewhat could be beneficial:

I'm not suggesting there should be 50 versions of a given library in the archive, but allowing 2-4 versions would greatly simplify packaging, and would allow such unification efforts to take place incrementally, via transitions *in the archive* and *in collaboration with upstream*, rather than *all at once before a new package can be uploaded*.

Triplett outlined the problems that developers encounter when trying to package a project of this sort. They can either try to make it work with the older libraries available in Debian, upgrade the libraries in Debian and fix all the resulting problems in every package that uses them, or simply bundle the required libraries. The first two are enormously difficult in most cases, so folks settle for bundling, which is undesirable but unavoidable:

Right now, Debian pushes back heavily on bundling, and *also* pushes back heavily on all of the things that would solve the problems with unbundled dependencies. That isn't sustainable. If we continue to push back on bundling, we need to improve our tools and processes and policies to make it feasible to maintain unbundled packages. Otherwise, we need to build tools and processes and policies around bundled dependencies. (Those processes could still include occasional requirements for unbundling, such as for security-sensitive libraries.)

Adrian Bunk is concerned with handling security problems in a world with multiple library versions. He said that these ecosystems seem to not be interested in supporting stable packages for three to five years, as needed by distributions such as Debian stable or Ubuntu LTS. More library proliferation (version-wise) just means more work for Debian when the inevitable CVE comes along, he said.

But Triplett said that he is not expecting there to be a lot of different library versions, but that at times it might make sense to have more than one:

I'm talking about packaging xyz 1.3.1 and 2.0.1, as separate xyz-1 and xyz-2 packages, and allowing the use of both in build dependencies. Then, a package using xyz-1 can work with upstream to migrate to xyz-2, and when we have no more packages in the archive using xyz-1 we can drop it.

That's different from requiring *exactly one* version of xyz, forcing all packages to transition immediately, and preventing people from uploading packages because they don't fork upstream and port to different versions of dependencies.

It seems safe to say that few minds were changed in the course of the discussion. Bunk and Triplett seemed to talk past each other a fair bit. And no one spoke up with some wild new solution to these problems. But the problems are not going to disappear anytime soon—or ever. Without some kind of shift, bundling will likely be the path of least resistance, at least until some hideous security problem has to be fixed in enough different packages that bundling is further restricted or prohibited. That would, of course, then require a different solution.

The approach currently being taken by Smedegaard, Praveen, and others to tease out the dependencies into their own packages has its attractions, but scalability and feasibility within a volunteer-driven organization like Debian are not among them. The size and scope of the open-source-creating community is vastly larger than Debian or any of its language-specific teams, so it should not come as a surprise that the distribution is not keeping up. Debian is hardly alone with this problem either, of course; it is a problem that the Linux distribution community will continue to grapple with.

to post comments

Debian discusses vendoring—again

Posted Jan 13, 2021 1:07 UTC (Wed) by jafd (subscriber, #129642) [Link] (38 responses)

In Fedora, they have already decided to stop shipping individual RPMs for Node libraries, and only have node, node-devel, and npm packaged: https://fedoraproject.org/wiki/Changes/NodejsLibrariesBun...

Chasing all the little packages for all the little libraries is a huge waste of energy, both electrical and cognitive. In that ecosystem, it makes sense to package whole applications and ship all their node_modules with them, rather than every single bit and piece.

I hope Debian ends up doing the same, it's madness otherwise.

Debian discusses vendoring—again

Posted Jan 13, 2021 7:50 UTC (Wed) by LtWorf (subscriber, #124958) [Link] (36 responses)

The issue is that fedora offers support for 5 months and debian for 5 years, and if some js library turns out to have a security bug, then the security team is supposed to hunt down possibly hundreds of copies of the same library and fix the issue.

Due to how bad js dependencies work, a single project can include multiple copies of the same library. So only within a single project there can be duplicates that need to be hunted down and fixed.

Fedora has none of those issues since it is a testing distribution for whatever new thing red hat wants to experiment this month.

Debian discusses vendoring—again

Posted Jan 13, 2021 8:31 UTC (Wed) by epa (subscriber, #39769) [Link] (22 responses)

The security bug in the Javascript library might only be exploitable if some particular function is called in some particular situation. If the library is bundled with an application, but the application doesn't call that function, then the application does not have a security hole in practice and there is no urgent need to patch it.

This suggests that the tracking of security bugs should follow the same granularity as packages. If you package an application from upstream, and it includes its own copy of some library code, it is upstream's job to check for vulnerabilities and publish new releases which fix them. Debian's job is to publish new versions of the library as a standalone package, for the benefit of applications that don't bundle it but do things "the Debian way". This seems like a fairer way to share out the work, so that upstream takes some of the extra costs of bundling, not just taking the benefits and leaving the distribution to do the extra work.

If upstream cannot commit to keep on top of security holes and publish updates for a five year period, or can't commit to providing a stable series rather than "that release is obsolete, upgrade everything", then I guess the application is not suitable for inclusion in a distribution with a 5 year maintenance window. That applies equally well whether or not the application bundles its own libraries.

Debian discusses vendoring—again

Posted Jan 13, 2021 8:53 UTC (Wed) by bangert (subscriber, #28342) [Link] (1 responses)

not being very well versed in the security area, the above seems to me to be wrong.

say you have a function that is vulnerable to a privilege escalation, but the application in question is not calling it, so it is "safe".
now assume, another part of the application has a remote code execution bug - boom, the "safe" privilege escalation is all of a sudden not so safe any more...

apart from that, figuring out if a specific function bundled in a dependency is used in a project is orders of magnitudes harder, than figuring out if the project includes a given dependency (which itself can be difficult enough).

Debian discusses vendoring—again

Posted Jan 13, 2021 10:06 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

In general, everything running inside a single process is in the same privilege domain. Once you've got arbitrary code execution in the app, having more libraries mapped in doesn't matter - they're not going to escalate your privileges.

Debian discusses vendoring—again

Posted Jan 13, 2021 9:05 UTC (Wed) by Lionel_Debroux (subscriber, #30014) [Link] (1 responses)

Well, for multiple reasons, especially cost-related ones - most developers of FLOSS remain unpaid, or at least not compensated to amounts anywhere near the time they spend on it - very few upstreams provide updates to software releases in such a way that these releases would be suitable for 5 years of security maintenance for a distro (i.e. over 5 years of upstream security maintenance, given the freeze period and also the gap between the latest upstream release and the beginning of the freeze period). Also, many upstreams do not maintain a stable release series either.

Debian discusses vendoring—again

Posted Jan 13, 2021 18:27 UTC (Wed) by hkario (subscriber, #94864) [Link]

many upstreams don't maintain a stable API, let alone a stable branch

Debian discusses vendoring—again

Posted Jan 13, 2021 9:52 UTC (Wed) by LtWorf (subscriber, #124958) [Link] (17 responses)

> The security bug in the Javascript library might only be exploitable if some particular function is called in some particular situation.

So instead of patching all the copies of the library you now need to audit each and every function call to the vulnerable library, and then still patch it wherever it is vulnerable.

How is it any less work?

> it is upstream's job to check for vulnerabilities

They use npm, realistically they won't check for vulnerabilities.

But even if they did… would they backport the fixes to whatever version debian is using? If not, then it means they can't be used in a stable distribution, because they'd change everything when fixing the security bug as well.

By the way this behaviour is what pushed distributions to drop mysql and move to mariadb en masse. Because oracle doesn't backport fixes and doesn't say what the issues are. They just say "some security issue is fixed" and so you must upgrade.

> so that upstream takes some of the extra costs of bundling

Won't happen. An incredible amount of those js packages are like 20 lines of code, that depend on other 1-3 similarly small packages.

The authors write them and forget about them in several cases. Also many of them write on windows and have no idea about linux.

> then I guess the application is not suitable for inclusion in a distribution with a 5 year

The issue is that following your criteria, basically no js application is suitable for inclusion.

Debian discusses vendoring—again

Posted Jan 13, 2021 10:08 UTC (Wed) by mjg59 (subscriber, #23239) [Link] (10 responses)

> How is it any less work?

Not having to verify that an update doesn't also break functionality you depend on is less work than having to verify that everything still works fine with a nominally compatible security backport.

Debian discusses vendoring—again

Posted Jan 13, 2021 11:52 UTC (Wed) by LtWorf (subscriber, #124958) [Link] (9 responses)

Backports of security bugs don't change APIs.

Debian discusses vendoring—again

Posted Jan 13, 2021 12:21 UTC (Wed) by mjg59 (subscriber, #23239) [Link] (7 responses)

Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha

(No)

Debian discusses vendoring—again

Posted Jan 13, 2021 14:45 UTC (Wed) by LtWorf (subscriber, #124958) [Link] (6 responses)

Care to point out 1 patch done by a distribution security team that did change API?

Debian discusses vendoring—again

Posted Jan 13, 2021 14:58 UTC (Wed) by pizza (subscriber, #46) [Link] (5 responses)

> Care to point out 1 patch done by a distribution security team that did change API?

Sure. I lost soft scrollback in my linux consoles, because upstream's "fix" for the security hole was to strip out the feature entirely. Maintaining that feature would have required devising an independent fix (which IIRC never happened) or reverying the upstream changes, leaving that hole open.

Debian discusses vendoring—again

Posted Jan 14, 2021 13:11 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (4 responses)

That's not an API.

Debian discusses vendoring—again

Posted Jan 14, 2021 13:56 UTC (Thu) by pizza (subscriber, #46) [Link] (3 responses)

Okay, fine, if you want to split hairs, I'll be more explicit.

The linux kernel API that allowed for software-based console scrollback was completely removed as part of a "stable kernel security update" from my distribution.

Sure, the patch that removed it originated with the upstream kernel, but it was still an API change that the distro pushed out to its users.

Debian discusses vendoring—again

Posted Jan 15, 2021 14:01 UTC (Fri) by jafd (subscriber, #129642) [Link] (2 responses)

Was the scrollback something you could use programmatically via a kernel API? Or maybe it exposed an ioctl(2) API? Or was there another driver that stopped working entirely because it was using some of the functions that had been a part of scrollback and got obliterated?

I'm confident the world could be better without this very bizarre definition of "API" both you and mjg59 are using here.

Debian discusses vendoring—again

Posted Jan 15, 2021 15:20 UTC (Fri) by farnz (subscriber, #17727) [Link]

It was - TIOCLINUX subcode 13 (undocumented, but existed) let you control the scrollback from user space. There are documented ioctls for copy and paste you can use to programmatically extract data from a VC.

Debian discusses vendoring—again

Posted Jan 22, 2021 5:19 UTC (Fri) by HelloWorld (guest, #56129) [Link]

This definition isn't bizarre at all.

Debian discusses vendoring—again

Posted Jan 15, 2021 14:12 UTC (Fri) by jafd (subscriber, #129642) [Link]

Not always, and certainly not in the javascript/NodeJS world.

I won't get into the madness of the neighboring thread (where they are discussing a user-visible change which is not an API change), but here's a very real scenario I'm seeing in a proprietary app that's my job to work on:

1) for $REASONS, we're stuck using a tool of the major version N. It's rather expensive to move to a tool of major version N+1 or N+2.
2) by now, some fourth-order dependencies of the tool had security vulnerabilities discovered, and the only path forward is to bump their major versions. Because JavaScript developers don't believe in backports. A dozen small libraries, but they are all over the place.
3) bumping them will need also bumping the third-order dependencies and so on, which is going to break the tool of version N, no longer maintained.
4) this kind of yak-shaving can take a lot of time, and I'm not counting the regressions that crept in because the migration documentation is far from being complete or accurate. Oh, and new versions of the tool produced buggy output which wasn't our fault. Back to the drawing board we are.

So, basically, until we find time to bring the tool to N+2, we're hosed according to "npm audit". Thankfully the tool is not user-facing, but you can see how "minor fixes" sometimes necessitate bigger changes a project may not be ready to embrace yet.

Debian discusses vendoring—again

Posted Jan 13, 2021 13:03 UTC (Wed) by khim (subscriber, #9252) [Link] (5 responses)

> The issue is that following your criteria, basically no js application is suitable for inclusion.

Depends on the promises they make.

> Won't happen. An incredible amount of those js packages are like 20 lines of code, that depend on other 1-3 similarly small packages.

Which, in turn, means that any JS application is POS which can and would be owned by malicious input presented to it.

That's why all these talks about trying to fix libraries, etc is pointless. You don't need to do it.

Instead you have to think what protections you have in place for the inevitable breakage which results. How these applications are compartmentalized is important. How data is protected from them is important.

How these apps are updates is NOT important because it's just pointless waste of time to try to plug holes in the sieve.

Debian discusses vendoring—again

Posted Jan 13, 2021 13:37 UTC (Wed) by pizza (subscriber, #46) [Link] (1 responses)

> Depends on the promises they make.

That's easy. Not only do they make no promises, they explicitly disclaim warranties, including merchantability and fitness for any purpose.

Debian discusses vendoring—again

Posted Jan 13, 2021 16:09 UTC (Wed) by smoogen (subscriber, #97) [Link]

Wait you mean all those licenses and such which no one reads but says that the writer disclaims all responsibility actually mean something?

Personally I look at all these Javascript/Python/etc to being similar to having every call in libc/libm etc as separate libraries versus 'bundled' together as a single item.

Debian discusses vendoring—again

Posted Jan 15, 2021 3:37 UTC (Fri) by VolkerWeissmann (guest, #144200) [Link] (2 responses)

> Which, in turn, means that any JS application is POS which can and would be owned by malicious input presented to it.

What does POS stand for?

Debian discusses vendoring—again

Posted Jan 15, 2021 14:13 UTC (Fri) by jafd (subscriber, #129642) [Link]

A "Piece Of Excrement".

Debian discusses vendoring—again

Posted Jan 15, 2021 15:50 UTC (Fri) by MKesper (subscriber, #38539) [Link]

Pile of shit, I guess?

Debian discusses vendoring—again

Posted Jan 13, 2021 9:01 UTC (Wed) by zdzichu (subscriber, #17118) [Link] (1 responses)

Last sentence of you comment is blatantly false.

First, Fedora offers 13 months of support for each version.

Second, Fedora is fully fledged distribution with stable releases and various variants (for example Workstation, Server). It is NOT "a testing distribution".

I am a Fedora maintainer. I am not associated with Red Hat - most of us aren't.

Debian discusses vendoring—again

Posted Jan 13, 2021 9:54 UTC (Wed) by LtWorf (subscriber, #124958) [Link]

It was a hyperbole. Remains the issue that 13 months is a smaller number than 60.

Debian discusses vendoring—again

Posted Jan 13, 2021 10:15 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Realistically Kubernetes is much better supported by upstream than Debian. Google probably has an order of magnitude more security people working on it than the whole Debian security team. Ditto for many other packages.

It's just not something that Debian should tackle. Sure, it goes against the old model of Linux distros but times change...

Debian discusses vendoring—again

Posted Jan 14, 2021 0:58 UTC (Thu) by dvdeug (subscriber, #10998) [Link]

Except we're not talking about Kubernetes; we're talking about GSA. GSA almost certainly doesn't have an order of magnitude more security people than Debian. And when we're done with GSA, we're going to be talking about an RPG written in an open source JS framework. Part of the problem is there is no line here; the big items may have support, but the small items are what make up most of the system, and they're endless.

Debian discusses vendoring—again

Posted Jan 13, 2021 12:06 UTC (Wed) by roc (subscriber, #30627) [Link] (1 responses)

Recently in our Rust project (https://pernos.co) we integrated https://github.com/EmbarkStudios/cargo-deny into our CI so that as soon as a CVE is published for any library our project depends on, our CI starts failing. That CVE data is managed at https://github.com/RustSec/advisory-db and could be consumed by all manner of tools.

I don't know the NPM world, but if they don't have something like this already, they need it. Distro packagers could use such a tool to detect which packages contain bundled dependencies that need updating.

Debian discusses vendoring—again

Posted Nov 6, 2023 11:43 UTC (Mon) by j0057 (subscriber, #143939) [Link]

I agree, the distro model is something from the world of C where system-provided dynamic libraries are the only way to re-use code, and even there many applications have started vendoring their dependencies.

Languages other than C have proper packaging support, and it's now up to the distros to adapt and make use of the information there is. Hunting down which application uses which library is really not that difficult, unless you're unable to tap into the metadata that a given packaging ecosystem already provides.

Debian discusses vendoring—again

Posted Jan 13, 2021 12:12 UTC (Wed) by cesarb (subscriber, #6266) [Link] (6 responses)

> The issue is that fedora offers support for 5 months and debian for 5 years,

Fedora offers support for 13 months, not 5 months.

> and if some js library turns out to have a security bug, then the security team is supposed to hunt down possibly hundreds of copies of the same library and fix the issue.

Fedora uses a trick to help with that: as explained at https://fedoraproject.org/wiki/Bundled_Libraries every package which includes a bundled library will have a "Provides: bundled(package) = version", so it's easy to find which packages have a given bundled library, and on which version.

Debian discusses vendoring—again

Posted Jan 13, 2021 18:57 UTC (Wed) by drawks (guest, #144158) [Link] (5 responses)

I recall years ago that there was some suggestion that debian could similarly use the "Built-Using:" metadata tags for indicating that a given package has some vendored code included in it. The "Built-Using:" tag already exists for tagging the handful of statically compiled packages which cannot avoid vendoring (busybox, various installer components, etc). The existing presumption is that the vendored components are available at package build time as satisfiable "Build-Depends:". The difference in the npm situation is that at build time the dependencies which are to be bundled would not be packaged separately, but would come from $ambiguous_source and the build system would have to be trusted to tag in the "Built-Using:" data.

This type of solution is a find technical solution if not 100% philosophically "clean" in the way debian intends. I would posit that such a solution would be fine for third party package providers and it would be a big step towards solving real world problems if Debian would just outline that this is a best practice for third parties and that first party debian packages would still be required to provide individually packaged packaged dependencies at build time. Softening the the limits on the number of versions of a library can be in the debian repositories would further make this a maintainable situation moving forward with a reasonable path for all various upstreams to cut between major versions of their dependencies.

at any rate thats my .02 maybe someone can tell me why I'm completely wrong :)

Debian discusses vendoring—again

Posted Jan 13, 2021 19:35 UTC (Wed) by smcv (subscriber, #53363) [Link] (4 responses)

This isn't what Built-Using is for. A new metadata field could absolutely be used to track what's vendored into each package, but Built-Using is not it.

Built-Using is for the situation when we have some code in one Debian package, and we copy that code into a different Debian package during the second package's build. For instance, if you build busybox-static while you have libc6-dev_2.31-9 installed, version 2.31-9 of libc.a gets statically linked into the busybox-static binary. Now, as long as we have that copy of busybox-static in the archive, we also have to keep a copy of the glibc_2.31-9 source package around, otherwise we'll be missing the source code for some of our binaries (which is a LGPL violation).

Which upstream packages are vendored into the source package for which Debian packages could be tracked in a decentralized way by inventing a new metadata field, vaguely similar to Fedora's use of Provides, but at the moment it's tracked centrally instead: https://salsa.debian.org/security-tracker-team/security-t...

Debian discusses vendoring—again

Posted Jan 13, 2021 20:09 UTC (Wed) by drawks (guest, #144158) [Link] (3 responses)

Your description of how it works doesn't sound too different to vendoring, especially for the go/rust type of situations where you are literally vendoring by way of statically linking. I'd be unopposed ot using a new metadata tag that is more specifically scoped for this case, but my point still stands that this is a usable mechanism in all the ways I have described.

Debian discusses vendoring—again

Posted Jan 14, 2021 17:07 UTC (Thu) by smcv (subscriber, #53363) [Link] (2 responses)

The resulting binary package has similar properties, but the way you get there is different. With a package that uses Built-Using, a no-changes rebuild will make it pick up the current versions of the packages that it was Built-Using (for example its statically linked copy of glibc, or its statically linked copy of a Go or Rust library). With a package that uses vendoring, the vendored versions of its dependencies are part of the package's source code and will not change without an update to the source package.

Debian discusses vendoring—again

Posted Jan 15, 2021 3:37 UTC (Fri) by VolkerWeissmann (guest, #144200) [Link] (1 responses)

If you have
mydependency = "1.2.3"
in your Cargo.toml, then cargo won't update it if you rebuild it.

Debian discusses vendoring—again

Posted Jan 15, 2021 4:36 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

I think you want `"=1.2.3"` there for that. Without any comparison operator, you get `^` or `~` (depending on a zero major version) which means "any compatible version" in practice.

Debian discusses vendoring—again

Posted Jan 13, 2021 15:35 UTC (Wed) by ncm (guest, #165) [Link]

I start with the asumption that Node.js is, itself, not fixable at the distro level, end of story.

But, what about the rest? The problem to solve is things like Go packages that want to static-link their dependencies.

One way forward is to consider programs that want to static-link as being, effectively, scripts. So, like scripts, they are not packaged as complete executables. Their dependencies do not refer to a specific version, but provide a link to where to get any specific version wanted. The thing in /usr/bin is a script that checks for a cached build and, *if needed*: invokes the build tool, which uses local copies where found and usable, and downloads updates where needed, and links.

A package- or dist-upgrade doesn't pull new versions of dependencies; it just notes where local, cached copies are stale, and flushes. On next execution -- exactly as with scripts -- invalidated builds are scrapped, and rebuilt.

It means that to use a Go program, you need a Go toolchain, but that is not a large burden.

It means that the responsibility of the package system is only to flush caches when breaking or security updates happen. The target itself arranges to be able to start up quickly, on the 2nd run, much like the Python packages we already deal with.

Debian discusses vendoring—again

Posted Jan 13, 2021 2:49 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

A thing to consider: Debian's dependency resolution machinery is already barely adequate. If you add several packages that have complex library dependencies, then the satisfiability solver is just going to die completely.

This already can happen during complex upgrades, and thousands of JS/Go packages will just make it inevitable.

Debian discusses vendoring—again

Posted Jan 13, 2021 8:49 UTC (Wed) by jond (subscriber, #37669) [Link]

Indeed. Whilst it’s tempting to let those who want to boil the ocean packaging individual npm modules to get on with it, it causes active harm to the rest of the project. Yours is but one example.

Debian discusses vendoring—again

Posted Jan 13, 2021 7:48 UTC (Wed) by Archimedes (subscriber, #125143) [Link] (10 responses)

It is unfortunate that due to the download during build (one thing I personally am totally sure that that is the utterly wrong way to go) less and less packages will be packaged in the distribution.
One (small to big) thing the developers of the software (which understandably are not interested in packaging for myriads of linux distributions) looses the input of others which need to incooperate his package in a larger infrastructure.

- So others try to rebuild his package (and not "download this docker image to rebuild us" desasters)
- so follow the configuration of the build (and add/remove features of it)
- follows the (build) dependencies (and might discover problematic ones)
- follows the run dependencies (like needs db or webserver or ... and provides configuration integration)
- puts the results in the "correct place" (depending on the style of the distribution).
- adds the package to the "system" like add to systemd, add to rsyslog (so depending on the distribution choices)

The user maybe looses the "newest" version but gains a more checked one and he gets one single way to update the whole systems and can be (at least more) sure that the package is integrated in *his* choices of the distribution.

I personally try to never download build packages from the developer but follow:
- 1. try the distribution one
- 2. try the "experimental/unstable" version of the package
- 3. download the source, try to build and (thus have to ) self maintain the version.

If all that fails, i reconsider hard if the package is that interesting and some times another one is then the better suited one.

Debian discusses vendoring—again

Posted Jan 13, 2021 8:24 UTC (Wed) by josh (subscriber, #17465) [Link] (3 responses)

For the record, while I believe that "one major version" and "unvendor everything" are frequently conflicting policies, I absolutely think Debian has the right policy in prohibiting all network access during builds. The distribution must remain self-contained and reproducible.

Debian discusses vendoring—again

Posted Jan 13, 2021 9:48 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

Is it possible that, once the distro goes "stable" (I'm not a Debian user, does that mean packages are no longer updated to newer versions), that is the point at which "un-vendoring" starts to occur?

So yes it will dump a lot of work on the security people in a way, but the rule would become "any security fixes to a vendored module must include un-vendoring it".

Cheers,
Wol

Debian discusses vendoring—again

Posted Jan 13, 2021 10:14 UTC (Wed) by LtWorf (subscriber, #124958) [Link] (1 responses)

You can't un-vendor.

A vendored module might have patches that are not in the original.

In stable, updates are generally done only for security or serious bugs. In general any bug fix is allowed, but no new features or changes.

So you don't get surprises every day.

The download at compile time is awful because then you basically can't audit what's in the binary, can't have reproducible builds, can't ensure that it will still build tomorrow when the domain it is downloading from is sold to someone else.

Debian discusses vendoring—again

Posted Jan 18, 2021 13:10 UTC (Mon) by amck (subscriber, #7270) [Link]

This is confusing two issues.

De-vendoring is a pain, but possible.

Updates to stable for security fixes need to be done on a stable branch, porting just the security fix to the update - not including any other changes relative to existing stable. This is painful but possible.

The versioning naming done makes this possible. e.g. apache2-bin on my stable machine has version "2.4.38-3+deb10u4", the +version part showing the change which falls between stable and testing as required.

The real challenge is (a) bundling (b) overwhelming human resources and the resolution manager in apt.

Debian discusses vendoring—again

Posted Jan 13, 2021 13:41 UTC (Wed) by kleptog (subscriber, #1183) [Link] (5 responses)

The issue is we don't want to spend a lot of time packaging a lot a NPM/Python modules in DEB packages. So perhaps the answer is to not do that. Instead, declare a set of particular versions of NPM/Python packages as being "recognised/approved/whatever" and mirror them to a Debian NPM/Python package repo and then the buildbots can access specific versions of those packages.

This gives you reproducible builds because you can simply refer to the Debian recorded versions of those packages (the package build script could provide a list of "associated" packages with versions and hashes). You need to somehow filter the network traffic of the buildbot to only use access these repositories. This also means if you note that one of those "associate" packages has a security issue, you can immediately identify the Debian packages that used it and take measures.

This does mean Debian would need to manage a whole new set of repositories, but there are tools for this. Still a ridiculous amount of work, so it depends on whether the benefits are significant enough.

Debian discusses vendoring—again

Posted Jan 13, 2021 21:15 UTC (Wed) by plugwash (subscriber, #29694) [Link]

> The issue is we don't want to spend a lot of time packaging a lot a NPM/Python modules in DEB packages.

I don't think this is correct.

Actually producing a package that builds and installs takes negligable time. What takes the time is.

1. Documenting/researching the copyright/license situation and removing anything that violates Debian standards.
2. Waiting for the ftpmasters to validate that what you have done for 1.
3. Deciding what do do about dependency versions, trying to find the path between having piles of versions of the same thing hanging around (which is bad from a security point of view) and breaking stuff when a package is updated.

Debian discusses vendoring—again

Posted Jan 13, 2021 21:48 UTC (Wed) by lkppo (guest, #97462) [Link] (3 responses)

That sounds like a good alternative. Another solution would be to have scripts to automatically bulk package the libraries of node, pip, composer, etc. managers in DEB format, independently of the applications that use them. All that remains for the application maintainer is to use the dependencies he needs and possibly plug the holes.

Debian discusses vendoring—again

Posted Jan 14, 2021 3:34 UTC (Thu) by pabs (subscriber, #43278) [Link]

The problem with language ecosystem packages is that they often differ from the upstream VCS repo, sometimes even missing source code or build tools to build some generated files.

That said, there are already tools to convert language ecosystem packages to Debian source packages. I think Debian needs to go further than that and integrate more of those things into debhelper and similar.

https://wiki.debian.org/AutomaticPackagingTools

Debian discusses vendoring—again

Posted Jan 14, 2021 14:34 UTC (Thu) by pj (subscriber, #4506) [Link] (1 responses)

The other problem is that debian enforces 'one version of a thing' while other package managers don't - npm, for example, lets every package have its own dependency sub-tree, so there may be multiple versions of the same package in use (even major versions, so they have different APIs!). Fixing this for debian requires the version-in-the-name hack, which isn't too bad, but figuring out which packages _need_ this fix might be more difficult. I guess Debian could package them all with the major version, maybe with a kind of 'realm' prefix: npmlib-foojs-1 for foojs v1.x.

Debian discusses vendoring—again

Posted Jan 26, 2021 4:42 UTC (Tue) by calumapplepie (guest, #143655) [Link]

The central issue is that we don't want to have multiple versions of a package, even when we can. Putting the version in the name is long-established practice for core packages that have major versions: it's the best way to handle upgrades, for instance. However, tracking three different versions of a package means that security and bugfix support becomes almost impossible: we now need to backport fixes not once, but many times.

The fact that apt doesn't permit per-package dependency trees is intentional. We don't want duplicate (or nearly-duplicate) files on our systems, our mirrors, or our security trackers. NPM's ability to let you have a dozen versions of every package just leads to headaches when you need to update one.

Debian discusses vendoring—again

Posted Jan 13, 2021 8:16 UTC (Wed) by xophos (guest, #75267) [Link]

Libraries with hundreds or thousands (yes it happens) of dependencies are dangerous by definition. Alowing multiple versions of those multiplies the problem. Just don't include them and don't use them. Yes, maybe some of them are handy but the risks are just to high.
Those who don't value their privacy and security still have the option of installing that crap without the help of debian. It's even easy.

Debian discusses vendoring—again

Posted Jan 13, 2021 9:25 UTC (Wed) by Karellen (subscriber, #67644) [Link] (4 responses)

I'm talking about packaging xyz 1.3.1 and 2.0.1, as separate xyz-1 and xyz-2 packages, and allowing the use of both in build dependencies. Then, a package using xyz-1 can work with upstream to migrate to xyz-2, and when we have no more packages in the archive using xyz-1 we can drop it.

It seems odd to me if this hasn't been part of the discussion already. Debian already includes packages for gtk-2, gtk-3 and gtk-4, and qt-4, qt-5 and qt-6 as two very prominent examples of parallel semver major releases, but I know I've seen quite a few other similar smaller examples browsing the package lists. Why would that not be an obvious part of the background for everyone involved?

Debian discusses vendoring—again

Posted Jan 13, 2021 13:27 UTC (Wed) by khim (subscriber, #9252) [Link] (2 responses)

The big difference is in timing. GTK+ and QT+ have different versions, sure. But upstream supports these. There are parallel releases — but these are supposed to be parallel-installed.

And the whole discussion is about things like Jenkins where two versions are not supposed to be installed and where about three months (but we don't know how much, exactly) is supposed to be called “Long Term Support” (I wish I was joking, I'm not).

Debian discusses vendoring—again

Posted Jan 15, 2021 6:25 UTC (Fri) by murukesh (subscriber, #97031) [Link] (1 responses)

According to https://www.jenkins.io/download/lts/, they seem to have settled on a 12-week cadence. But considering they say the normal release cadence is weekly (!), 12 weeks seems like a very long time in comparison (comparing with, say, Ubuntu, where the ratio of LTS/normal support duration is 60 months/9 months ~ 6.66).

Debian discusses vendoring—again

Posted Jan 22, 2021 2:48 UTC (Fri) by khim (subscriber, #9252) [Link]

You couldn't release anything weekly. There are just not enough time to find and fix problems in any new features.

The most you can promise if you “release” weekly is “hey, it passed our automatic testing and therefore we hope it's not completely broken”. That's not a release. 20 years ago it would be called “alpha version” (not even “beta”: beta is supposed to be tested by some testing teams before release, not just pass CI bot). I'm not even sure that thing which they call “an LTS release” can be compared to “normal” stable release of XX century software which you can actually trust.

The really strange result of all that activity is the fact that as “release cadence” shortens the actual, tangible, changes become less and less frequent. But resource consumption and number of bugs skyrockets. Simply because nobody can understand how the whole thing is supposed to work — and the amount of band-aids applied to pile of existing band-aids which ensure that the whole thing doesn't crash immediately upon startup is just endlessly grows.

As Hoare (original inventor of Quicksort) said: there are two ways of constructing a software design — one way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.

What we are looking on in that pile of weekly “releases” and pile of “containers” is the end result of application of 2nd principle.

Debian discusses vendoring—again

Posted Jan 13, 2021 21:50 UTC (Wed) by plugwash (subscriber, #29694) [Link]

"quite a few" yes, but still a small minority of the total packages. Packaging multiple major versions of the same software is neither explicitly prohibited nor explicitly permitted. In practice it is strongly frowned-upon, though it does happen.

If you just package a new version every time upstream pushes semver and don't go back and clean things up you can quickly end up with a whole pile of versions of a whole pile of libraries.

Some people advocate that it's better to package a new version first in parallel, then go back and port software, rather than trying to do the transition in one bang. The fear is without the carrot of getting the new version of your pet project into testing and/or the stick of having your pet project removed from testing, the work will never get done and the version proliferation will become a reality.

And it gets even worse if the upstream community doesn't see version proliferation or proliferation of tiny unmaintained packages as problems.

Debian discusses vendoring—again

Posted Jan 13, 2021 10:29 UTC (Wed) by dottedmag (subscriber, #18590) [Link] (18 responses)

Semver discussion is unfortunately a distraction, because people make mistakes. As an upstream author I don't have time to test my code against all minor versions of a dependency, and I regularly see dependencies breaking their backward compatibility promise and not bumping major version as a result. Therefore as an author I can only vouch for the particular set of dependencies I have tested my software against. If a distribution wishes to run it against another set of dependencies, then all bets are off, both due to Hyrum's law and due to aforementioned fallibility of humans.

Moreover, the talk of security updates is obsolete for a large part of ecosystem: as long as the upstream cares about portability to Windows and macOS, that upstream already has to care about security updates, so having a heads-up from a distribution security team is helpful, but it is ultimately a duplicate effort, except for the narrow case of a security vulnerability in a dependency specific to Linux.

Debian discusses vendoring—again

Posted Jan 13, 2021 10:41 UTC (Wed) by LtWorf (subscriber, #124958) [Link] (17 responses)

What makes you think that upstream does care about security?

And even if they care, would they backport the security fixes or just tell users to upgrade to the next major version that changes everything as well?

The "use the latest" approach to security doesn't work for stable distributions.

Debian discusses vendoring—again

Posted Jan 13, 2021 11:17 UTC (Wed) by dottedmag (subscriber, #18590) [Link] (16 responses)

> What makes you think that upstream does care about security?

To repeat my original comment: "as long as the upstream cares about portability to Windows and macOS, that upstream already has to care about security updates". If upstream does not care about security on other platforms, then using this software is dangerous no matter what.

> The "use the latest" approach to security doesn't work for stable distributions.

As an upstream author I don't really care about distributions, all I care about is users who can use my software. So bundling everything into a single static binary is the approach that works best for me and for them (I don't even care about libc in this case, yay stable kernel interfaces and X11 / Wayland!), relying on a small set of platform libraries is the next best thing (there are weird corner cases, but more-or-less it works), giving a distributor full control over dependencies is a nightmare.

"Everything in a static bundle" works only for projects that are serious about tracking security bugs in their dependencies, of course. However portability to anything beyond Linux forces one to think about it.

Debian discusses vendoring—again

Posted Jan 13, 2021 12:00 UTC (Wed) by LtWorf (subscriber, #124958) [Link] (12 responses)

Developers want users to use the latest version.

However developers also introduce incompatibilities all the time.

For example in gnome updates break extensions, so users after upgrading need to patch their extensions or not use them until someone else does this job.

I understand you don't care, but users care. Users don't use one software, they use hundreds of them and might have scripts to use many of them at once, so when they change, they need to fix whatever broke.

If your newest and greatest version doesn't have any feature that they need, they might not care about installing it to make you happy, and might prefer to keep using the old one so that they don't have to fix their scripts, configurations, and so on.

So that is why stable distributions exist, and why people use them, regardless of what you personally might wish.

> yay stable kernel interfaces and X11 / Wayland!

See, you appreciate stability yourself but don't want to provide it… Proving my point that stability is needed.

What if you use gtk3 and they tell you "there is a security bug but you need to use gtk4 for the fix"? Then you'd have to rewrite your software for gtk4, it would no longer work for any platforms that got dropped, you need to rewrite all of your tests and so on.

All in all, a very selfish comment :)

Debian discusses vendoring—again

Posted Jan 13, 2021 12:31 UTC (Wed) by dottedmag (subscriber, #18590) [Link] (9 responses)

> See, you appreciate stability yourself but don't want to provide it…

Well, I don't break interfaces. Shocking! Problem solved.

> What if you use gtk3 and they tell you "there is a security bug but you need to use gtk4 for the fix"? Then you'd have to rewrite your software for gtk4, it would no longer work for any platforms that got dropped, you need to rewrite all of your tests and so on.

That's the gist of the problem, the CADT model: https://www.jwz.org/doc/cadt.html

The solution is simple. Do not depend on libraries whose authors regularly drop backward compatibility or don't provide extended security support for several major versions. These libraries are not worth the hassle.

Debian discusses vendoring—again

Posted Jan 13, 2021 12:39 UTC (Wed) by pizza (subscriber, #46) [Link] (8 responses)

> The solution is simple. Do not depend on libraries whose authors regularly drop backward compatibility or don't provide extended security support for several major versions. These libraries are not worth the hassle.

Ah, okay, so you just excluded pretty much everything web facing (and nearly the entirety of the javascript, python, go, and rust ecosytems)

Meanwhile, in the real world, the choice is "use this library" or "write something equivalent from scratch". Repeated dozens of times.

Debian discusses vendoring—again

Posted Jan 13, 2021 12:51 UTC (Wed) by dottedmag (subscriber, #18590) [Link] (7 responses)

This is obviously true for codepaths that are security-sensitive: depending on a CADT-model Web-facing or other untrusted-input library is a recipe for disaster (or increased blood pressure and loss of sleep at least).

Fortunately, not everything is security-sensitive, so the requirements may be relaxed there, and untrusted input boundaries are not that hard to contain if that's understood as a necessary part of engineering process.

> Meanwhile, in the real world, the choice is "use this library" or "write something equivalent from scratch". Repeated dozens of times.

Hard to reimplement, bad security policy => isolate. This one is tough. Containerization helps somehow, but it's best if this set is empty.

Easy to reimplement, bad security policy => drop. Over time I find more and more code falls into this bucket.

Hard to reimplement, good security policy => use. Precious. Rare.

Easy to reimplement, bad security policy => use if fits exactly. Freebies. Nice to have.

Debian discusses vendoring—again

Posted Jan 13, 2021 12:52 UTC (Wed) by dottedmag (subscriber, #18590) [Link]

> Easy to reimplement, bad security policy => use if fits exactly. Freebies. Nice to have.

This one should read "good security policy"

Debian discusses vendoring—again

Posted Jan 13, 2021 13:36 UTC (Wed) by pizza (subscriber, #46) [Link] (5 responses)

> This is obviously true for codepaths that are security-sensitive: depending on a CADT-model Web-facing or other untrusted-input library is a recipe for disaster (or increased blood pressure and loss of sleep at least).

Welcome to software development in the real world, where the _only_ consideration [1] is "how much will it cost this quarter"

(FWIW I don't disagree with anything you have written; but at the end of the day, the folks paying our salaries decide what our development priorities are)

[1] Beyond what is legally mandated by law or private contract (eg required by your payment processor or insurance carrier)

Debian discusses vendoring—again

Posted Jan 13, 2021 13:45 UTC (Wed) by dottedmag (subscriber, #18590) [Link] (4 responses)

I happen to be making these decisions and I know how to translate them to the money.

«This critical-and-not easy-to-replace library has N 0-day exploits with RCE per year. The choices are: 1) use it as is and sustain M break-ins, each conservatively estimated to cost $X / year in lost revenue, cleanups, damage control etc plus $Q for initial use of this library; 2) proactively monitor and patch vulnerabilities, estimated cost is $Y / year plus $Q for initial use of this library; 3) develop a replacement in-house, estimated cost is one-time $Z and $W / year maintenance»

Sure it takes some experience and data to start making these decisions, but once the data is there it becomes much easier to decide. Also over time (3) becomes cheaper as problems don't tend to be unique every time.

Debian discusses vendoring—again

Posted Jan 13, 2021 14:47 UTC (Wed) by pizza (subscriber, #46) [Link] (3 responses)

> 3) develop a replacement in-house, estimated cost is one-time $Z and $W / year maintenance»
> Also over time (3) becomes cheaper as problems don't tend to be unique every time.

You left out the cost of cleaning up after exploits found in your in-house library. Because your in-house code is far (far!) more likely to be full of holes than random F/OSS libraries that are used by a variety of other folks.

You also left out the fact that if it's in-house, $Z and $W are paid entirely by your organization, so while (3) might get smaller (relative to itself) over time, it will forever remain much larger than using a F/OSS library whose $Z and $W approach $0, as you're not shouldering the entire price of development and maintenance yourself.

In the end, most organizations go with (1) because they simply don't have the resources to do anything differently. (3) tends to only happen in areas where cost is not a primary consideration (eg military, medical, safety, and other highly regulated industries) or when there are specific requirements that cannot be met any other way. Varying degrees of (2) is where most organizations end up, typically balanced on the trailing edge of cost-benefit analyses.

Debian discusses vendoring—again

Posted Jan 13, 2021 18:36 UTC (Wed) by hkario (subscriber, #94864) [Link] (2 responses)

(3) also makes sense if any of the existing solutions don't share the goals you have (platform support, maintenance windows, etc.)

and just because you develop in-house doesn't mean you have a). started from zero, b). don't open-source it so that others that do share your goals can contribute

Debian discusses vendoring—again

Posted Jan 13, 2021 19:31 UTC (Wed) by pizza (subscriber, #46) [Link] (1 responses)

oh, quite right.

It's just that when you make that decision to go your own way, you can't assume that anyone else will ever contribute anything, much less shoulder the majority burden of developing/maintaining it.

Debian discusses vendoring—again

Posted Jan 13, 2021 19:36 UTC (Wed) by dottedmag (subscriber, #18590) [Link]

This equally applies to every open source library ever published, doesn't it? :)

Debian discusses vendoring—again

Posted Jan 13, 2021 12:32 UTC (Wed) by dottedmag (subscriber, #18590) [Link]

> However developers also introduce incompatibilities all the time. For example in gnome updates break extensions, so users after upgrading need to patch their extensions or not use them until someone else does this job.

BTW that's why I wouldn't touch anything in this ecosystem with a very long pole.

Debian discusses vendoring—again

Posted Jan 13, 2021 12:36 UTC (Wed) by pizza (subscriber, #46) [Link]

> I understand you don't care, but users care. Users don't use one software, they use hundreds of them and might have scripts to use many of them at once, so when they change, they need to fix whatever broke.

I guess users need to put their money where their mouth is, eh?

Debian discusses vendoring—again

Posted Jan 13, 2021 15:52 UTC (Wed) by liori (guest, #117124) [Link] (2 responses)

> As an upstream author I don't really care about distributions, all I care about is users who can use my software.

And as a user, I don't want to download my applications each from a different source. I don't want to remember to update them, each using a different tool. And I want to reduce the risks involved with updates breaking my workflows.

I want to outsource all of this to the distribution. And, for me, when an application is prepared so that outsourcing all this work to the distribution is easy is a feature I value.

This is why I'm using a Linux distribution and not, let say, LFS.

Debian discusses vendoring—again

Posted Jan 13, 2021 19:45 UTC (Wed) by dottedmag (subscriber, #18590) [Link] (1 responses)

> And as a user, I don't want to download my applications each from a different source. I don't want to remember to update them, each using a different tool.

That's a solved problem, I believe.

> And I want to reduce the risks involved with updates breaking my workflows.

I feel for you and many more (myself included) who are either stuck with old version of software and no support or forced to upgrade to incompatible version. As developers we ought to treat our users better and break less, that's for sure.

Debian discusses vendoring—again

Posted Jan 14, 2021 13:15 UTC (Thu) by LtWorf (subscriber, #124958) [Link]

> That's a solved problem, I believe.

It is, for software that respects licenses, policies, and is not a nightmare to package.

But you just said you don't want to do any of those things with the software you write. So it's not solved for your software.

Debian discusses vendoring—again

Posted Jan 13, 2021 15:22 UTC (Wed) by pabs (subscriber, #43278) [Link]

More comments on Hacker News:

https://news.ycombinator.com/item?id=25758863

Debian discusses vendoring—again

Posted Jan 13, 2021 16:26 UTC (Wed) by cortana (subscriber, #24596) [Link]

Debian's package of docker ("docker.io") vendors some of its dependencies. From <https://packages.debian.org/source/sid/docker.io#pdownload> you can see that cli, libnetwork and swarmkit are included within the docker.io source package, rather than packaged separately.

There's some info about this at <https://www.collabora.com/news-and-blog/blog/2018/07/04/d...>.

But I guess these are the easy cases: where the dependency has no other reverse-dependencies. Where there are other reverse-dependencies, vendoring becomes less acceptable...

Solved problem in computer science

Posted Jan 14, 2021 2:32 UTC (Thu) by davecb (subscriber, #1574) [Link]

First seen in Multics, reinvented in Solaris and again in glibc. Due to be reinvented any time now (;-))

--dave

TL;DR, in reverse order by date:
https://leaflessca.wordpress.com/2018/09/03/avoiding-an-n...
https://cacm.acm.org/magazines/2009/11/48444-you-dont-kno...
https://www.usenix.org/legacy/publications/library/procee...
https://research.swtch.com/vgo-mvs
https://leaflessca.wordpress.com/2017/02/12/dll-hell-and-...
https://lwn.net/Articles/712318/
https://research.swtch.com/version-sat

Debian discusses vendoring—again

Posted Jan 14, 2021 16:07 UTC (Thu) by gnu_lorien (subscriber, #44036) [Link] (5 responses)

I hope that Debian and other distributions find a solution to this problem. About a decade ago when I wanted to check out a new piece of server software I'd often install it from the package manager. In the last five years or so I don't even look for the distribution package. I look for the container image. I look for the curl -X | bash script. I, personally, trust my distribution more than these other methods, but the software isn't there.

Looking at the landscape I hope that Debian sees this as something they must figure out how to do rather than rejecting these other software packaging methods as "bad" or as "not fitting our constraints." It's clear from the explosion of modern packaging systems that what developers want is the ability to vendor and choose the libraries they want to use for themselves. I know this has often been my biggest impediment to packaging software for any particular distribution too: That I only have the given choices of libraries.

As for my suggestion about actually solving the vendoring into packages issue, I would love to see the dependencies downloading into the source build and patches sent upstream that allow npm to work in an offline mode. Once you have the files locked in place as part of your source then the distro can work with the major package managers to extract the data that Debian needs. Right now there are clearly proprietary tools to do this and automatically scan for vulnerabilities. It would be nice to see them pulled into a larger free software effort.

Debian discusses vendoring—again

Posted Jan 15, 2021 14:49 UTC (Fri) by LibreTan (guest, #144205) [Link] (2 responses)

I feel that one of the possible solutions to this might be as follows:

Debian can only provide LTS for those Packages which provides LTS as upstream.

Example:
Firefox provides ESR so include it in Debian Stable release.

For all other software which does not provide LTS upstream it should work as OSTree layering only or work through Flatpak only.

If upstream is not providing LTS for their software then how can Debian?

Debian discusses vendoring—again

Posted Jan 16, 2021 3:44 UTC (Sat) by foom (subscriber, #14868) [Link]

I bet the vast majority of software already in Debian does not provide any LTS support, whatsoever.

Now, there won't be CVEs for most issues that were fixed only in main devhead in most software, because nobody is really looking closely enough.

But in the rare case that there is such a CVE, generally Debian would just have to backport the patch.

Debian discusses vendoring—again

Posted Feb 9, 2021 6:22 UTC (Tue) by plugwash (subscriber, #29694) [Link]

The problems I see are.

1. Many upstream's idea of "LTS" is far shorter than Debian's idea of regular support.
2. While some projects may be on top of security issues in their dependencies I would wager the majority are not.

For Firefox they have resorted to moving to new upstream "LTS" release series within stable releases of the distro, that it just about tolerable for an end-user app like Firefox but it's really not reasonble for things that are key infrastructure components (and even for firefox it's problematic because firefox updates force rustc updates...........)

Debian discusses vendoring—again

Posted Jan 22, 2021 1:08 UTC (Fri) by nix (subscriber, #2304) [Link]

In the Rust ecosystem, Cargo can already do exactly this -- a feature that was added because its first major use case, Firefox, needed it. (Firefox has a *lot* of vendored-in Rust packages, and it wanted to a) use Cargo like everything else, b) build without downloading stuff and c) not have to maintain this massive collection of vendored-in packages by hand.)

Debian discusses vendoring—again

Posted Jan 22, 2021 1:57 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

> As for my suggestion about actually solving the vendoring into packages issue, I would love to see the dependencies downloading into the source build and patches sent upstream that allow npm to work in an offline mode.
Go supports this ('go mod download'). We use it to optimize our Docker builds.

Another solution might a Debian-hosted third-party package repository mirror. I know that people are going to attack me with pitchforks, but maybe adopting third-party package managers is actually a better idea than trying to fit everything into deb-src?