DNF 3: better performance and a move to C++

By Jake Edge
March 28, 2018

It has only been a few years since DNF replaced Yum as the default Fedora package-management tool; that was done for Fedora 22 in 2015, though DNF had been available for several earlier Fedora releases. Since that time, DNF development has proceeded; it started a move from Python/C to all C in 2016 and has made multiple releases over the years. From an outsider's perspective, no major changes seem necessary, which makes the announcement of DNF 3, and a move to C++, a bit surprising to some.

For many years, Yum was the package-management front-end for many RPM-based distributions, including Fedora and RHEL. But it suffered from poor performance and high memory use; part of that was attributed to its iterative dependency solver. DNF was meant to fix those problems. It uses the libsolv dependency resolution library developed by openSUSE, by way of the hawkey library.

Though it wasn't a perfect drop-in replacement for Yum, DNF did replace it. But, even though DNF performed better, often much better, than its predecessor, the project continued to focus on making it faster. Ultimately, that's a large part of the reasons behind DNF 3.

Since the switch to DNF, it has steadily improved performance-wise, while maintaining the features and commands that users have been accustomed to since way back in the Yum days (just switching from "yum" to "dnf" on the command line). Under the covers, though, things were shifting around: hawkey got subsumed into libhif, which turned into libdnf. It is libdnf that is the main focus of the upcoming changes; DNF itself is envisioned as a small wrapper around the library.

In the announcement on the DNF blog, the focus on performance is evident. There is a graph comparing the in-development version of libdnf with the versions that shipped in Fedora 26 and 27. In all five of the queries tested, the new code is faster; in some cases massively so (from 77 seconds to 2.5 and 40 to 2.4, for example). Beyond better performance, the DNF 3 effort is also targeting consistent behavior between various package-management components (between DNF and PackageKit, in particular), a better defined and maintained API, and compatibility ("as much as possible") with the existing C and Python APIs. For a number of reasons, the project believes that switching to C++ will help it achieve those goals.

DNF project leader Daniel Mach posted a note about the announcement to the Fedora devel mailing list on March 22. It will probably not come as a surprise that some of the reaction was about the language choice. Matěj Cepl said:

When switching the programming [language] than I would think there are some better C-successors than C++, namely Rust? Mad rush of giving up on 46 years old language and switching to one which is just 33 years old seems a bit bizarre to me.

Using Rust would require LLVM, though; that's a step too far for Neal Gompa:

I'm okay with not dealing with LLVM for my system package manager, thank you very much. I'd be more open to Rust if Rust also could be built with GCC, and thus supported across literally everything, but no one is investing in that effort.

Gompa was also concerned about the difficulty of programming in Rust versus C++. Martin Kolman thinks C++ makes a better choice, rather than simply following the most-recent trend: "it is not bad to be a bit conservative when core system components are concerned". Given that the project has already made its choice and started development down that path, C++ is basically a fait accompli at this point.

Marcin Juszkiewicz raised the issue of sharing repository metadata between users, which is something of a longtime sore point for some. If a regular user runs DNF to query about the availability or version of some package, DNF will go off and download the latest metadata into /tmp. To some, that's a big waste of time, to others, who might not have a speedy internet connection, it is more or less intolerable. Since there is already a copy of the metadata available (root's version), it should be used by everyone, he said.

The downside of that idea, of course, is that the metadata in /var/cache/dnf may be out of date; regular users should not be able to update the system copy, so there are worries about giving outdated responses to these queries. Tom Hughes asked if Juszkiewicz had a proposal on how to share the metadata "in a secure way". He also wondered what should happen if the metadata was out of date.

Juszkiewicz had a fairly straightforward thought about how a regular user would interact with the metadata: "Use. Not download. And inform (like always) how old that data is." As he and others in the thread pointed out, Debian's APT separates the metadata update from the query functionality, while DNF (like Yum before it) combines the two; if a DNF query finds out-of-date metadata, it seeks to remedy that before proceeding. Several thread participants seemed to prefer the APT scheme (or at least some way to share read-only access to the metadata). It is not at all clear what, if anything, will happen with that, but it is an annoyance, at least, for some.

Some other thoughts and enhancements were batted around a bit, but the overall impression is one of muted approval of the idea. Most probably just hope that the changes don't really affect them at all, other than seeing faster query responses. That is probably the DNF developers' hope as well; package managers are the kinds of programs that can be mostly ignored—so long as they work.

to post comments

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 2:20 UTC (Thu) by jhoblitt (subscriber, #77733) [Link] (16 responses)

Does this mean that the `dnf` command itself is going to be python free? That would be a step towards smaller minimum docker images that need a python interpreter. It would also mean that a broken system python doesn't brick the package manger (this has happened to me...).

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 3:22 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (15 responses)

Have you looked into microdnf for the container use case? Might not need this in the future.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 15:15 UTC (Thu) by jhoblitt (subscriber, #77733) [Link] (3 responses)

https://github.com/rpm-software-management/microdnf ? I was completely unaware of it. Are there any public docker images using it? A quick search turned up `registry.access.redhat.com/rhel7-atomic` but I don't have a rhel subscription.

My team has been moving towards using `alpine` for minimal containers but being musl based, it often doesn't play nice with 3rd party binaries. There is a `glibc` `alpine` variant I've been experimenting with.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 15:19 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

In addition to RHEL, CentOS & Fedora images etc should be.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 16:26 UTC (Thu) by zdzichu (subscriber, #17118) [Link] (1 responses)

fedora-minimal is.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 17:05 UTC (Thu) by jhoblitt (subscriber, #77733) [Link]

It took me awhile to find a `fedora-minimal` but it is certainly a lot smaller that the official docker hub images:

registry.fedoraproject.org/fedora-minimal:latest -- 115MB
docker.io/fedora:27 -- 235MB

DNF 3: better performance and a move to C++

Posted Apr 3, 2018 7:58 UTC (Tue) by dmach (guest, #123441) [Link] (10 responses)

Microdnf is definitely an option if you're looking for minimal system footprint.
Unfortunately it sometimes behaves differently than DNF.
That's why we're currently consolidating the whole software management stack.

Sooner or later, microdnf will gain most of the DNF features through libdnf.
What's not clear yet is how to deal with plugins, which are written in Python.

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 17:47 UTC (Wed) by hkario (subscriber, #94864) [Link] (1 responses)

wouldn't graceful failure handling be possible in case module load fails?

also, I don't think removing support for python plugins is a good idea: ease of writing them lowers the barrier to entry and tools like Gimp are only better for support of them

DNF 3: better performance and a move to C++

Posted Apr 6, 2018 6:16 UTC (Fri) by dmach (guest, #123441) [Link]

I'm not sure about the graceful failure handling.
Sometimes it's better to report an error than ignore it and exit with 0.
That could make a false assumption that everything works perfectly.

About the plugins support - we do *not* have any plans to remove them from Python DNF.
On the other hand, it would be great to support plugins directly in libdnf,
in order to make them available to all tools based on the library.
We can't link libdnf with Python to keep the footprint and deps small.
Once the software management stack consolidation is finished,
we'll definitely look into this. Now it's early to say how the solution will look like.

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 18:04 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

Why should a tool like DNF even have plugin support?

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 18:16 UTC (Wed) by jhoblitt (subscriber, #77733) [Link] (6 responses)

Why shouldn't it? YUM/DNF plugins are widely used.

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 20:07 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

Debian does it just fine without plugins. Simply provide helper executables that change configuration files as required (see: fastermirror). There's really no reason for a package manager to have in-process plugins.

And some plugins (like langpack) should just be provided as core functionality.

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 20:31 UTC (Wed) by jhoblitt (subscriber, #77733) [Link] (4 responses)

You can make the same modularity argument about just about any software. Why does a kernel need loadable modules? Why does `dlopen` need to exist? Why does an operating system need a package manager? Because example X does without it.

I gather that your satisfied with wrapper scripts like `apt-fast` and similar but it is arguably a poor experience for inexperienced end users. It also places more demands on project maintainers.

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 21:13 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

> You can make the same modularity argument about just about any software.
Absolutely.

> Why does a kernel need loadable modules?
By the same token, the kernel doesn't have pluggable CPU schedulers (and there was a huge flame war about that, just ask Con Kolivas).

> I gather that your satisfied with wrapper scripts like `apt-fast` and similar but it is arguably a poor experience for inexperienced end users. It also places more demands on project maintainers.
Why should there be "apt-fast"?!? The default apt should already be fast without any plugins.

Package retrieval is basically a solved problem, the set of plugins is stable for ages. Integrate them all into the core and forget about it.

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 22:46 UTC (Wed) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

> Package retrieval is basically a solved problem, the set of plugins is stable for ages. Integrate them all into the core and forget about it.

Not quite. There are some which are stable but there is plenty of experimentation in this area of packaging. A recent example is the modularity project implemented via a plugin. There are also things that apply in specific circumstances (snapper) or only one distro among the many which use dnf now (subscription-manager plugin). Several of the plugins have been integrated into core when it made sense (delta rpms in yum used to be handled via a plugin originally) but not all of them fit that way.

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 23:03 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

That's fine. Just fork a package and experiment with it. Internal modularity in the source code is, of course, important.

DNF 3: better performance and a move to C++

Posted Apr 5, 2018 0:17 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

> That's fine. Just fork a package and experiment with it. Internal modularity in the source code is, of course, important.

That is painful compared to plugins and it only covers one of the use cases I was talking about

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 3:28 UTC (Thu) by marcH (subscriber, #57642) [Link] (11 responses)

> I'm okay with not dealing with LLVM for my system package manager, thank you very much.

I had a quick look at the original thread and couldn't find any elaboration. Anyone with any clue about what the concerns are? I could imagine something like "portability matters" but it sounds tied to LLVM more specifically?

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 5:16 UTC (Thu) by edomaur (subscriber, #14520) [Link]

Yeah, I'd verymuch would like more information about that too...

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 6:07 UTC (Thu) by Lionel_Debroux (subscriber, #30014) [Link] (3 responses)

Likewise, I think it would be great if he could elaborate on his concerns.

Indeed, LLVM has both newer design and more permissive licensing, which enable the framework to get more, not less, popular over time. The Google and Apple ecosystems, which carry huge weight, have both switched to LLVM - arguably for licensing reasons, but they wouldn't have done so if it hadn't become a decent, and even good, option... Even MS packages Clang for non-Windows in modern versions of VS. Features like CFI aren't available in GCC, and GCC's sanitizers are a step below upstream's sanitizers, AFAIK.

That's not to say that the LLVM family is perfect. For a start, it's another fact that LLVM supports fewer ISAs than GCC does, but equally obviously, it supports all of the ones which really matter to the main traditional Linux distros. For these distros (and core components thereof), resisting the use of LLVM, for fuzzy / unspecified reasons, probably isn't a wise choice for the mid- and long-term... perhaps especially so for Fedora, which aims at being on the leading / bleeding edge ("First").

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 6:14 UTC (Thu) by magfr (subscriber, #16052) [Link] (1 responses)

How well developed is the LLVM support for S/390?

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 8:16 UTC (Thu) by josh (subscriber, #17465) [Link]

Seems to work, and Rust supports s390.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 23:53 UTC (Thu) by lsl (guest, #86508) [Link]

> Likewise, I think it would be great if he could elaborate on his concerns.

I'd start with the level of care and attention received by the GCC and LLVM packages in Fedora. GCC developers are very active about tracking down issues that tend to occur when a new GCC release hits Fedora and the huge package collection gets rebuilt with it. This kind of symbiotic relationship with the upstream currently doesn't exist for LLVM so it's probably not a good idea to rely on it for super-critical distribution infrastructure until this has changed. Without someone suitably-motivated driving this, it won't happen though.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 12:00 UTC (Thu) by epa (subscriber, #39769) [Link] (5 responses)

Isn't it just that DNF is a core system package, and that would make LLVM a hard requirement to build the core system? It would be better to keep dependencies and the build system simpler for the most essential packages. Otherwise every port of Fedora (or other libdnf-using distros) to a new architecture would require having two working compilers, GCC and LLVM.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 13:17 UTC (Thu) by iq-0 (subscriber, #36655) [Link] (4 responses)

In that case LLVM wouldn't be the problem, but the use of "yet another programming language" (aka Rust). LLVM only limits (or enables, depending on how you look at it) Rust's platform support.

Rust being build on LLVM is just an implementation detail and Rust doesn't consider LLVM to be the only backend for native code generation, even though it's likely to be the main one for the foreseeable future.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 21:28 UTC (Thu) by epa (subscriber, #39769) [Link] (3 responses)

Have I misunderstood? Rust only works with LLVM. Rust programs can't be built with GCC as the backend (work on that is still experimental).

That's not a restriction of Rust-the-language, but as Rust-the-programming-environment it certainly is. Some future version of Rust might work with GCC, just as some future Fedora release might build the kernel and everything else with LLVM. But right now, using Rust would mean adding a dependency on LLVM, and hence two separate compilers in the base system.

Or are you saying that LLVM is just an implementation choice, and it would be quite possible to program DNF in Rust without introducing a dependency on LLVM? If so, how?

DNF 3: better performance and a move to C++

Posted Mar 30, 2018 6:50 UTC (Fri) by iq-0 (subscriber, #36655) [Link] (2 responses)

> Have I misunderstood? Rust only works with LLVM. Rust programs can't be built with GCC as the backend (work on that is still experimental).

No you understood correctly.

You stated that "would make LLVM a hard requirement to build the core system" as their possible motivation. But that is just backwards. LLVM is just a dependency of Rust the language. So either it's problematic of adding yet another language to the core system build requirements, which is a perfectly valid reason for them to express, or you state that you won't support some language because of some specific reason.

But saying it's "because of LLVM" is like stating that you won't use a programming language because it depends on tcmalloc instead of glibc malloc or because it uses yacc instead of bison.

> Or are you saying that LLVM is just an implementation choice

Yes that's what I'm saying.

> and it would be quite possible to program DNF in Rust without introducing a dependency on LLVM? If so, how?

And that is not what I meant.

Rust can create fully standalone binaries just fine, so there should not be a runtime dependency on LLVM.

Rust only introduces LLVM as a build dependency and even in that case someone created 'mrustc' which IIRC acts as a transpiler to C and could even be used to break that hard dependency for obscure platforms. I read somewhere that the Debian Rust maintainers were considering that for some of the more obscure platforms that don't have native LLVM support.

DNF 3: better performance and a move to C++

Posted Mar 30, 2018 13:10 UTC (Fri) by khim (subscriber, #9252) [Link] (1 responses)

> But saying it's "because of LLVM" is like stating that you won't use a programming language because it depends on tcmalloc instead of glibc malloc or because it uses yacc instead of bison.

If programming language depends on custom allocator and does not play well with glibc one then it's enough reason to avoid it. C# and Java come to mind.

mrustc sounds an interesting aolution for that problem, though. How well is it supported in Rust community?

DNF 3: better performance and a move to C++

Posted Apr 5, 2018 8:31 UTC (Thu) by HelloWorld (guest, #56129) [Link]

> If programming language depends on custom allocator and does not play well with glibc one then it's enough reason to avoid it.
Could you explain why?

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 3:37 UTC (Thu) by marcH (subscriber, #57642) [Link] (27 responses)

> Mad rush of giving up on 46 years old language and switching to one which is just 33 years old seems a bit bizarre to me.

I keep hearing that C++ *can* be used in a safe, non-C way that can avoid corruption, crashes and security issues just like modern languages, it's "just the matter of sticking to the right subset", correct? Which of course almost never happens because there's not even a compiler flag to tell you whenever you leave that safe subset...

Would a team used to C be able to stick to such a safe C++ subset? Are they even interested in such an effort?

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 4:16 UTC (Thu) by roc (subscriber, #30627) [Link] (7 responses)

And no-one can say exactly what that safe C++ subset consists of.

Oh well. Perhaps safety isn't a big concern for dnf because it doesn't run with privileges and/or doesn't process untrusted input...

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 22:19 UTC (Thu) by k8to (guest, #15413) [Link] (5 responses)

Hmm, dnf doesn't run with privileges? How does it perform the installs?
This might be dnf 101, but I'm not finding much about dnf having privilege separation on the interwebs.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 22:26 UTC (Thu) by roc (subscriber, #30627) [Link]

Forgot the <sarcasm>, sorry.

DNF 3: better performance and a move to C++

Posted Mar 30, 2018 6:22 UTC (Fri) by epa (subscriber, #39769) [Link] (3 responses)

Non-privileged package management is something I have often missed. In Fedora an ordinary user can install packages from the Fedora repositories on the system, which is certainly better than having to know the root password to do it, but there is no way to install an RPM package in your own home directory and have it interoperate properly with system-installed packages. Fixed paths relative to the filesystem root are just too ingrained.

DNF 3: better performance and a move to C++

Posted Mar 30, 2018 6:40 UTC (Fri) by karkhaz (subscriber, #99844) [Link] (2 responses)

This is something language-specific package managers have done really well compared to distribution ones, IMO. `pip install --user foo` has exactly the same effect for you as `sudo pip install foo`, but nobody else sees any difference. But with distro packages, there's no option to do this; if you don't have root, then you typically have to get the source and build the program yourself.

Slightly tangential, but I always thought it would have been good to have a replica of the Filesystem Hierarchy Standard in each user's home directory (or perhaps in a dotfile under the home directory), i.e. every user gets ~/.usr/bin, ~/.usr/lib, ~/.usr/var, etc. I have no idea why the XDG standard used weird names when there are already perfectly standard names; they put several things under ~/.local (bin, lib, share), but there's also the ~/.config directory which could perfectly well be called ~/.local/etc, and ~/.cache could be ~/.local/var/cache or something.

Having users' home directories be that uniform might make distro-supported user-installable packages a lot easier implementation-wise, too!

DNF 3: better performance and a move to C++

Posted Apr 1, 2018 17:38 UTC (Sun) by epa (subscriber, #39769) [Link]

The language-specific package manager does let you install a per-user copy, but AFAIK it doesn't have a straightforward way to promote a single-user installation to a system-wide one. I'd like to be prompted 'package foo-0.5.2 is now installed globally; remove your local foo-0.5.1?'.

DNF 3: better performance and a move to C++

Posted Apr 5, 2018 8:40 UTC (Thu) by HelloWorld (guest, #56129) [Link]

There is a distro that allows this: NixOS.

DNF 3: better performance and a move to C++

Posted Apr 5, 2018 7:39 UTC (Thu) by HelloWorld (guest, #56129) [Link]

> And no-one can say exactly what that safe C++ subset consists of.
Actually Bjarne Stroustrup and Herb Sutter are specifying that right now, the effort goes by the name of C++ Core Guidelines. It can be found on GitHub.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 5:05 UTC (Thu) by ncm (guest, #165) [Link] (2 responses)

The safe subset is don't use mad shite inherited from C, like raw pointer arithmetic. When you compare an int to a size_t, use !=, not <.

But you still have integer overflow, in C++ (as in Rust) to watch out for. And be sure to constrain your array and vector indices.

The main thing is, have fun! If coding in modern C++ isn't fun, you are doing it wrong.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 5:37 UTC (Thu) by roc (subscriber, #30627) [Link] (1 responses)

In Rust debug builds integer overflow triggers a panic.

Sticking to "modern C++" does not necessarily avoid safety issues. Sometimes it makes them worse. See https://github.com/isocpp/CppCoreGuidelines/issues/1038 for example.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 22:21 UTC (Thu) by k8to (guest, #15413) [Link]

It seems like a shame that arranging for behavior on overflow is so obscure in C and C++ programming circles, still. If C++ developers regularly arranged for known behavior and decided on policy for it, a whole class of bugs would be removed in that environment.

Certainly Rust having a sane default that is well-known is a significant improvement.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 6:54 UTC (Thu) by rvfh (guest, #31018) [Link] (5 responses)

I am using C++11 and have no malloc/free nor new/delete in my code at all. All is managed using either unique or shared pointers.
In the same way, C++ always to not lock/unlock a mutex, but take an object that locks at construction and unlocks at destruction.
This allows for rather safe (as in: no deadlock, no memory leak, no dangling pointers) code already!
And of course there is much more...

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 8:15 UTC (Thu) by andresfreund (subscriber, #69562) [Link] (1 responses)

How does this guarantee freedom of deadlocks?

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 8:51 UTC (Thu) by rvfh (guest, #31018) [Link]

What it does guarantee is that you won't forget to unlock...

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 11:58 UTC (Thu) by roc (subscriber, #30627) [Link] (2 responses)

C++ code is still vulnerable to dangling references even if it uses no raw pointers. I mentioned a string_view example up above. Other common bugs occur where you have getter methods returning references to class members; it's easy for these references to accidentally outlive the object.

Using shared pointers a lot can be really slow because of all the atomic operations you have to perform, even if you're traversing what should be a read-only data structure.

Using RAII to ensure mutex locks are paired with unlocks is a step up from raw pthread_mutex_lock. But how do you ensure that the right mutexes are held while you access the data they protect? Rust ensures this. C++ frameworks typically don't.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 15:13 UTC (Thu) by kpfleming (subscriber, #23250) [Link] (1 responses)

Unfortunately the example you posted above did in fact use a raw pointer, it was just cleverly disguised :-)

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 21:22 UTC (Thu) by roc (subscriber, #30627) [Link]

Well sure, but once you admit that references and 'this' are just cleverly disguised raw pointers, you have no hope of getting away from them in C++.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 12:06 UTC (Thu) by epa (subscriber, #39769) [Link] (2 responses)

This is the effort to make a safer subset of C++ that can be checked at compile time: http://isocpp.github.io/CppCoreGuidelines/

They say 'the rules are designed to be supported by an analysis tool' but I don't know whether there is currently such a tool.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 21:26 UTC (Thu) by roc (subscriber, #30627) [Link] (1 responses)

That effort is moribund. The "Lifetime Safety" document in the C++ Core Guidelines is far from complete and has not been updated in any way since December 2015.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 21:29 UTC (Thu) by roc (subscriber, #30627) [Link]

Correction: if you consider the Core Guidelines as a whole, that's definitely not moribund. However, without lifetime safety analysis it leaves many gaping safety holes, and in some cases is actually a safety regression, not an improvement.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 14:24 UTC (Thu) by iq-0 (subscriber, #36655) [Link] (3 responses)

> Would a team used to C be able to stick to such a safe C++ subset? Are they even interested in such an effort?

For the most part: yes. Wrapping your data structures inside safe C++ often makes them more ergonomic and leads to less and cleaner code. The incentive is pretty strong.

> I keep hearing that C++ *can* be used in a safe, non-C way that can avoid corruption, crashes and security issues just like modern languages

This one is a bit tougher as there are 3 separate statements:
- avoid corruption
- avoid crashes
- avoid security issues

Basic corruption avoidance is greatly improved when using unique and shared smart pointers in C++. But these features only help reduce common occurrences of corruption through manual memory management. They won't fix all use after free bugs through references or guard against corruption through data races when using multi threading. Garbage collection will help guard against the first and stricter data management or alternative threading models help guard against the second.

Avoiding crashes is often tied to avoiding corruption. But a significant part of avoiding crashes is also in reducing possible API abuse. Stricter type systems will often help reduce this even further. The fact that C++ makes using C so easy also keeps many parts from still depending on C APIs, which will not help guard against basic unintended API abuse.

Avoiding security issues goes a long way beyond basic corruption avoidance and crash reduction. Most security issues aren't really down to the programming language itself (though certain aspects around basic datastructures help, like eg. having collision resistent hash tables), but have much more to do with the ecosystem surrounding it: How do you handle dependencies? How mature and/or well maintained are your dependencies?

C++ is definitely a step up from C in these aspects, but most dynamic languages and more modern languages will be a lot better than C++ in these areas.

But there are a lot more aspects being considered in this case. From resource usage (bye bye most dynamic languages), runtime dependencies (Java, Mono) to good integration with the current C code base (hello C++ and Rust) to familiarity of the current maintainers (bye Rust?). From a purely technical standpoint I'd personally recommend Rust, but from a project standpoint I can see where the choice for C++ would come from. Maintainers and contributers are a projects most valuable resource, even more valuable than all the code that was, is or will be.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 19:15 UTC (Thu) by marcH (subscriber, #57642) [Link] (2 responses)

> From a purely technical standpoint I'd personally recommend Rust, but from a project standpoint I can see where the choice for C++ would come from. Maintainers and contributers are a projects most valuable resource, even more valuable than all the code that was, is or will be.

Makes sense, thx. There was also this good point on the list: "I think it is not bad to be a bit conservative when core system components are concerned." etc.

I'm surprised Go wasn't even mentioned. It seems very successful in the not too distant containers sphere which seems very sensitive to performance too. Does a package manager really need to micro-manage memory? https://blog.golang.org/go15gc

DNF 3: better performance and a move to C++

Posted Mar 30, 2018 7:12 UTC (Fri) by iq-0 (subscriber, #36655) [Link] (1 responses)

> I'm surprised Go wasn't even mentioned. It seems very successful in the not too distant containers sphere which seems very sensitive to performance too. Does a package manager really need to micro-manage memory? https://blog.golang.org/go15gc

I expect that there are multiple reasons for not being considered. I guess some of these are:
- The languages are very different. They'd have to fully rewrite entire chunks instead of a more gradual migration.
- The use of garbage collection might be a reason why they switched away from Python. A good garbage collector often exhibits a higher baseline RSS, which might be a problem on resource constrained systems.
- Go <-> C interop isn't considered to be very efficient. So, depending on which parts they'd want to rewrite first, there is a good chance that performance will suffer before becoming better.
- Upstream Go has (very) limited platform support. Gccgo can be used to work around this, but that would mean a workaround for a number of primary supported platforms.

And there are of course the philosophical differences, like to C++ gives you things like a rich type system, templating and exception based error handling. Go goes almost completely the other direction: simpeler type system, no templating, no metaprogramming and written out error handling. Both have their pros and cons.

But the project was probably looking for a way to incrementally make the current code base less cumbersome/easier. Switching to Go would not be the quickest way to get there.

DNF 3: better performance and a move to C++

Posted Apr 5, 2018 0:01 UTC (Thu) by lsl (guest, #86508) [Link]

> - Upstream Go has (very) limited platform support. Gccgo can be used to work around this, but that would mean a workaround for a number of primary supported platforms.

Which ones? The gc compiler (the one you get from golang.org) supports AArch64, AMD64, ARM, MIPS, PPC64 (BE and LE), S390x and i386 in its main development branch. There are further (incomplete) ports available that aren't merged upstream yet (SPARC, RISC-V).

That should cover all machines supported by Fedora (and then some). Maybe you looked at the binary distributions? Those are only provided for a subset. All arches are required to have CI builders running the testsuite on every commit, though. Results are available from https://build.golang.org.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 14:35 UTC (Thu) by cry_regarder (subscriber, #50545) [Link] (2 responses)

MarcH is a classic flame starter on LWN. Please consider before you engage with them.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 14:45 UTC (Thu) by marcH (subscriber, #57642) [Link] (1 responses)

[reference required]

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 22:36 UTC (Thu) by k8to (guest, #15413) [Link]

Perhaps this is a case of insensitive indexing. We get a lot of flames in march.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 6:56 UTC (Thu) by meyert (subscriber, #32097) [Link] (13 responses)

Yes, CPU perfomance ist of course the main problem of an tool that is mostly IO bound....
And not the quality of the tool and the plugins. E.g. I did hit bugs with dnf system upgrade for so many different releases!

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 9:15 UTC (Thu) by muep (subscriber, #86754) [Link]

In my experience, DNF seems somewhat CPU intensive when compared to other popular package managers. Not enough to be a problem on modern desktops or laptops. However, on older laptops or ARM devices, having a less CPU intensive DNF would make package management noticeably more enjoyable. I have no idea if the major bottlenecks are in DNF or in the libraries that it uses, but this still looks like something to look forward to.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 16:36 UTC (Thu) by antiphase (subscriber, #111993) [Link] (10 responses)

Fixing bugs is boring. Rewriting from scratch in a new language is much more interesting, and think how much benefit people will see from being able to install more packages per second.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 17:01 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (9 responses)

> Fixing bugs is boring. Rewriting from scratch in a new language is much more interesting

Moving from C to C++ can and does fix a number of bugs, improve security and performance. Many projects including say GCC has done it for similar reasons.

> think how much benefit people will see from being able to install more packages per second.

People do benefit from faster installations and upgrades.

DNF 3: better performance and a move to C++

Posted Mar 31, 2018 0:21 UTC (Sat) by Twirrim (subscriber, #122505) [Link] (1 responses)

> People do benefit from faster installations and upgrades.

I'd be really curious where shaving a few seconds off on package upgrades or installations is that advantageous. Yes speeding up yum is a nicer user experience, but this seems to be well into the territory of diminishing returns.

DNF 3: better performance and a move to C++

Posted Mar 31, 2018 2:16 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link]

> I'd be really curious where shaving a few seconds off on package upgrades or installations is that advantageous.

Two major places atleast, initial installation and full release upgrades. Anaconda, the Fedora installer doesn't support upgrades anymore. It is all handled through the dnf system upgrade plugin and for each of these use cases, the performance differences are substantial.

There are plenty of other places including package builds in the buildsystem or QA test systems which are done in a clean chroot, where a few seconds can add up over time.

DNF 3: better performance and a move to C++

Posted Mar 31, 2018 17:41 UTC (Sat) by HenrikH (subscriber, #31152) [Link] (6 responses)

How does moving from C to C++ improve performance?

DNF 3: better performance and a move to C++

Posted Mar 31, 2018 18:07 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link] (4 responses)

> How does moving from C to C++ improve performance?

It certainly can but I think you are mixing up two things. Read the announcement linked in the article. Dnf 3 includes both performance work and a move from C to C++. They are not necessarily related.

DNF 3: better performance and a move to C++

Posted Mar 31, 2018 18:23 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

Addendum, if the question was really, can C++ improve performance over C? the answer is of course, it can

http://benchmarksgame.alioth.debian.org/u64q/cpp.html

It is not uniformally better but a lot of C programs end up duplicating parts of C++ poorly. A move to C++ can help in such circumstances.

DNF 3: better performance and a move to C++

Posted Apr 1, 2018 17:05 UTC (Sun) by HenrikH (subscriber, #31152) [Link] (1 responses)

Well of course C++ will be better if the C programmer badly duplicates parts of C++. I was more worried that compilers these days would produce better code in the C++ path than in the C one.

DNF 3: better performance and a move to C++

Posted Apr 2, 2018 13:24 UTC (Mon) by ncm (guest, #165) [Link]

It is quite routine for a C++ program to be faster than a C effort in the same space. The difference is ultimately traceable to C++ being better at encapsulating logic in libraries. The consequence is that there is a lot of power packaged in libraries now that any program can draw upon, that would simply be too much work to bother with reproducing at the call site, in C.

That power includes application of algorithms that are too finicky to code for just one use; performance tuning, likewise; and data structures, again. C has inlining now, but C++ is able to make much more effective use of it, and the optimizers understand that use, so can squeeze out what used to be called "abstraction overhead" (when there usually was some).

Furthermore: because it is easier to work with values directly in C++, rather than by reference, the optimizer has more to work with, not bothered by potential aliasing.

DNF 3: better performance and a move to C++

Posted Apr 1, 2018 16:58 UTC (Sun) by HenrikH (subscriber, #31152) [Link]

I'm not mixing anything up, it was a direct question to your post where you wrote "Moving from C to C++ can and does fix a number of bugs, improve security and performance.", you clearly say that moving from C to C++ can and does improve performance.

DNF 3: better performance and a move to C++

Posted Apr 7, 2018 9:57 UTC (Sat) by HelloWorld (guest, #56129) [Link]

Templates will generate code specialized for different types which can speed things up immensely. Try sorting a list of doubles with std::sort and qsort. The latter will be slower.

There's also a more subtle reason: C++ allows you to abstract over things that C doesn't, hence allowing better code re-use. And you're more likely to use good algorithms and data structures if somebody else built them generically and put them into an easy-to-use library than if you have to build them yourself.

DNF 3: better performance and a move to C++

Posted Apr 3, 2018 10:19 UTC (Tue) by jarmar (guest, #103679) [Link]

I was unable to search the package repositories for a certain package because dnf required too much RAM to do so (this was on a VPS with 512M RAM), so its poor performance affected me directly.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 18:13 UTC (Thu) by clugstj (subscriber, #4020) [Link] (30 responses)

"giving up on 46 years old language and switching to one which is just 33 years old seems a bit bizarre to me"

So the age of a language automatically makes it a poor choice?

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 19:04 UTC (Thu) by marcH (subscriber, #57642) [Link]

> So the age of a language automatically makes it a poor choice?

Yes _when_ said age goes back to the pre-internet era when safety, security and reliability were very distant concerns if at all.

Of course C++ has continued to evolve since, but:
- it doesn't really seem to provide the tools to *enforce* its users to evolve with it
- also based on comments above it doesn't seem it went as far as newer languages.

DNF 3: better performance and a move to C++

Posted Mar 29, 2018 21:33 UTC (Thu) by roc (subscriber, #30627) [Link]

We've learned a lot about programming languages in 33 years, and many newer languages benefit from those advances.

Of course C++ has evolved to absorb some of those lessons too, modulo backward compatibility constraints. But using 33-year-old C++ would be truly daft.

DNF 3: better performance and a move to C++

Posted Mar 30, 2018 19:41 UTC (Fri) by flussence (guest, #85566) [Link] (27 responses)

And calling it a 33 year old language deliberately ignores the fact that -std=c++14 is the default in GCC 7, with c++17 around the corner. Languages evolve, prejudices against them had better learn to keep up or you'll just sound like a COBOL/Solaris fanboy.

From where I'm standing, ${insufferable trendy web2.0 flavour of the month} has 33 years of tooling to catch up on. (Hint: start with distcc and ccache.)

DNF 3: better performance and a move to C++

Posted Apr 3, 2018 20:12 UTC (Tue) by cpitrat (subscriber, #116459) [Link] (6 responses)

Funny that your hint is about tools to deal with the languages design flaws (compile time doesn't have to be an ossue, see golang for example)

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 1:27 UTC (Wed) by zlynx (guest, #2285) [Link] (3 responses)

That's because Go hardly optimizes at all. Just a little bit.

If you use precompiled headers and -O0 on GCC it is nearly as fast at compiling as Go is. GCC is pretty slow with -flto and -O3 but Go can't do that at all.

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 7:15 UTC (Wed) by cpitrat (subscriber, #116459) [Link] (2 responses)

That's not true, even with precompiled headers, no LTO and -O0 compiling hundreds of C++ files (5-10k sloc) usually take minutes. This even impacts the way you code (pimpl and other tricks for faster compilation). I don't know much of Go but I observed that building a project of the same size takes a few seconds.

DNF 3: better performance and a move to C++

Posted Apr 5, 2018 15:34 UTC (Thu) by zlynx (guest, #2285) [Link] (1 responses)

I just retried my project in debug mode. 642 C++ and header files. 150,000 lines of code.

> real 0m7.059s
> user 0m14.177s
> sys 0m2.159s

Here is github.com/coreos/etcd with go build -a -v. 1,620 Go files, 538,484 lines of code.

> real 0m11.980s
> user 0m53.516s
> sys 0m4.843s

Both of those were done on a 4c/8t CPU with 8 threads.

So yes, Go builds faster. But not by a whole lot. And a build time of "minutes" is not realistic.

DNF 3: better performance and a move to C++

Posted Apr 5, 2018 17:10 UTC (Thu) by excors (subscriber, #95769) [Link]

On the other hand, something like https://www.boost.org/doc/libs/1_66_0/libs/spirit/example... takes (on my reasonable CPU) 7 seconds to compile a single small source file with g++ -O0. Spirit is absurdly bad for this, but other libraries that make heavy use of fancy templates can be pretty expensive too, and it's an aspect of library design that is often completely ignored. (https://github.com/fmtlib/fmt#benchmarks is one of the few places I've seen compile time benchmarks given the same prominence as runtime benchmarks.)

That's certainly a manageable problem (you can design libraries in ways that don't involve enormous amounts of inline template code), but it takes a bit of knowledge and discipline to make sure you avoid dependencies that significantly hurt your project's compile times.

DNF 3: better performance and a move to C++

Posted Apr 4, 2018 21:21 UTC (Wed) by flussence (guest, #85566) [Link] (1 responses)

Fast compiles are there to woo the PHP crowd with its thirst for instant gratification - a bit like how both languages have unauthenticated-curl-pipe-sh functionality built right into the include keyword.

Go has its own coping mechanisms as a consequence - usually involving multiple nested containment OSes per app.

DNF 3: better performance and a move to C++

Posted Apr 5, 2018 20:51 UTC (Thu) by lsl (guest, #86508) [Link]

> a bit like how both languages have unauthenticated-curl-pipe-sh functionality built right into the include keyword.

Go, of course, does not have that. It's a popular misconception.

On the language level, the strings used with the import statement are just that: opaque strings that identify a package but are not assigned any further meaning. For namespacing reasons it makes sense to somehow coordinate who owns which package names. Instead of coming up with with some Go-specific name registry, the convention is to just piggy-back on the DNS.

The build tooling uses these strings for finding source code within your file system. So if you import "a/b/c" that means to use the module located at $GOPATH/src/a/b/c. This convention is basically doing the same job that is done by pkg-config and arrangement of files within /usr/include and /usr/lib in the C world. If you really want to, you can ignore the build system and just call the compiler and linker yourself, passing the appropriate files and command line flags (-L, etc.).

Neither of those (compiler/build system) will retrieve anything from the network or otherwise execute arbitrary code. The build system has a source code generation step (for calling out to things like yacc), but you need to invoke it explicitly (go generate), it's not invoked during normal compilation.

The only thing that does talk to the network is the 'go get' tool. It's for developers, to quickly check out a source repo in the right place. It just uses your local Git program (or Mercurial, ...) under the covers.

Do you see why that "unauthenticated-curl-pipe-sh" analogy doesn't make much sense? The only way that Go tools talk to the network is by invoking git. So unless you have a habit of cloning arbitrary git repos and piping their contents into a shell ...

DNF 3: better performance and a move to C++

Posted Apr 7, 2018 15:08 UTC (Sat) by HelloWorld (guest, #56129) [Link] (19 responses)

> From where I'm standing, ${insufferable trendy web2.0 flavour of the month} has 33 years of tooling to catch up on
Seriously? For the longest part of its history, the tooling for the C and C++ ecosystem has been complete and utter garbage. People were (and are) using insane crap like autotools and CMake until something like Meson came along. And it took ages until decent editors (with code completion, jump to definition, refactoring etc.) came along, and they were clearly inspired by other languages.

Anybody who thinks that tooling is a good reason to use C++ has never seen what good tooling looks like.

DNF 3: better performance and a move to C++

Posted Apr 7, 2018 15:30 UTC (Sat) by halla (subscriber, #14185) [Link] (18 responses)

It's weird how opinions spread... Just saying "cmake is insane crap" doesn't make it insane crap, and it is, of course nothing of the kind. But if people keep saying it, I guess it'll stick in some easily influenced people's minds, which would be a pity. CMake is fine.

CMake is leagues better than autotools, of course, and Meson or QBS or Scons or Waf or whatever "alternative" system is popular with people who cannot use CMake because KDE decided to use it in 2006, have about a decade of catch-up and acquiring complications to do before it can contend.

And if they survive that long, I guess people will call it "insane crap" as well. Every real-world build system is complicated.

DNF 3: better performance and a move to C++

Posted Apr 8, 2018 11:00 UTC (Sun) by HelloWorld (guest, #56129) [Link] (5 responses)

Just saying "cmake is insane crap" doesn't make it insane crap, and it is, of course nothing of the kind

Oh, it absolutely is. Its custom programming language alone is enough of a reason not to use it. You can't even do the most basic thing a function is supposed to do: return a value, for crying out loud! Instead, what people resort to is assigning values to global variables whose name is derived from the function's parameters. Functions can't have named parameters either, so what you do is you pass certain “keywords” (for lack of a better word) interleaved with other function arguments and hope that the values you're passing don't clash with the “keywords” defined by the function. Needless to say, it doesn't have any meaningful programming features, like real lists, dictionaries or, heaven forbid, first class functions. The worst part about it is that they claim this to be for “simplicity”, yet NO OTHER LANGUAGE does things like that, making the whole thing feel weird and awkward to just about anybody who's had any exposure to programming.

Every real-world build system is complicated.

Which is why you need a proper programming language, and not something insane like CMake.

Now, all this might be justifiable (although not really) if the end result was actually reasonably easy to use. So what does some simple task, like building a simple program with one library dependency, look like with CMake? Something like this:

project(hello-world C)
cmake_minimum_required(VERSION 2.6)
PkgConfig to detect GTK+ headers/library files
find_package(PkgConfig REQUIRED)
pkg_check_modules(GTK3 REQUIRED gtk+-3.0)
include_directories(${GTK3_INCLUDE_DIRS})
link_directories(${GTK3_LIBRARY_DIRS})
add_definitions(${GTK3_CFLAGS_OTHER})
add_executable(hello main.c)
target_link_libraries(hello ${GTK3_LIBRARIES})

This is garbage, why am I using a build system if I end up having to configure include directories, linker flags and locate source files and all that crap? It's the build system's job to do that! What does it look like in a language with at least somewhat sane tooling, like Scala and sbt?

libraryDependencies += "org.scalaz" %% "scalaz-core" % "7.2.21"

This is the complete build definition. Now you can keep pretending that CMake is somehow not insane and stupid. Meanwhile, I'll just sit here and laugh at you.

DNF 3: better performance and a move to C++

Posted Apr 8, 2018 20:29 UTC (Sun) by halla (subscriber, #14185) [Link] (4 responses)

"Oh, it absolutely is. "

No, it isn't.

DNF 3: better performance and a move to C++

Posted Apr 8, 2018 20:35 UTC (Sun) by HelloWorld (guest, #56129) [Link]

I rest my case.

DNF 3: better performance and a move to C++

Posted Apr 9, 2018 2:18 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

You're suffering from Stockholm syndrome.

CMake is crap. It's a slightly better crap than most other C++ build systems but it's still crap compared to modern build systems for other languages.

DNF 3: better performance and a move to C++

Posted Apr 9, 2018 2:37 UTC (Mon) by pizza (subscriber, #46) [Link] (1 responses)

Your comparison is as poignant as complaining that an exotic sports car doesn't work well for hauling loads of rock around the local quarry.

Different tools for different purposes; is that really so hard to comprehend?

DNF 3: better performance and a move to C++

Posted Apr 9, 2018 12:10 UTC (Mon) by HelloWorld (guest, #56129) [Link]

> Your comparison is as poignant as complaining that an exotic sports car doesn't work well for hauling loads of rock around the local quarry.
Since Cyberax didn't make a comparison, I guess you meant to respond to me.

> Different tools for different purposes
Here's my question? How exactly does the purpose of CMake – building C and C++ programs – necessitate a custom programming language for your build system that has “functions” that can't return values, doesn't have any useful data structures and is generally weird and quirky?
The answer is it doesn't, hence “different purposes” don't justify anything here.

DNF 3: better performance and a move to C++

Posted Apr 8, 2018 11:14 UTC (Sun) by HelloWorld (guest, #56129) [Link] (11 responses)

Oh, and one more thing: maybe you're right, maybe there is no way to make a simpler build system for C/C++ because building C/C++ programs is inherently more complicated. But you know what? It's just stupid to claim that C/C++ is better because there are complicated tools to solve problems that other languages don't have in the first place.

DNF 3: better performance and a move to C++

Posted Apr 8, 2018 12:07 UTC (Sun) by pizza (subscriber, #46) [Link] (10 responses)

> It's just stupid to claim that C/C++ is better because there are complicated tools to solve problems that other languages don't have in the first place.

s/solve problems/use capabilities/

Pretending otherwise is just willful ignorance.

DNF 3: better performance and a move to C++

Posted Apr 8, 2018 14:26 UTC (Sun) by HelloWorld (guest, #56129) [Link] (9 responses)

How is not having a module system in 2018 (and having to muck around with header files and the like as a consequence) a capability?? Sorry, I don't buy it. Most of the complexity of C/C++ build systems is entirely incidental, not essential.

DNF 3: better performance and a move to C++

Posted Apr 8, 2018 18:14 UTC (Sun) by pizza (subscriber, #46) [Link] (2 responses)

Every one of those "superior" languages/runtimes/etc are themselves implemented with a C/C++ codebase, which itself needs compiling, which in turn requires all sorts of complexities in the name of working on a variety of compilers and operating system/hardware platforms.

The real world is messy.

DNF 3: better performance and a move to C++

Posted Apr 8, 2018 19:34 UTC (Sun) by HelloWorld (guest, #56129) [Link]

> Every one of those "superior" languages/runtimes/etc are themselves implemented with a C/C++ codebase
This is neither true (e. g. Go's compiler and runtime are implemented in Go and a little assembly) nor is it relevant.

DNF 3: better performance and a move to C++

Posted Apr 8, 2018 19:35 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

Rust compiler is implemented in Rust. It does depend on LLVM which is in C++, but there's a work on a pure Rust implementation ( https://github.com/Cretonne/cretonne ).

Golang compiler is written in Go with runtime at more than 99% in Go.

Heck, even Java these days is planning to use Java-based runtime.

DNF 3: better performance and a move to C++

Posted Apr 10, 2018 3:54 UTC (Tue) by linuxrocks123 (subscriber, #34648) [Link] (5 responses)

I'm trying to make a generically useful build system, for C++ and other languages. It has the capability of scanning files to automatically discover #includes, among other nice things like auto-adding new CPP files to the build. It works right now but it doesn't do parallel compilation (there's no reason it couldn't in theory; I just haven't bothered).

https://github.com/linuxrocks123/bake

I wasn't impressed with CMake when I used it.

DNF 3: better performance and a move to C++

Posted Apr 10, 2018 5:06 UTC (Tue) by karkhaz (subscriber, #99844) [Link] (4 responses)

To get parallel compilation, I would suggest a system like what I use. I have something similar to your baker.py, except that there are usually multiple of them---one in each subdirectory of a project. Each baker.py emits the build information for that directory (maybe subdirectories too, if they don't have their own baker.py), and calling the script from the top level gathers up all the build recursively. This keeps the dependency information local and easy to understand in modular way.

The difference with my version is that I haven't written my own build tool---my "baker.py" always emits ninja syntax, rather than your custom syntax that needs its own build tool. So running baker.py is akin to running `configure`, and I then separately run ninja to build. You get parallel compilation and many other goodies for free with ninja.

If you don't like cmake, there's no need to write an entire build+metabuild system---cmake is just the meta-build aspect. As for builds, I don't think the fact that ninja is wonderful is controversial, so you should just emit ninja from your meta-build system and use that to build.

I haven't published it because it's ugly and needs a rewrite but hopefully the idea is clear w.r.t. how your system works.

DNF 3: better performance and a move to C++

Posted Apr 10, 2018 10:45 UTC (Tue) by HelloWorld (guest, #56129) [Link] (2 responses)

What's the point of having a separate “meta build system”? Nobody else does that, see for instance Shake, sbt or mill. As far as I can see, you only need a “meta build system” if your underlying build system sucks.

DNF 3: better performance and a move to C++

Posted Apr 10, 2018 11:00 UTC (Tue) by karkhaz (subscriber, #99844) [Link] (1 responses)

1. It's faster. There should be no need to calculate the dependency graph every time you build. So the meta-build system emits the dependency graph, and the job of the build tool proper is merely to execute commands in accordance with the graph. You run the build tool repeatedly, and only re-run the meta-build tool when the dependency graph changes.

A big reason why ninja is so much faster than make is that it doesn't evaluate environment variables or have any kind of fancy logic (there's no support for all of the functions and expansions that make has). All of the decision-making should be done at configure time, there's no need to repeat it every time you build.

2. Multiple build directories, each with a different build configuration. With out-of-tree builds, I can have one build directory for release builds, one for debug, one for building only a sub-project, etc. You typically configure each of these directories once, e.g. by passing parameters or environment variables to the meta-build system; and then the build command is just `ninja` in each one. If the meta-build and build commands are combined, then you need to pass the parameters/variables _every single time_ you run a build, and need to remember which parameters are associated with each directory.

DNF 3: better performance and a move to C++

Posted Apr 10, 2018 12:29 UTC (Tue) by HelloWorld (guest, #56129) [Link]

Figuring out whether the dependency graph changed is one of the things that a good build system will do for you rather than having you think about whether you need to invoke the meta build system or not. And in fact the two-stage build systems do do that, I don't have to re-run the meta build system every time I add an #include in a .c file even though that does change the dependency graph as there is now an edge from the .c to the .h file. I don't see why other changes to the build graph should be handled differently.

As for your other argument, I don't buy that either. Release and debug builds should simply be different targets, and the build system should give you sufficient means of abstraction so that the common parts can be easily shared between the two.

DNF 3: better performance and a move to C++

Posted Oct 24, 2018 9:09 UTC (Wed) by linuxrocks123 (subscriber, #34648) [Link]

Ha! Yeah, it looks like I basically just cloned Ninja with bake. Hadn't heard of it before; thanks for the pointer. I'm still going to parallelize bake, probably tomorrow now that I turned my attention to it in response to your comment and found an easy way to do it.

The two systems are somewhat different, probably most notably in that a Bakefile calls its clients to generate the dependency tree, and will give a previous client's output as input to a later client for modification. Of course, I haven't actually used that for anything. Also the -sub feature looks unique, though I haven't actually written a baker.py that uses it yet, so perhaps it's harder to make use of than I suspected. No harm in keeping it around, though.

(Also yeah it's months later; sorry for not seeing this at the time.)