The value of specs

By Jake Edge
July 8, 2015

The value of design specifications ("specs") for open-source projects is something of an open question. Some projects, with the Linux kernel perhaps being the most prominent, eschew specs in favor of code. Other projects, such as various OpenStack sub-projects, have a fairly heavyweight process that requires specs for most proposed features. In a recent openstack-dev discussion, the value of requiring specs for the Nova compute component was called into question.

Nova manages the compute resources for an OpenStack cloud. Those resources are in the form of different kinds of virtual machines (VMs) from hypervisors such as KVM, Xen, VMware, and Hyper-V or from container technology like LXC or Docker.

That discussion started with a June 24 post from Nikola Đipanov that was rather negative about the whole spec process:

Specs don't work! They do nothing to facilitate good design happening, if anything they prevent it. The process layered on top with only a minority (!) of cores [core team members] being able to approve them, yet they are a prereq of getting any work done, makes sure that the absolute minimum that people can get away with will be proposed. This in turn goes and guarantees that no good design collaboration will happen. To add insult to injury, Gerrit and our spec template are a horrible tool for discussing design. Also the spec format itself works for only a small subset of design problems Nova development is faced with.

Đipanov cited a few examples where he saw that the spec process had gone wrong that he had encountered just that week. He noted that a big part of the problem is that Nova is so large and tightly coupled that a heavyweight process has been put in place to, effectively, slow or stop changes from being made. It is the tight coupling that needs to be addressed, but that the process itself is preventing that.

But Daniel Berrange and others did not see things quite that way, with Berrange pointing out that the situation was far worse before the Nova project adopted specs:

When I was reviewing features in Nova before specs came along, I spent [a lot] of time just trying to figure out what on earth the code was actually attempting to address, because there was rarely any statement of the problem being addressed, or any explanation of the design that motivated the code.

New OpenStack features always require a blueprint in Launchpad, but many components, including Nova, have adopted a requirement that most features need specs based on a project-specific template. Blueprints are typically a much simpler statement of the problem to be solved, while specs require a great deal more detail, including design information, use cases, impacts, and more. In addition, new features are only approved for a single six-month development cycle; if they spill over into the next cycle, they must be re-reviewed and approved again.

While specs have made things much better, there are still a number of problems with the process, Berrange said. It is too rigid and bureaucratic, and too many features are being pushed into the spec process that could simply be handled with just a blueprint. Also, tying the spec review and approval schedule to that of the overall development cycle is counterproductive: "We should be willing to accept and review specs at any point in any cycle, and once approved they should remain valid for a prolonged period of time - not require us to go through re-review every new dev cycle as again that's just creating extra burden." In addition, as Đipanov also noted, there is only a subset of the core team (which is "already faaaar too small", Berrange said) that can approve specs, which creates further bottlenecks.

Others strongly agreed that specs have made things better, but some questioned whether the Gerrit code-review tool was the best mechanism for reviewing specs. One of the reasons for requiring specs (and placing them into Git repositories so they could be reviewed via Gerrit) was the inability to comment on blueprints in Launchpad. But code-review tools foster a line-by-line approach, which is not optimal to review specs, as Technical Committee chair Thierry Carrez noted:

Line-by-line code review tools are excellent for reviewing the correctness of lines of code. When switching to specs, you retain some of that "review correctness of all lines" mindset and tend to spot mistakes in the details more than mistakes in the general idea.

Part of the problem is that the spec template is overkill for many features, Carrez said. It would be better to start small and build more into a spec as it gets reviewed:

IMHO the ideal tool would allow you to start with a very basic description of what feature you want to push. Then a discussion can start, and the "spec" can be refined to answer new questions or detail the already-sketched-out answers. Simple features can be approved really quickly using a one-sentence spec, while more complex features will develop into a full-fledged detailed document before they get approved. One size definitely doesn't fit all. And the discussion-based review (opposed to line-by-line review) discourages nitpicking on style.

You *can* do this with Gerrit: discourage detail review + encourage idea review, and start small and develop the document in future patchsets as-needed. It's just not really encouraging that behavior for the job, and the overhead for simple features still means we can't track smallish features with it. As we introduce new tools we might switch the "feature approval" process to something else. In the mean time, my suggestion would be to use smaller templates, start small and go into details only if needed, and discourage nitpicking -1s.

Đipanov agreed with Carrez's ideas, and suggested that investigating other tools might be in order. On the other hand, Kyle Mestery noted that the Neutron networking component had recently switched from a heavyweight spec-based process to one that uses "request for enhancement" (RFE) bugs instead. The reasons behind the switch, as outlined in a blog post from Mestery, sound rather similar to the complaints heard in the Nova thread. So far, that switch is working out well, Mestery said.

The RFE process was also championed by Adam Young. He strongly agreed with Đipanov that Gerrit was not the proper tool for the job and suggested that keeping the documentation with the code (and keeping them both in sync) would avoid "bike shedding about Database schemas". But Berrange said that hearkened back to the days before specs for Nova, which "really didn't work at all - code reviews are too late in the workflow to start discussions around the design, as people are already invested in dev work at that point and get very upset when you then tell them to throw away their work".

But Young is fairly adamant that the spec process is holding back progress in the code:

Specs prevent code from being written. If you think too much code is written, then, yes, you will like specs. If, on the other hand, you think that things should be implemented and tested before being posted to the central repo, then specs are not nearly as valuable as end user docs. I think and design in Code, not in specs. There are too many details that you don't discover until you actually write the code, and thus the specs often do not reflect the reality of the implementation anyway.

On the other hand, the spec process not necessarily the real bottleneck, as James Bottomley pointed out; review bandwidth will not magically increase simply by removing specs from the process. He is concerned that precious reviewer time may be wasted on things that should already have been accepted (because they are an obvious bug fix, say) or rejected (for bogus code). Reducing the number of reviews required to get to a resolution is the way to stretch review resources.

There is another, possibly overlooked, advantage to the spec process that Tim Bell raised: it allows operators and other users without Python knowledge to "give input on the overall approach being taken". If commenting is left until code review time, it leaves out those who aren't able to read the code—and who may have important thoughts based on running OpenStack in production. Đipanov acknowledged that, but is still concerned about the weight of the process for many of the features proposed for Nova.

In a summary post, Đipanov outlined the positives and negatives with regard to the Nova process that had emerged from the discussion. The strident "specs don't work" attitude from his initial post is replaced with a more even-handed view. The post also makes some concrete suggestions for moving forward.

Full-blown specs should not be required from the outset, he suggested. Instead, a simpler blueprint that is mirrored into the repository could be used and a spec should only be created if multiple core team members (or a larger number of contributors) request one (by making a negative vote on the blueprint). In addition, feature approval should not necessarily expire when a release is made—expiration should strictly be for the specific features that require it. Lastly, new tools should be considered that would facilitate a more nimble process, perhaps along the lines of what Carrez described.

At some level this is a struggle between those of a more "agile" mindset and those who are more process-oriented. It seems that there is broad agreement that improvements are needed to the current Nova development process, but where and how those changes come is not yet clear. The OpenStack project, though, has multiple components, each with its own process, that can be studied to see what works and what doesn't—and why. Beyond that, sub-projects like Nova can also look at the wider free-software world for ideas. A bit of observation and iteration is likely all that is required to find some useful improvements to the Nova development process.

to post comments

The value of specs

Posted Jul 9, 2015 8:25 UTC (Thu) by jezuch (subscriber, #52988) [Link] (4 responses)

> Some projects, with the Linux kernel perhaps being the most prominent, eschew specs in favor of code.

Not really. Each commit has a detailed commit message (and I assume that maintainers won't accept anything that does not explain *precisely* why it's needed and how it does what's needed - apart from trivial things like whitespace fixes) and each merge Linus does has a list of changes it brings in. If that's not specs than I don't know what it is ;)

Meanwhile it looks like OpenStack fell victim to the Pendulum Principle: it was bad so they swung to the polar opposite, which, as it turned out, is also bad. Now they have to resist the temptation to swing back again :)

The value of specs

Posted Jul 9, 2015 11:08 UTC (Thu) by dgm (subscriber, #49227) [Link] (3 responses)

I don't believe that commit messages are specs, the same way code commens aren't. And that is basically for two reasons:

1. they come *after* the code, and

2. they have roughly the same granularity as the code.

Specs should be much higher level (read: light on details) than code, and should serve as a basis for discussion before coding starts, (and interpretation of the code afterwards).

The value of specs

Posted Jul 9, 2015 12:16 UTC (Thu) by andresfreund (subscriber, #69562) [Link]

> 1. they come *after* the code, and

I'm not convinced that's a contradiction. Specs that are done without any sort of reference implementation usually end up missing major issues. Which is why these days many standards comittee require at least one or two implementations before passing a standard.

I can agree on other levels that most commit messages aren't really standards though.

The value of specs

Posted Jul 9, 2015 13:26 UTC (Thu) by jwarnica (subscriber, #27492) [Link]

There are really two different things not clearly highlighted. Specs, and *concepts*. A specification is a technical description of a concept. I agree, generally, that software specs are rather pointless - clean code is easier to write and read than clean prose, and can actually be executed - but the hiccup usually boils down to if the participants agree that the concept itself is a a good idea.

For example, "kdbus". ISTM (from last weeks reporting) that as much bikeshedding over implementation details are happening as the fundamental question of if it is a good idea. Despite only very few people not wanting it in the kernel, period. I doesn't need a 50 *word* spec, but an upfront consensus ACK on "kernel mode dbus" is all it would go a long way to dealing with negative people. In their mind, they aren't being annoying: the concept hasn't been approved, ergo, it is fair game to dispute the concept at every posted diff.

The value of specs

Posted Jul 9, 2015 21:29 UTC (Thu) by danpb (subscriber, #4831) [Link]

Yes that's right - a piece of work may result in a large series of patches, and the spec is really describing the work as a whole. It gives important background information as to the ultimate use cases of the feature, and describes what alternative implementations may have been considered, and generally calls out other important design aspects. There's no perfect equivalent in other projects, but rather than commit messages, OpenStack specs are probably closest to the git cover letter concept that is usually sent as the introduction to a series of patches, when patch review takes place on a mailing list. The specs process idea was in a large part motivated by inadequacies in the combination of Gerrit code review tool and Launchpad blueprints. Among Gerrit's many flaws is the poor handling of patch series and most specifically the inability to supply a "cover letter" to a series of patches, so reviewers have little context besides the patch. OpenStack had been using launchpad blueprints to provide design info prior to specs, but it is an awful tool for this purpose as there's no effective method for commenting & tracking review feedback in blueprints. Specs tried to formalize the creation of rough design docs to substitute for lack of cover letters to patch series, as well as providing a mechanism to review & comment on proposals upfront to avoid people wasting time on code that was incorrectly designed.