PEP 572 and decision-making in Python

The "PEP 572 mess" was the topic of a 2018 Python Language Summit session led by benevolent dictator for life (BDFL) Guido van Rossum. PEP 572 seeks to add assignment expressions (or "inline assignments") to the language, but it has seen a prolonged discussion over multiple huge threads on the python-dev mailing list—even after multiple rounds on python-ideas. Those threads were often contentious and were clearly voluminous to the point where many probably just tuned them out. At the summit, Van Rossum gave an overview of the feature proposal, which he seems inclined toward accepting, but he also wanted to discuss how to avoid this kind of thread explosion in the future.

Assignments

Van Rossum said that he would try to summarize what he thinks the the controversy is all about, though he cautioned: "maybe we will find that it is something else". The basic idea behind the PEP is to have a way to do assignments in expressions, which will make writing some code constructs easier. C has this, as does Go, but the latter uses some extra syntax that he finds distasteful.

The problem with the C-style of assignments is that it leads to this classic error:

    if (x = 0)
        ...

    if (0 = x)
        ...

Python vowed to solve this problem in a different way. The original Python had a single "=" for both assignment and equality testing, as Tim Peters recently reminded him, but it used a different syntactic distinction to ensure that the C problem could not occur. Python has always "prided itself" on not giving any possible way to have that mistaken assignment problem without having to resort to tricks like Yoda style.

A classic example of where these kinds of assignments would be quite useful is in pattern matching:

    m = re.match(p1, line)
    if m:
        return m.group(1)
    else:
        m = re.match(p2, line)
        if m:
            return m.group(2)
        else:
            m = re.match(p3, line)
            ...

    if m := re.match(p1, line):
        return m.group(1)
    elif m := re.match(p2, line):
        return m.group(2)
    elif m := re.match(p3, line):
        ...

Another motivating code pattern is the "loop and a half". It was once common for processing a file by line, but that has been solved by making file objects iterable; however other non-iterable interfaces still suffer from patterns like:

    line = f.readline()
    while line:
        ...  # process line
	line = f.readline()

    while True:
        line = f.readline()
	if not line:
	    break
	... # process line

    while line := f.readline():
        ... # process line

Another example is with comprehensions (e.g. list, dictionary). Sometimes programmers value the conciseness of the comprehension to the point where they will call an expensive function twice. He has seen that kind of thing, even in the code of good programmers.

But Python has done pretty well for 28 years without this functionality. A lot of people have reacted to the idea—in various different ways. Part of what people were looking for was examples from real code, not toy examples that were generated to justify the PEP. Peters and others found realistic examples from their own code where the feature would make the code shorter and, more importantly, clearer, Van Rossum said. All of those examples were too long to fit on his slides, however.

Contentious debate

One of the reasons that the debate has been so contentious, he thinks, is that there are so many different syntactic variations that have been suggested. Here is a partial list of the possibilities discussed in the threads:

    NAME := expr
    expr -> NAME
    NAME <- expr
    expr {NAME}
    NAME = expr
    expr2 where NAME = expr1
    let NAME = expr1 in expr2
    ...

Another contentious issue was the idea of sub-local scopes that got mixed into the PEP early on. The idea is to have implicit scopes that are only active during the execution of a statement; it is potentially useful, but there are some quirks and corner cases. In the end, it got scratched from the PEP.

Overall, the idea has been "incredibly controversial", he said. What one person thinks is more readable, another person thinks is less readable. The benefits are moderate and everyone has their own favorite syntax. Sub-local scope added other oddities and would have required new bytecodes in order to implement it.

The PEP also bundled a fix for another problem, which is a "weird corner case" in comprehensions at class scope. That should be removed from PEP 572 and turned into its own PEP, he said.

A poll was taken on PEP 572, but the additional corner-case fix was part of the PEP. That muddied the poll, since those who wanted the assignment feature but did not want (or weren't sure of) the comprehension fix did not have a way to vote; a new poll will need to be done. The PEP moved to python-dev prematurely, as well.

Python is not a democracy, Van Rossum said. But generally folks agree with his decisions "except when I don't accept [their] favorite change".

Mark Shannon wondered if the attendees of the Python Education Summit might have some thoughts on the feature. Van Rossum acknowledged that some have said the feature makes it harder to teach Python, but he is not really sure of that, in part because he does not know how people learn the language. Nick Coghlan said the problem is trying to describe the difference between = and :=, but Van Rossum suggested that teachers not use := in instructional code. However, he does recognize that sites like Stack Overflow will lead some newbies to copy code in ways that might be confusing or wrong.

Decision-making

The larger issue from this PEP is "how we make decisions", Van Rossum said. There were many long responses in the threads, mostly against the feature. Overall, there was "way too much email". There were many misunderstandings, digressions, explanations, both right and wrong, and so on. Part of the problem is that there is no real way to measure the effectiveness of new language features.

In the end, he had to stop reading the threads so he wouldn't "go insane". Chris Angelico, who is the author of the PEP, could not be at the summit, but Van Rossum suggested that he stop responding in the threads to try to tamp things down. He wondered how to "dig our way out" of situations like this. It got to the point where people were starting new threads in order to try to get the attention of those who had muted older threads with too many messages.

Łukasz Langa suggested that "dictators should dictate"; Van Rossum should perhaps use his role to put a stop to some of that kind of stuff. But if not, then Van Rossum may just have to defer and stop following the threads, as he did. Langa said that he never follows python-ideas for exactly this reason.

Van Rossum said that the PEP had four revisions that were discussed on python-ideas before moving it to python-dev; he "thought we had it right". Langa wondered if there were other PEPs with a similar kind of response. Static typing (also known as "type hints") is one that Van Rossum remembered. Shannon thought that did not have as many negative postings from core developers as PEP 572 has had. Van Rossum agreed that might be the case but did remember a few core developers in opposition to type hints as well.

Victor Stinner suggested that the python-ideas discussion be summarized in the PEP. Van Rossum said that he thought many who responded had not read the discussion section in the PEP at all. He noted that the python-ideas discussion was better than the one on python-dev, even though it too had lots of passionate postings. There are fewer people following python-ideas, Christian Heimes said. Van Rossum wondered if the opposition only got heated up after he got involved; people may not have taken it seriously earlier because they thought he would not go for it.

Ned Deily suggested that the pattern for summit discussions be used to limit how long a discussion goes on; perhaps give five days before a decision will be made. The Tcl project has a much more formal process, where core developers are required to vote on proposals, but he didn't know if the Python project wanted to go down that path. It might make sense to have someone manage the conversation for PEPs, Van Rossum said. He is familiar with the IETF process from the late 1990s, which had some of that. He actually borrowed from the IETF to establish the PEP process.

But Barry Warsaw believes that PEP 572 is an outlier. Since it changes the syntax of the language, people tend to focus on that without understanding the deeper semantic issues. He suggested that perhaps a small group in addition to the PEP author could shepherd these kinds of PEPs. But in the end, people will keep discussing it until a pronouncement is made on the PEP one way or the other.

Van Rossum said that he is generally conflict-averse; he prefers not to just use his BDFL powers to shut down a discussion. Angelico is somewhat new at writing PEPs of this sort; Van Rossum thinks that Angelico probably would not have kept pushing it if he and Peters had not jumped into the discussion. Steve Dower said that perhaps some PEPs could be sent back with a request to get some others to work with the author on it.

Brett Cannon pointed out that the PEP editors are not scrutinizing PEPs super closely before they move to python-dev; it is mostly a matter of making sure there are no huge problems with the text. It might make sense to have a working group that tried to ensure the PEP was in the right state for a quality discussion. Another idea would be to have a senior co-author on PEPs of this nature, Van Rossum said. In addition to being an expert on the subject matter of the PEP, they could use their authority to help steer the conversation.

a = 1
b = 2
[x+1 for x in a, b]

[x := 5 for x in a, b]

a = 5
def foo():
    a = 6
    f = lambda x: a := 7
    return f
print(a)
f = foo()
print(a)
f(0)
print(a)

m = re.match(p1, line)
if m:
    return m.group(1)
m = re.match(p2, line)
if m:
    return m.group(2)
m = re.match(p3, line)
if m:
    return m.group(3)
...

for i in [(p1,1),(p2,2),(p3,3)]:
    return re.match(i[0], line).group(i[1])

if re.match(p1, line) as m:
    return m.group(1)
elif re.match(p2, line) as m:
    return m.group(2)
elif re.match(p3, line) as m:
    ...

while f.readline() as line:
    ...  # process line

def as_loop(fn, *args):
    while True:
        res = fn(*args)
        if not res:
            return
        yield res
…
f = open(…)
for line in as_loop(f.readline):
    …

while True:
    line = f.readline()
    if not line:
        break
    …

with open(…) as f:
    for line in iter(f.readline, ''):
        …

>>> fp = open('/etc/os-release')
>>> next(fp)
'NAME="Arch Linux"\n'