[go: up one dir, main page]

|
|
Log in / Subscribe / Register

Fedora mulls its "python" version

By Jake Edge
July 3, 2019

There is no doubt that the transition from Python 2 to Python 3 has been a difficult one, but Linux distributions have been particularly hard hit. For many people, that transition is largely over; Python 2 will be retired at the end of this year, at least by the core development team. But distributions will have to support Python 2 for quite a while after that. As part of any transition, the version that gets run from the python binary (or symbolic link) is something that needs to be worked out. Fedora is currently discussing what to do about that for Fedora 31.

Fedora program manager Ben Cotton posted a proposal to make python invoke Python 3 in Fedora 31 to the Fedora devel mailing list. The proposal, titled "Python means Python 3", is also on the Fedora wiki. The idea is that wherever "python" is used it will refer to version 3, including when it is installed by DNF (i.e. dnf install python) or when Python packages are installed, so installing "python-requests" will install the Python 3 version of the Requests library. In addition, a wide array of associated tools (e.g. pip, pylint, idle, and flask) will also use the Python 3 versions.

The "Requests" link above does point to a potential problem area, however. It shows that Requests for Python 3 III is not fully finished, with an expected release sometime "before PyCon 2020" (mid-April 2020), which is well after the expected October 2019 release of Fedora 31. The distribution already has a python3-requests package, though, so that will be picked up as python-requests in Fedora 31 if this proposal is adopted. There may be other packages out there where Python 3 support is not complete but, at this point, most of the major libraries have converted. [Update: As noted in the comments, this paragraph is based on complete confusion about the version numbers of Requests.]

The proposal is owned by Miro Hrončok and Petr Viktorin, who work on the Python team at Red Hat. Viktorin presented at the 2016 and 2018 Python Language Summits on the problems inherent in the transition to Python 3 for distributions. Both have also been involved in reworking the recommendations that the Python core development team provides to distributions. Those are contained in PEP 394 ("The 'python' Command on Unix-Like Systems"). The rework consists of loosening those guidelines to allow distributions to choose Python 3 for their python; it has garnered the support of the Python steering council.

One of the concerns about the proposal is the specter of another similar transition for Python down the road. As "stan" put it: "Wouldn't this mean that when python4 is released, there is a replay of the python2 -> python3 transition experience?" But Hrončok pointed out that the packages are not being renamed, just that unadorned "python" will resolve to "python3"; shebang ("#!") lines for Python code shipped in packages will need to directly invoke python3, not simply python.

That led Cotton to wonder what the advantage of having the unadorned "python" was; why couldn't there simply be python3 (and python2 if it is installed)? Hrončok noted that users attempting to use Python on Fedora systems might find the following sequence to be suboptimal:

    $ python
    bash: python: command not found

Oh right!

    $ sudo dnf install python
    No match for argument: python
    Error: Unable to find a match

Oh well?

He continued: "Fedora is equipped with fully operational, ready to use and develop on Python interpreter, why not let people find it?" He also pointed out that the upstream Python team was not pushing for this change, but simply allowing for the possibility. Fedora could still decide to not have a python and it would not run afoul of the recommendations. But he strongly suggested that, in any case, python no longer point to Python 2 (even though that would not run afoul of the recommendations either).

The idea of having no python was not particularly popular. Administrators could make that happen on their systems, if they wished, or point python to Python 2 for that matter. But, as Gerald Henriksen put it:

[...] someone sitting down to use a tutorial to learn Python is going to expect to type python and get something they can use, particularly going forward when Python3 really just [becomes] Python as Python2 fades from memory.

In answer to a question from Adam Williamson, Hrončok said that the reasoning for making the change for Fedora 31 was mostly due to the dates involved, as the proposal describes: "The name 'Python' will not refer to software that will be unmaintained upstream for most of Fedora 31's lifetime and retired from Fedora 32." The latter piece refers to the plan to stop shipping Python 2 as python2 in Fedora 32. A legacy python27 package will be created (and may be allowed as a dependency for a small number of packages).

Zbigniew Jędrzejewski-Szmek asked about perhaps postponing the proposed change to Fedora 32 as well:

I think that having this middle state where python2 is available but python points to python3 for exactly one release will be more confusing that switching directly to the final state where python2 is gone and python simply means python3.

Stephen John Smoogen thought that having a release with no python at all would be useful to suss out packages that have still not fixed their shebang lines and such. But Michael Catanzaro said that not having a /usr/bin/python would wreak havoc for software that needs to run on both Linux and macOS, such as WebKit.

If Fedora has /usr/bin/python, then at least we have a *chance* to make the scripts we care about work in both python2 and python3 (our current plan). Whereas without /usr/bin/python, we're really out of options. So I very much support /usr/bin/python -> /usr/bin/python3. It will cause some problems for us and we'll have a difficult transition period, but at least there will exist the possibility of transitioning.

Others will also want to be able to simply use #!/usr/bin/python in their scripts. The distribution can pick a particular version to target in its scripts, but users may have different agendas. Mátyás Selmeci said:

I work on multiple platforms so I make my utility scripts work on both Python 2 and 3 and only use the standard library. It would be super annoying if I had to have multiple versions just because of the shebang line.

There was a fair amount of discussion but pointing python to Python 3 seems like a relatively uncontroversial move. There really wasn't any impassioned argument against it, which is perhaps a little surprising. But Fedora has been moving steadily down the path to retiring Python 2 more or less in lockstep with the Python core developers. One wonders when it will get to the point that legacy Python 2 users will have to get their interpreter from outside of Fedora entirely—maybe by 2025?


Index entries for this article
PythonLinux distributions


to post comments

Fedora mulls its "python" version

Posted Jul 3, 2019 23:21 UTC (Wed) by Kamilion (subscriber, #42576) [Link] (9 responses)

Oh, come ON!

Can we PLEASE PLEASE just switch to the 'py' launcher instead of invoking the python executable directly?

https://github.com/python/cpython/commits/master/PC/launc...

Yes, it was meant for windows, but it works just fine on unix too.

I'd much rather
#!/usr/bin/py -3

or
#!/usr/bin/py -3.6-64 -u

than breaking everything relying on #!/usr/bin/python or #!/usr/bin/env python or the python3 variants.

Fedora mulls its "python" version

Posted Jul 4, 2019 0:00 UTC (Thu) by dezgeg (guest, #92243) [Link] (5 responses)

"#!/usr/bin/py -3.6-64 -u" does not work correctly on Unix (or at the very least, on Linux) since shebangs can have only one argument. I.e. that will execute /usr/bin/py with a single argument "-3.6-64 -u".

Fedora mulls its "python" version

Posted Jul 4, 2019 14:21 UTC (Thu) by clopez (guest, #66009) [Link] (3 responses)

Maybe the program py is smart enough to detect this and splits itself the arguments? (No idea if it does that, but in theory it can)

Fedora mulls its "python" version

Posted Jul 4, 2019 17:37 UTC (Thu) by brettcannon (guest, #132970) [Link] (2 responses)

Splitting arguments appropriately is surprisingly hard do to shell differences in quoting (e.g. you can't blindly split on whitespace).

Fedora mulls its "python" version

Posted Jul 4, 2019 21:02 UTC (Thu) by marcH (subscriber, #57642) [Link]

Uwe Waldmann's "A Guide to Unix Shell Quoting"
... is a detailed explanation of the issues.

The interview from the creator explains how it came to be like this: https://www.computerworld.com.au/article/279011/-z_progra...

+ Greg's BashFAQ as usual.

Fedora mulls its "python" version

Posted Jul 11, 2019 2:08 UTC (Thu) by Hello71 (guest, #103412) [Link]

conveniently, python provides a "shlex" module for this functionality.

Fedora mulls its "python" version

Posted Jul 4, 2019 19:31 UTC (Thu) by Deleted user 129183 (guest, #129183) [Link]

> "#!/usr/bin/py -3.6-64 -u" does not work correctly on Unix (or at the very least, on Linux) since shebangs can have only one argument.

On GNU/Linux (and FreeBSD) it’s possible to use `#! /usr/bin/env -S py -3.6-64 -u`… but not always unfortunately, since this feature has been only introduced in coreutils 8.30.

Fedora mulls its "python" version

Posted Jul 4, 2019 5:51 UTC (Thu) by rsidd (subscriber, #2582) [Link] (2 responses)

The py launcher seems to be a workaround for the fact that there is no shebang notation on windows. So on windows, it declares itself the launcher for .py files and does the right thing including figuring out the right version of python to use. What is the advantage on linux? Why would
#!/usr/bin/py -3
be preferable to
#!/usr/bin/python3
and how would it not break existing #!/usr/bin/python scripts?

Fedora mulls its "python" version

Posted Jul 4, 2019 17:37 UTC (Thu) by brettcannon (guest, #132970) [Link]

The Python launcher is partially to deal with the lack of shebang line on Windows, partially to make it easier to say "just use the newest version of Python". But I honestly wouldn't use the Python launcher in a shebang line as that's what /usr/bin/env (mostly accomplishes) for you, e.g. `/usr/bin/env python3` is close (enough) to `/usr/bin/py -3`.

So for UNIX I would say the Python launcher is more about the ease of launching Python, i.e. you can be super-lazy and just say `py` and it will do the thing you want (whether that's launching the newest version of Python found on PATH or use the Python in an activated virtual environment).

And FYI, I'm working on writing the Python Launcher for UNIX at https://crates.io/crates/python-launcher.

Fedora mulls its "python" version

Posted Jul 5, 2019 12:13 UTC (Fri) by Kamilion (subscriber, #42576) [Link]

>how would it not break existing #!/usr/bin/python scripts?

Ideally, with the launcher present, modern scripts would be advised to depend on that, instead of the legacy 'python' becoming a symlink to python3/4. Seeing 'python' at all in a shebang line would be an immediate 'red flag' that the following script was likely a legacy one written before 2020.

Python4's not exactly all that far away, and like linux 4.x -> linux 5.x there should be no breaking changes like the 2 -> 3 unicode situation.

Moving over to the launcher now "as the modern choice" seems like a better move than sticking with the python/python2/python3/python4/python5 problems if the executable name keeps getting bumped with the major.

(for those wondering; yes, the launcher does split the shebang arguments line itself, as well as handling what I called 'legacy' shebangs using env, python, or python3, ostensibly to make unix shebangs work 'as expected' without modification on windows without the benefit of execute permissions / chmod +x)

Fedora mulls its "python" version

Posted Jul 3, 2019 23:24 UTC (Wed) by cjwatson (subscriber, #7322) [Link] (2 responses)

This article seems to confuse "version 3.x of the Requests library for Python" with "Requests for Python 3", which I think must be a mistake and is certainly unwarranted. Requests 2.x supports both Python 2 and 3, and has done for quite some time.

Fedora mulls its "python" version

Posted Jul 4, 2019 0:23 UTC (Thu) by gps (subscriber, #45638) [Link]

Indeed. requests fully supports Python 3 as https://pypi.org/project/requests/ states.

Fedora mulls its "python" version

Posted Jul 4, 2019 0:50 UTC (Thu) by jake (editor, #205) [Link]

> This article seems to confuse "version 3.x of the Requests library for Python" with "Requests for Python 3", which I think must be a
> mistake and is certainly unwarranted. Requests 2.x supports both Python 2 and 3, and has done for quite some time.

Hey, maybe that explains the Fedora versions for Python 3! :) ... sigh ... I got confused by the "maintenance mode only" note here: https://2.python-requests.org/en/master/ and the Request III stuff ... but really should have looked closer ... my apologies ...

thanks for pointing it out ...

jake

Fedora mulls its "python" version

Posted Jul 4, 2019 0:56 UTC (Thu) by dezgeg (guest, #92243) [Link] (12 responses)

Having used Arch Linux where this python -> python3 mistake was made early on, the experience was just endless pain with no benefit. Using or compiling anything from source that happened to contain Python code was practically guaranteed to contain these pointless and unnecessary steps:

1) Have the build fail with "Missing parentheses in call to 'print'" in some random 10 line build script 2) Cursing and gnashing teeth 3) Mass-sed 's|/usr/bin/python|/usr/bin/python2|' 4) Find out you missed some places that decided to use '/usr/bin/env python' instead, and repeat steps 1-3.

Even in 2018 I had to deal with that exact same crap in U-Boot's build system where some pure Python2 scripts were with a `/usr/bin/python` shebang. Or alternatively, some scripts that had been ported to support both 2 and 3 at some time, but then regressed due to nobody actually testing them with Python3.

Please do not repeat this mistake in other distros.

Fedora mulls its "python" version

Posted Jul 4, 2019 1:12 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

"Having used Arch Linux where this python -> python3 mistake was made early on, the experience was just endless pain with no benefit."

For one, Python 3 has come a long way since then and is much more widely used but even otherwise, Fedora's proposal isn't the same thing. In particular python3 RPM package only provides /usr/bin/python3 and python-unversioned-command is a subpackage of python3 RPM package that provides the symlink from python -> python3 and that is only a recommended package but NOT a strict dependency. So if you want to remove /usr/bin/python, you can always just remove the subpackage and create your own symlink from python -> python2 and not run into the same breakage.

Fedora mulls its "python" version

Posted Jul 4, 2019 7:35 UTC (Thu) by Conan_Kudo (subscriber, #103240) [Link] (7 responses)

There is one big difference. Fedora doesn't allow packaged stuff to use the unversioned shebang. Our packaging policy and our build environments force packagers to fix that shebang to point to either Python 2 or Python 3. We've been forcing that choice since Fedora 28. So the unversioned python binary is purely for developer convenience.

Fedora mulls its "python" version

Posted Jul 4, 2019 8:28 UTC (Thu) by cpitrat (subscriber, #116459) [Link] (6 responses)

This was the same on arch. The problem was with anything coming from outside of arch. At that time, python3 migration was not at the same stage, but I'm pretty sure many migrated to python3 without changing the shebang (or missing some).

Fedora mulls its "python" version

Posted Jul 4, 2019 12:01 UTC (Thu) by khim (subscriber, #9252) [Link] (5 responses)

Python2 goes away in one year, anyway. Packages which still haven't transitioned would give you grief no matter what (if they haven't switched in 9 years it's unlikely that they would switch in 10).

So you would face worse problems with these systems soon. Fedora, like usual, just does that half-year too early.

Fedora mulls its "python" version

Posted Jul 4, 2019 12:33 UTC (Thu) by cpitrat (subscriber, #116459) [Link] (3 responses)

Packages which didn't migrate are not related to the problem here. Although using versionned binary in shebang is recommended, you can totally keep python (PEP394 says it can be used if compatible with both versions).

So valid Python 3 scripts kept compatible with Python 2 by careful developers could be broken by removing python.

Fedora mulls its "python" version

Posted Jul 4, 2019 12:38 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

> So valid Python 3 scripts kept compatible with Python 2 by careful developers could be broken by removing python.

I don't follow. In a future of release of Fedora, if /usr/bin/python points to /usr/bin/python3, valid Python 3 scripts will work just fine

Fedora mulls its "python" version

Posted Jul 4, 2019 13:16 UTC (Thu) by cpitrat (subscriber, #116459) [Link] (1 responses)

The article mentions discussion about removing completely the python link and providing only python2 and python3.

Fedora mulls its "python" version

Posted Jul 4, 2019 13:24 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

> The article mentions discussion about removing completely the python link and providing only python2 and python3.

That's not the actual proposal and people who proposed that backed off from it. So that's not happening

Fedora mulls its "python" version

Posted Aug 2, 2019 11:35 UTC (Fri) by mgedmin (guest, #34497) [Link]

> if they haven't switched in 9 years it's unlikely that they would switch in 10

You're seriously underestimating the power of the last minute.

Fedora mulls its "python" version

Posted Jul 4, 2019 10:59 UTC (Thu) by moltonel (subscriber, #45207) [Link] (1 responses)

On my systems I've had "python" point to the latest python since around python 3.3 or 3.4, so about 4 to 5 years.

Yes, there's the occasional "/usr/bin/python" bangline that actually needs python 2.7, but IMHO that's really an issue with that package rather than the underlying OS, and thankfully it's easy to fix either locally or upstream.

When you ask for unversioned "foo", you get the latest installed "foo". If there are multiple versions of "foo" in the wild, you can either be compatible with a range of versions, or ask for a specific version. That's a pretty basic principle of version handling, I find it strange that it should somehow not apply to python.

Fedora mulls its "python" version

Posted Jul 4, 2019 21:23 UTC (Thu) by marcH (subscriber, #57642) [Link]

> When you ask for unversioned "foo", you get the latest installed "foo". If there are multiple versions of "foo" in the wild, you can either be compatible with a range of versions, or ask for a specific version. That's a pretty basic principle of version handling, I find it strange that it should somehow not apply to python.

Simply because some people expect(ed) no major revision ever and backward compatibility forever. There's no /bin/sh2 for instance and I bet there'll never be. Even if you don't agree it's understandable. BTW I realize /bin/sh2 is comparing apples to oranges a bit but that's not me doing the comparison, you get the idea and you may even find better examples.

Now that python3 is out and not backward compatible, some people still try to ignore that new version 3 thing and still cling to the "python == python2" backward compatibility idea. "If it ain't broke don't fix it"

So the validity of the "python -> python2" request relies entirely on how long python2 will be supported. Upstream but not just.

Fedora mulls its "python" version

Posted Jul 4, 2019 21:07 UTC (Thu) by marcH (subscriber, #57642) [Link]

> Mass-sed 's|/usr/bin/python|/usr/bin/python2|' 4) Find out you missed some places that decided to use '/usr/bin/env python' instead, and repeat steps 1-3.

How well were these patches received upstream?

Fedora mulls its "python" version

Posted Jul 4, 2019 10:09 UTC (Thu) by Otus (subscriber, #67685) [Link] (6 responses)

As a user, I'd expect python to run python2 if that is available. Any script where I have /usr/bin/python in the hashbang line will expect to run on python2. Otherwise I would have specified python3.

Sure, once there is no python2 on the system, trying to run python3 instead of throwing an error may be preferable.

Could there be a stub that determines if it's an interactive session (give me 3) or a script (2 if available)?

Fedora mulls its "python" version

Posted Jul 4, 2019 10:58 UTC (Thu) by peter-b (guest, #66996) [Link] (4 responses)

As a user, I'd expect python to run python3 if that is available.

Oh look, different users have different expectations. Which is what this article is about.

Fedora mulls its "python" version

Posted Jul 4, 2019 11:06 UTC (Thu) by Otus (subscriber, #67685) [Link] (3 responses)

Yes. So, as always, I'd advocate for keeping existing behavior. That's what people may depend on.

Fedora mulls its "python" version

Posted Jul 4, 2019 11:28 UTC (Thu) by Cato (guest, #7643) [Link] (2 responses)

Python 2 stops being supported in Jan 2020, so it's sensible for future releases of distros to make Python 3 the default.

Fedora mulls its "python" version

Posted Jul 4, 2019 13:00 UTC (Thu) by Otus (subscriber, #67685) [Link] (1 responses)

At that point? Yes, like I wrote originally.

Before 2 is dropped from the distro? I disagree.

Fedora mulls its "python" version

Posted Jul 4, 2019 13:19 UTC (Thu) by cpitrat (subscriber, #116459) [Link]

So you argue that it's better to do it in a way that there's no workaround if it breaks something?
Because if you make python point to python3 only after python2 is not available at all for the distro, it means a user who encounter a problem cannot just fix the link to point back to the working version. They have to downgrade to the previous version of the distro. That sounds a bit silly.

Fedora mulls its "python" version

Posted Jul 4, 2019 11:30 UTC (Thu) by mti (subscriber, #5390) [Link]

And also make the /usr/bin/python stub select depending on some environment variable:

export PYTHON=3 foo.py

Works also for scripts, makefiles etc that calls Python programs indirectly.

Fedora mulls its "python" version

Posted Jul 4, 2019 14:16 UTC (Thu) by joekiller (guest, #126069) [Link] (2 responses)

Apple seems to be side stepping the whole issue by removing scripting languages. If you want one, install it. Somewhat of a Windows like experience. vi is also being excluded from core distributions as well. Considering the debate around Python's batteries being included or not, it seems the trend is to slim stuff up.

Package managers are easier to use than ever so perhaps batteries not included is the correct baseline.

Fedora mulls its "python" version

Posted Jul 4, 2019 14:57 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link]

Fedora can't get rid of Python because too much critical infrastructure is written in it. For example, your suggestion of using the package manager to install Python if you want it won't work because the package manager is written in Python.

Fedora mulls its "python" version

Posted Jul 4, 2019 15:04 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

> Apple seems to be side stepping the whole issue by removing scripting languages

Won't work at all for Fedora. Everything from the installer to the package manager has Python code but also, there are is a ton of open source projects using scripting language. No single vendor has Apple kind of money, power or control


Copyright © 2019, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds