Development
Perl 5.16 and beyond
Perl 5's yearly release rhythm is well established. Because major releases come out every single year, a major release no longer introduces a slew of new features. Instead, it consists of a smaller set of features and bug fixes. Upgrading to a new major version is easier than it's ever been. So why upgrade at all? What's changed in Perl 5 recently, and where is it going?
Syntax extensions outside the core
One of the most important changes has been a push toward making the Perl 5 core more extensible. This work started with the Perl 5.12.0 release, and has continued since then. Many of the internal APIs have been cleaned up and documented. It has also become much easier to write extensions that work like Perl builtins, and even to extend the syntax in entirely new ways. Extensions can modify the "optree" while Perl is compiling your code. The optree is the Perl interpreter's internal representation of a piece of code. It's what the interpreter executes when it runs your program. When extensions are compiled down to ops, they can be as fast as an implementation in the core would be.
The smart match feature that was added in 5.10.0 provides a good example. Smart match includes the "given" and "when" builtins, as well as the "~~" operator. In the code, smart match examples can look like:
$value ~~ @array
@array ~~ qr/foo/
The behavior of ~~ varies based on the type of both operands. The
first example checks to see if $value is a member of
@array. The second example checks to see if any member of
@array matches the regular expression.
The Perl 5 Porters eventually realized that this feature's behavior is
much too complicated. Unfortunately, this realization happened after the
feature shipped, so we can't simply remove or change it out from under
existing users.
What we need is a way to change the behavior in future releases while still providing access to the old behavior. We could, of course, just do this in the Perl core's C code. If the "enable old behavior" flag is on, we use the old code path, otherwise we use the new path. While this is feasible, it's not desirable. It clutters the core, and the more features that go down this path, the messier the core gets.
Jesse Luehrs's smartmatch module lets us alter the behavior of the smart match feature in a lexical scope by loading a module. The smartmatch::engine::core module implements the original behavior as introduced in 5.10.0. If we include "use smartmatch 'core'" in our own code, the smart match operator and builtins use the old behavior. Meanwhile, another module can include "use smartmatch 'sane'" and get a different behavior.
Because this module takes advantage of hooks in the Perl interpreter for syntax extensions, it can actually insert itself into the compiled Perl optree. This means that the extension can be as fast as the core implementation without being part of the core C code. When the smartmatch behavior in the core is changed, this module can be shipped with that release. We can even arrange for code that includes "use v5.14" to use the old smart match behavior, while code that includes "use v5.18" gets the new behavior.
This extensibility also makes it easy to prototype new features as CPAN modules. Florian Ragwitz's List::Gather module on CPAN adds gather and take "builtins" which compile down to operations in the parsed program. This syntax provides a nice way to create an array from an iterating operator:
use List::Gather;
my @list = gather {
while (<$fh>) {
next if /^\s*$/;
next if /^\s*#/;
last if /^(?:__END__|__DATA__)$/;
take $_ if some_predicate($_);
}
take @defaults unless gathered;
};
This syntax could easily be included in the core simply by bundling the List::Gather module in the core distribution. All we need to do is make use v5.18 load List::Gather behind the scenes. There's no need to alter the core at all.
What's new in Perl 5.16.0
Perl 5.16.0 doesn't have any huge features, but it does have a collection of small features and bug fixes that will make Perl 5 better. In 5.16.0, we now have a __SUB__ token. This token returns a reference to the current subroutine, letting us write cleaner recursive closures:
use feature 'current_sub';
my $factorial = sub {
my $val = shift;
if ( $val > 1 ) {
return $val * __SUB__->( $val - 1 );
}
else {
return $val;
}
};
print $factorial->(5);
The 5.16.0 release also adds support for the Unicode 6.1 standard, in addition to several other Unicode improvements. We now have much better support for Unicode characters in symbol names (packages, methods, etc.). The new fc operator and \F escape implement the Unicode foldcase operation, which does proper case-folding for all languages.
As part of the previously mentioned push to make the interpreter internals more extensible, 5.16.0 documents a number of functions for manipulating "pads". A pad (or scratchpad) is the data structure that stores lexical variables for each subroutine. This API was already in use by some modules, like List::Gather, but documenting it means that module authors can rely on the API's stability.
There has also been a lot of work on the core documentation. Perl 5.16.0 ships with a new object-oriented programming tutorial, and the object-oriented reference documentation has been rewritten from scratch with expanded coverage.
In addition to these changes, Perl 5.16.0 has many other bug fixes, performance improvements, documentation improvements, and core module updates.
A detour through the Perl 5 ecosystem
Just talking about the core when talking about the state of Perl 5 misses much of what makes Perl Perl. Perl's greatest strength has always been the Comprehensive Perl Archive Network (CPAN) and the larger Perl community. This is a huge topic, so I'll only hit a few highlights.
If you haven't looked at Perl recently, you may not have heard of Moose. Moose borrows from Common Lisp Object System, Perl 6, and many other languages. Moose provides a clean declarative API for declaring classes and roles. This eliminates huge amounts of boilerplate that Perl's native object system requires, allowing developers to focus on what their class does, rather than how it does it. Moose also provides a self-hosted metamodel, which means that it can be extended by writing classes and roles using Moose itself. Moose has been adopted by many CPAN authors for their own modules, and there are dozens of Moose extensions available.
Perl 5 has a number of excellent web frameworks available. Catalyst, Dancer, and Mojolicious are all mature, well-supported, and widely used. All of these frameworks build on top of the PSGI spec and Plack tools. These are inspired by Python's WSGI and Ruby's Rack respectively. Any application or framework that implements the PSGI spec can easily be deployed using FastCGI, standalone servers like Starman, mod_perl, or even plain old CGI. This makes writing and deploying Perl web applications easier than it's ever been.
Perl also has a number of Object-relational mapping modules, but the most popular is DBIx::Class. It has been under development since 2005, and has seen wide adoption throughout the community, attracting a number of core contributors, as well as inspiring dozens of extensions.
The services built around the CPAN archive are also exciting. The MetaCPAN API provides a web API to a database of every distribution ever uploaded to the CPAN archive. This API is open and freely usable, so anyone can build tools on top of it. There is already a new CPAN search site, also called MetaCPAN, that uses this API.
The CPAN Testers service collects test reports on every distribution uploaded to CPAN. Clients test the distributions on many different platforms and Perl versions. The service received its twenty millionth test report on March 7, 2012, and is currently receiving nearly 1 million reports a month.
The community is also busy organizing events. There are YAPCs (Yet Another Perl Conferences) this summer in the US, Germany, Brazil, and Japan, as well as many smaller workshops scheduled or in the works.
Future Plans for Perl 5
What happens after Perl 5.16.0? What will be in 5.18.0 or 5.20.0? Perl 5 development is volunteer driven, and I cannot commit anyone else's time. That said, here are some ideas that have been floated for future releases.
The project I'm most excited about is work on a MOP for the Perl 5 core. A MOP is a Meta Object Protocol. This is what Moose provides, but Moose does this as an extension. The goal is to create an API that can be implemented by the Perl 5 core and extended by modules like Moose. Putting this in the core has the potential to make modules like Moose much faster. However, the MOP is not just for Moose. It will be flexible enough to support multiple object systems, and will be usable as a minimal Object Oriented system all on its own, without extension.
As a bonus, this work will also include a few new bits of syntax. In particular, classes and methods will finally be distinct from packages and subroutines, and there will also be core support for roles, named method parameters, and attribute declaration.
I already mentioned the work on making the core more extensible. That work is an ongoing effort that opens up the possibility of breaking backward compatibility in a saner way, as with the smartmatch module example. This in turn frees up core developers to fix old design mistakes and introduce new features that might otherwise break old code.
Unicode work is also progressing. The 5.18.0 release will (hopefully) include support for set operations on Unicode character classes in regular expressions, as well as Unicode-related performance improvements.
While it's hard to predict the specifics of the future, I'm excited to see the activity and effort going into Perl 5 core development these days. The new release schedule, along with a move to Git, seems to have attracted some new contributors. Looking at the activity in the core and the community, Perl 5 is on a healthy path toward the future.
Brief items
Quotes of the week
If you think you know how to do something better, it's best to fork, work it out, and if you come up with something, then work to merge it back, if at all possible. If merging doesn't work, and it turns out that your stuff works better, people will migrate to it, keeping it alive.
Odds are, the fork will turn out to be a dead-end, and it will die off. But you will then know the reasons why, and not be so upset when others do things you disagree with.
That's the way evolution works, and it works quite well, it's why open source works as well as it does.
What did the (mostly closed source) competition do? It went into the exact opposite direction: Apple/iOS and Google/Android consist of around a hundred tightly integrated core packages only, managed as a single well-focused project. Those are developed and QA-ed with 10 times the intensity of the 10,000 packages that Linux distributions control. It is a lot easier to QA 10 million lines of code than to QA 1000 million lines of code.
Cinnamon 1.4 released
Cinnamon is a project started by Linux Mint, aimed at providing a GNOME2 experience using GNOME3 underpinnings. The project has announced the release of Cinnamon 1.4. The download page shows versions for Linux Mint 12, Ubuntu 11.10, Fedora 16, openSUSE 12.1, Arch Linux, Gentoo, and Frugalware. You can also grab the source from github.Crossroads I/O release 1.0.0
Crossroads I/O is a fork of the ZeroMQ messaging system. The Crossroads developers say:
Furthermore, the ZeroMQ community has also recently chosen to institute a light review process, which we feel is at odds with the technical quality and long-term goals we desire for the project.
The 1.0.0 release is the project's first; it includes a few API changes and a ZeroMQ compatibility mode.
Git project seeks discussion on "push" change
The Git project is considering a change to how the "push" operation works and is looking for comments from developers who may be affected by it. "In the current setting (i.e. push.default=matching), 'git push' without argument will push all branches that exist locally and remotely with the same name. This is usually appropriate when a developer pushes to his own public repository, but may be confusing if not dangerous when using a shared repository. The proposal is to change the default to 'upstream', i.e. push only the current branch, and push it to the branch 'git pull' would pull from."
H.264 support coming to Firefox
Mitchell Baker and Brendan Eich have both posted articles explaining the decision by the Mozilla project to add support for the H.264 video codec in Firefox if the underlying platform provides it. "What I do know for certain is this: H.264 is absolutely required right now to compete on mobile. I do not believe that we can reject H.264 content in Firefox on Android or in B2G and survive the shift to mobile. Losing a battle is a bitter experience. I won’t sugar-coat this pill. But we must swallow it if we are to succeed in our mobile initiatives. Failure on mobile is too likely to consign Mozilla to decline and irrelevance."
kconfig-frontends 3.3.0
Kconfig-frontends is a new project seeking to package the kernel's configuration system. "kconfig-frontends is not meant to replace the in-tree Linux kconfig (at least in the foreseeable future!), but really targets third-party projects that want an easy path to following the evolution of kconfig, without the burden to manually synchronise occasionally." The project plans to release shortly after each major kernel release.
KDevelop 4.3 released
Version 4.3 of the KDevelop development environment has been announced. New features include beginning C++11 support, better version control system integration, better integration with the projects.kde.org site, improved source formatting, and a number of performance improvements.notmuch 0.12
Version 0.12 of the notmuch email indexing system has been released. New features include reply-to-sender, better tagging operations in the Emacs interface, the ability to save attachments, Python 3.2 support, and more.xpra 0.1.0 released
Xpra calls itself "screen for X"; "it allows you to run X programs, usually on a remote host, direct their display to your local machine, and then to disconnect from these programs and reconnect from the same or another machine, without losing any state." Version 0.1.0, the first major xpra release, is now available. There is a long list of new features, most of which are performance- or security-related.
Newsletters and articles
Development newsletters from the last week
- Caml Weekly News (March 20)
- LibreOffice development summary (March 15; likely to be the last one)
- Perl Weekly (March 19)
- PostgreSQL Weekly News (March 18)
- Ruby Weekly (March 15)
Van Rossum: Python is not too slow (InfoWorld)
InfoWorld has a short interview with Guido van Rossum, the creator of Python. In it, he talks about Python 3, Unicode, the Global Interpreter Lock (GIL), and more. "At some point, you end up with one little piece of your system, as a whole, where you end up spending all your time. If you write that just as a sort of simple-minded Python loop, at some point you will see that that is the bottleneck in your system. It is usually much more effective to take that one piece and replace that one function or module with a little bit of code you wrote in C or C++ rather than rewriting your entire system in a faster language, because for most of what you're doing, the speed of the language is irrelevant."
Page editor: Jonathan Corbet
Next page:
Announcements>>