Kev009.com

Lucent 7 R/E 5ESS Telephone Switch Rescue

2024-07-03T20:52:00-07:00

I am still recovering from the fairly challenging logistical project of saving a Lucent 5ESS. This is a whale of a project and I am still in a state of disbelief that I have gotten to this point. Thanks to my wife, brother, and a few friends for their help and the University of Arizona which has a very dedicated and professional Information Technology Services staff.

It started when I saw some telephone history enthusiasts post about a construction bid at the University of Arizona. It turns out, U of A installed the 5ESS in the late 1980s in a rather forward thinking move that netted a phone system that handled the growth of the University, medium speed data anywhere a phone may be located (ISDN BRI or PRI), and copper and fiber plant that will continue to be used indefinitely.

At peak, it served over 20,000 lines. They've done their own writeup, The End of An Era in Telecommunications, that is worth a read. In particular, the machine had an uptime of approximately 35 years including two significant retrofits to newer technology culminating in the current Lucent-dressed 7 R/E configuration that includes an optical packet-switched core called the Communications Module 3 (CM3) or Global Messaging Server 3 (GMS3).

Moving 40 frames of equipment, this required a ton of planning and muscle. The whole package took up two 26' straight-trucks, which is just 1' short of an entire standard US semi-trailer.

Coming from the computing and data networking world, the construction of the switch was quite bewildering at first. It is physically made up of standard frames which are interconnected into rows not unlike datacenter equipment, but the frames are integrated into an overhead system for cable management. Internally, they are wired up usually within the row and quite a few cables route horizontally beween frames, but some connections have to transit up and over to other rows.

Line Trunk Peripherals hook up to a Switching Module Controller (SMC) directly or an OXU (Optical Cross Connect Unit) which hooks up to an SMC and reduces the amount of copper cabling going between rows. Alarm cables run directly to an OAU (Office Alarm Unit) or form rings in between rows that eventually end at the OAU. Optical connections go from OXUs to SMCs and then to the CM, copper test circuits home run to a Metallic Test Service Unit shelf. Communications Cables come out the top and route toward the wire frame, usually in large 128 wire cables but occasionally in smaller quantity for direct or cross connect of special services. A pair of Power Distribution Frames distribute -48V throughout the entire footprint, taking into account redundancy at every level.

All of this was neatly cable laced with wax string. Moving a single frame required hundreds of distinct actions that vary from quick, like cutting cable lace, to time consuming removal of copper connections and bolts in all directions.

We were able to complete the removal in a single five day workweek, and I was able to unload it to my receiving area in two days over the weekend where it now safely resides.

The next step will be to acquire some AC and DC power distribution equipment, which will have to wait for my funds to recover.

I should be able to boot the Administrative Module (AM), a 3B21D computer, up relatively soon by acquiring a smaller DC rectifier and that alone will be very interesting as it is the only use I know of the DMERT or UNIX-RTR operating system, a fault tolerant micro-kernel realtime UNIX from Bell Labs.

The system came with a full set of manuals and schematics which will help greatly in rewiring and reconfiguring the machine. After the AM is up, I need to "de-grow" the disconnected equipment and I will eventually add back in an assortment of line, packet, and service units so that I can demonstrate POTS as well as ISDN voice and data. In particular, I am looking forward to interoperating with other communication and computing equipment I have.

I will have to reduce the size of the system quite a bit for power and space reasons so will have spare parts to sell or trade.

Additional Pictures are available here until I have a longer term project page established.

This is too much machine for one man, and it is part of a broader project I am working on to build a computing and telecommunications museum. If you are interested in working on the system with me, please feel free to reach out.

FreeBSD on PowerNV

2017-12-06T18:09:00-07:00

At the Netflix vendor summit I gave a talk about FreeBSD on PowerNV, the bare metal firmware of POWER8 and POWER9.

PDF slides

The Art of Software Development Management

2016-10-14T02:24:00-07:00

A fallacy in software development and management thereof is the idea that software developers are fungible.

This is not intuitive for folks to reason about because programming does entail at a fundamental level accounting and bookkeeping. Put this into a container, grab this lock in this order, call this function with these parameters, free this file handle when done with it. We are describing contracts for execution and flow for data.

Why then is software development not a simple matter of accounting? Because that is the easy part. It has defined boundaries and someone will either succeed or fail in finding them. A program either compiles or does not. The software encounters runtime errors or runs through. It has incorrect logic or does what is expected by the user.

If one person is working on the software and is the sole user, as long as they understand it and can use it, all is well.

Efficiency and time to delivery begin to part from these true and false answers. Everything breaks down when more people are added or the software outlasts the original developer's grasp.

What now dominates the difficulty is not the objective truths of algorithms, but the subjective qualities of their implementation and surrounding infrastructure like maintainability, composition, documentation and deployment. This is governed by people. People understand, in vector space rather than Boolean, the intent of a function, module, library, piece of software, and entire system. People, in vector space, have the ability to explain the implementations, the trade offs, the false starts and the success. People, in vector space, have the ability to effect changes, and at vastly different rates. Developers can negate progress and value.

These vector space attributes are why software developers are not fungible.

When someone works on a piece of software long enough, they have a vision for what it should be and a history of what the problems have been along the way.

Someone who is well read and has written similar sized systems will have a faster rate of learning and confidence on a new code base.

Someone with a sense of ownership and quality will imbue quality in their work. Another, with little interest in the task at hand will not, because quality is not accidental.

Are the folks maintaining the software able to work by, with, and through any obstacle they encounter? Have they proven the ability to predict and meet deadlines? Can they bring new people in? Might someone leave and the overall direction be maintained? Is the organization willing to remove code and features? These compose shared values.

In order: a developer, team, organization, company, product and community carry subjective vector space qualities. Comparisons between developers, teams, organizations, companies, products and communities are increasingly more difficult in like order.

It is foolish to think any one of these are a simple interchange with so many continua. The dynamics are more akin to organ transplant.

Yet a grandmaster of software development has intuition for evaluating the performance of these different scale actors, selecting and nurturing high performance. This is the Art of Software Development Management.

Few people live it. Everywhere I've been has retained negative performance on teams because someone just might do something useful eventually, they are here and headcount is hard to get, or the employee is just a nice person without consideration for the consequence. I've seen enough entire teams that would do better to be on a permanent paid vacation than working on product for a company. Every company I know of pays developers at opposite ends of the overall vector a low multiple difference at most, while a management chain that undermines The Art of Software Development Management to the point of institutional devalue may easily earn two and three orders of magnitude more.

When we look at human performance in sport, it is somehow obvious even to the layman. It would be cruel to swap children in on an NFL offensive line. Yet in software development organizations, this kind of wholesale swapout happens all the time with largely the same result - the entire product gets battered, frustrated, and losses result. A more empathetic strategy for the gradient of performance is to have separate leagues and deliberate transition points. Mixed performance can happen by exception only on a very mature team, with a product that is stable enough so mentorship can occur without stressing the mentor or mentee.

This is largely why commercial software development quality is abysmal. I don't know how to effect change en masse from where I stand now, but I am doing my best to live to it.

Software Quality and the Table Saw

2016-10-14T01:35:00-07:00

Consider the table saw. If you are not familiar with woodworking, this is a type of shop saw. A circular blade comes out from a table surface. It can be adjusted with a decent amount of precision to make cuts of a specified depth and angle.

One might naively think this piece of equipment is easy to use. Dial in the settings and push the stock through. In the best case, you end up with sloppy cuts. In the worst case, you incur bodily harm.

The skilled worker will understand kickback and feeding all the way through. They will be mentally prepared to use the equipment, ensuring the space is clean, dust collection is working, that they have no loose clothing or jewelry to cause themselves danger.

A seasoned woodworker will build "jigs" - small projects designed to brace and feed stock through the table in a precise and safe manor. They will know when their job is complete, and when to throw away scrap and try again.

The tools available to build software today are about as complicated and usable as this table saw. An experienced software developer is demonstrable by their jigs and willingness to throw away swaths of code. The stakes are different, but too many developers do not bring to bear a respect for the tool, nor the harm they cause to other developers and users with sloppy use because the reaction time and blame assignment are so far removed.

FreeBSD UEFI Root on ZFS and Windows Dual Boot

2016-07-29T06:30:00-07:00

Somehow I've managed to mostly not care about UEFI until now. On my new laptop, I decided I should give it a go. There are some small benefits, nothing life changing, but booting multiple OSes is a lot easier especially if they are UEFI native, and you get a nice frame buffer the boot manager can use as will the OS before starting graphically (and after, if you don't have accelerated graphics drivers).

For reference, how I run FreeBSD desktop/laptop: digital-life

Thanks to

Most of this was cribbed from the following sources:

Eric McCorkle, Steve Hartland and others for adding ZFS boot and a ton of other improvements (GELI) to the FreeBSD UFI loader
Trond Endrestøl's blog, for mentioning refind and the overall UEFI landscape on FreeBSD
Calomel, for a decent overview of manual ZFS on root installation
/usr/src/usr.sbin/bsdinstall/scripts/zfsboot for some ZFS specifics
An imaging script my colleague Jason Wolfe did for ZFS and boot envs at work

Install Windows 10 or other UEFI OS

It's easiet if you install any other co-habiting OS first. Most OS installers assume they own the entire computer, and don't let you know much about what they are really doing, especially when manipulating booting.

Windows creates a large 100MB EFI partition, plenty of room for refind and other boot loaders.

Leave free space during the installer or shrink the partition using Windows Disk Manager.

Boot into a FreeBSD 11+ live environment

We just need a live FreeBSD enviroment to conduct our manual install. Make sure it is 11.0 or newer for UEFI boot1 zfs support.

The USB images with FreeBSD 11.0 and later -CURRENT snapshots have UEFI support integrated so they are directly bootable on UEFI machines. You could also use a CD/DVD or netboot.

Enable sshd, if needed

If you want to copy/paste from this blog to the machine being installed, bring up SSH.

mkdir /tmp/etc /tmp/root
mount_unionfs /tmp/etc /etc
mount_unionfs /tmp/root /root
echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config
passwd
service sshd onestart

Bring up a network interface

We'll need to grab refind during the installation.

Get a dhcp lease on your NIC or see the handbook for wireless setup

dhclient em0

Partition the drive

Add a couple GPT partitions. I'm doing a non-ZFS swap so I can coredump the kernel when doing FreeBSD development.

gpart add -a 4K -l swap0 -s 16G -t freebsd-swap nvd0
gpart add -a 4K -l zfs0 -t freebsd-zfs nvd0

Create a 4k aligned zpool

Standard practice these days, 4k align everything even if it's not a 4k-native disk.

Create a mountpoint and the inital zpool.

kldload zfs
sysctl vfs.zfs.min_auto_ashift=12
mkdir /tmp/zroot
zpool create -f -o altroot=/tmp/zroot -O compress=lz4 -O atime=off -m none zroot /dev/gpt/zfs0
zpool export zroot

Boot environment compatible ZFS datasets

Nest the root dataset under ROOT so we can use boot environments in the future with beadm

zpool import -o altroot=/tmp/zroot zroot
zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
zfs create -o mountpoint=/usr -o canmount=off zroot/usr
zfs create zroot/usr/home
zfs create -o setuid=off zroot/usr/ports
zfs create -o mountpoint=/var -o canmount=off zroot/var
zfs create -o exec=off -o setuid=off zroot/var/audit
zfs create -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/log
zfs create -o atime=on zroot/var/mail
zfs create -o setuid=off zroot/var/tmp
zpool set bootfs=zroot/ROOT/default zroot
chmod 1777 /tmp/zroot/tmp
chmod 1777 /tmp/zroot/var/tmp

Perform a manual install of the distribution

This is pretty easy.

cd /tmp/zroot
ln -s usr/home home
tar xvJpf /usr/freebsd-dist/base.txz
tar xvJpf /usr/freebsd-dist/lib32.txz
tar xvJpf /usr/freebsd-dist/kernel.txz

Set a few things up

Set some common configuration. You may also wish to set up networking, enable SSH, etc in the altroot rc.conf.

echo 'zfs_enable="YES"' >> /tmp/zroot/etc/rc.conf
echo 'dumpdev="AUTO"' >> /tmp/zroot/etc/rc.conf
echo 'powerd_enable="YES"' >> /tmp/zroot/etc/rc.conf
echo 'sendmail_enable="NONE"' >> /tmp/zroot/etc/rc.conf
echo 'zfs_load="YES"' >> /tmp/zroot/boot/loader.conf
echo 'kern.geom.label.disk_ident.enable="0"' >> /tmp/zroot/boot/loader.conf
echo 'kern.geom.label.gptid.enable="0"'  >> /tmp/zroot/boot/loader.conf
printf "/dev/gpt/swap0\tnone\tswap\tsw\t0\t0\n" >> /tmp/zroot/fstab
tzsetup -C /tmp/zroot
chroot /tmp/zroot/ passwd

Install refind

UEFI has lots of bells and whistles. We're going to use the refind boot manager. I'm relying on the "fallback" efi loader, bootx64.efi. You may need to toggle things around in your system's firmware for that to work, or teach the EFI NVRAM about refind. See the refind site for more details.

cd /tmp
fetch http://downloads.sourceforge.net/project/refind/0.10.3/refind-bin-0.10.3.zip
unzip refind-bin-0.10.3.zip
rm refind-bin-0.10.3.zip
mkdir /tmp/efi
mount_msdosfs /dev/gpt/EFI%20system%20partition /tmp/efi/
cd /tmp/efi/EFI/Boot
mv bootx64.efi bootx64-windows-10.efi
cp /boot/boot1.efi bootx64-freebsd.efi
cp -a /tmp/refind-bin-0.10.3/refind/icons .
cp -a /tmp/refind-bin-0.10.3/refind/refind_x64.efi bootx64.efi
cp /tmp/refind-bin-0.10.3/refind/refind.conf-sample refind.conf

As good hygiene, you might consider updating bootx64-freebsd.efi whenever a point release is done. You could also keep an eye out for refind updates. This is easily done since it's just a DOS filesystem.

Configure refind and add menu entries

Set the values of timeout, and scanfor to manual to speed things up a bit in refind.conf.

Then add a couple entries:

cat << EOF >> refind.conf

menuentry "FreeBSD/amd64 -CURRENT" {
    loader \EFI\Boot\bootx64-freebsd.efi
    icon \EFI\Boot\icons\os_freebsd.png
}

menuentry "Windows 10 Professional x64" {
    loader \EFI\Boot\bootx64-windows-10.efi
    icon \EFI\Boot\icons\os_win.png
}
EOF

Finish, reboot and enjoy!

That's it. Unmount the efi partition and reboot.

cd
umount /tmp/efi
reboot

You should be greeted by refind, otherwise take a look through your firmware boot order and make sure the firmware nvram for Windows Bootmanager isn't first.

GELI

Keep an eye out for GELI full disk encryption on top of ZFS on root.

Don't forget to do the swap partition as well.

Do one thing

2015-12-11T10:46:00-07:00

Do one thing per day. Make that thing meaningful. Commit a bugfix. Land a feature. Close a sale. Apply security patches. Write a blog post.

Do this and you will be more productive than most people.

Don't just go to meetings, and especially don't just "work". If you are currently "working", you are in a rut, your brain is idle, and you are not doing. You do not have a clearly defined objective in your mind's eye.

Goodbye Wordpress, hello Pelican

2014-04-06T07:40:00-07:00

I spent some time converting my Wordpress blog over to Pelican. Overall, it wasn't effortless and required quite a lot of manual clean up of links and images. But, I'm a pack rat and want to preserve my writings even if they are dated or silly now, so I pressed on.

But it's for the better. It's very comforting to ditch the massive security vulnerability of Wordpress. I no longer have any MySQL apps on any of my servers which makes administration simpler with just Postgres.

Static blog generators seem so blindingly obvious in retrospect.

I migrated comments to Disqus, which I'm more ambivalent about. On one hand, dealing with spam is tough and outsourcing basic protection is nice. On the other, I'm a fan of self-hosting and self-sufficent systems.
Disqus could become undesirable at any point in time. With an export, it wouldn't be too much effort to self-host those again though.

Speaking of retrospect, I launched this blog over seven years ago. Crazy how time flies.

Useful references:

Armed to the gills - Dell C6100 build out

2013-12-18T12:49:00-07:00

The fine folks over at ServeTheHome have done a fantastic job documenting the Dell C6100. This is a rather interesting system because:

It packs quite a bit of compute power into 2U for colocation
They are surprisingly cheap on the secondary markets

I'll lead you over to the ServeTheHome blog posts which deliver a nice overview:

My ideal config is a redundant set of servers running border services including routing, NAT, firewall, VPN and a redundant set of servers running a variety of applications, VMs, etc.

The C6100 comes in either a 12 disk 3.5" chassis, or a 24 disk 2.5" chassis. The disks are factory split between the systems in either three 3.5" per node, or six 2.5" per node.

My border networking nodes don't need a lot of disk space. In fact, they don't even really need RAID thanks to the magic of PF and CARP. So preferably I don't want to waste the hot swap bays on them. There are several options here.. PXE or iSCSI boot, USB thumb disk boot, or figure out how to cram some storage inside the unit. Turns out the last option isn't that hard if you need traditional local storage.

Using info from the ServeTheHome forums, I built a 5V power tap. I bound the two 5V rails and two grounds from the internal USB header to a donor 4-pin Molex to SATA power converter. I recommend sticking with a low power SSD like the Crucial M500 or certain Samsung units as this is stretching the USB-standard power envelope a bit at .5A per port.

Dell c6100 5v USB to SATA power tap

Here's a parts list per node:

Qty: 4 - Molex 12" picoblade precrimped wire - Mouser link
Qty: 1 - Molex 8 position picoblade connector - Mouser link
Qty: 1 - SATA power connectors salvaged from a 4-pin Molex converter
Qty: 1 - 6" SATA cables

Dell c6100 internal SSD storage

I mounted the internal SSDs with a stack of automotive trim tape on top of the Southbridge heatsink. A block of foam usually sits here to support mezzanine cards so I'm not too concerned. If need be, I can cold swap these disks without too much trouble while the other nodes continue to run.

This in turn frees up the hot swap bays to be split between the two app nodes. This is great because six 3.5" disks gives me the right balance of flexibility, capacity, performance, and cost. Rewiring this is a bit of a chore. You need to order two SFF-8087 to 4x SATA 7pin breakout cables (check Monoprice). You can reuse the two original cables with some creative wiring (altering the numbering of the front bays so they are right-to-left numbered), or purchase four of the aforementioned cables for correct numbering.

Full c6100 loadout

If you're running SSDs in an array, I recommend hunting down the LSI SAS2008 based "XX2X2" mezzanine card which runs at 6gbps and supports >2TB disks. You need various bits to install this, including new SATA cables, PCI-E riser, and metal brackets. At the time of my purchase, the easiest way to get this was buying the older "Y8Y69" mezzanine card with all those bits and buying a bare "XX2X2" card.

These hacks probably don't make much sense in all settings as they're fairly time consuming. If you have the budget, the 24 2.5" disk chassis is the way to go. But for personal use, this is a great build for me!

Reusable Pagination in Play! 2

2012-12-04T13:42:00-07:00

On the Play! Framework mailing lists I've seen reference to the sample Computer Database as a canonical example of paginating data. It's a good start but it's pretty specific to one data type. Follow along and we'll make a more general utility.

If we think abstractly about paginating a web application using Model-View-Controller:

M - A way to filter the dataset
V - A UI element for displaying the pagination and linking to others
C - A way to get Request parameters that define the page, the length of the page, and other filters

A screen shot of what we will be creating

Model

The specifics here depend upon how you are retrieving the data to display.

Squeryl provides a page(offset, pageLength) method that uses the DB's LIMIT and OFFSET. I use this to create a subset collection that I pass to view for iterating over. I also have a helper method to get a total data count.

Controller

Play's route and reverse routes take care of passing the page number around:

GET  /list       controllers.Notifications.list(page:Int=1)
GET  /list/help  controllers.Notifications.help
# NOTE: /list/:page MUST COME AFTER /list/[string] due to route priorities
GET  /list/:page controllers.Notifications.list(page:Int)

We get pretty URLs like /list /list/2 list/3 etc. Note that @routes.Notification.query(1) will result in /list, which is a nice touch. If /list/1 is your preference you may consolidate to one route. Also note that route priority (order) may affect things depending on the URL parameters you are using.

In the application controller, you need to pass an offset and pageLength to the Model. This returns a collection of a page worth of items.

Then, pass it all through to the view:

  def list(page:Int) = withUser { user => implicit request =>
    val pageLength = 10
    val notifications = Model.getNotifiactionsByUser(user, (page-1)*pageLength, pageLength)
    val count = Model.getCountByUser(user)
    Ok(views.html.notifications.index(notifications, count, page, pageLength))
  }

View

In our list view, we pass some variables through and iterate over the filtered collection of data to display. Then, we call a helper which creates the pagination UI element. The biggest thing to note here is the use of Scala's first class functions, specifically partial application, to delegate the page parameter to the paginate helper:

@(notifications:List[models.Notification], count:Int, page:Int, pageLength:Int)
  @for(n <- notifications) {
    <li>@n</li>
  }
  @includes.paginate(page, pageLength, count, routes.Notification.index(_))

View helper

"I am sorry I have had to write you such a long letter, but I did not have time to write you a short one" -- Blaise Pascal

This template code is pretty awful. I may refine it on the gist if time permits. Please comment if you have suggestions!

We take the page we're on, the items per page, the total query count, and the partial route and build a UI wiget. The lowbound and highbound helpers functions define how many pages to link.

Conclusion

I'm very interested in your take, as well as ways to clean this up! I'll update the post with good suggestions.

Playing with Play Framework and Plivo

2012-12-01T11:03:00-07:00

One of the interesting things about Scala is how well it composes. This is the primary "Scalable" in Scalable Language. I'm writing this up to share my experience and track my progression in a partially contrived but interesting example.

So I'm playing around with Plivo, which is a telecoms IaaS web service (similar to Twilio), and the Play! Framework. It uses a RESTful API, and we want to make sure the API callbacks are authentic. HMAC as usual.

This is your brain on Functional Programming

Scala makes it very easy to break up your problems into functions. Rather than loops with counters, mutable variables, and such, Scala encourages (but doesn't mandate) a functional style with immutable variables, first class functions, nested functions, etc. It generally leads to code that is both more concise and contains less bugs.

Nested functions

Play! provides an HMAC implementation (play.api.libs.Crypto.sign), but it converts the signature into a Hex ASCII string. Plivo Base64 encodes the HMAC byte (non-ASCII) signature. We have a few options to get the keys into equivalent formats. I chose to make some nested helper functions and encode my signature in their format. Since the functions are unlikely to be used elsewhere, the nesting keeps the namespace clean and locally reenforces that we have a specific format.

for comprehension (syntactic sugar around map)

One of the interesting things here is the use of Scala's for comprehension as a control structure, a technique I learned on the Play! mailing list. In the body, we have two variables that return Option type. If both are defined, the yield block fires with the params defined directly. Otherwise, the getOrElse block fires and handles errors.

List fun[ctions]

Play! returns the url-encoded POST data as a Map[String, Seq[String]] (because forms keys can have multiple values, i.e. check boxes). Plivo's signature is based on the full URL of the callback, and all the POST parameters, sorted by key, concatenated together.

I convert the parameter Map into a List[String, Seq[String]] (List of Tuples) because Maps don't have a sort operation. We'll soon be flattening it anyway, so the List is a pretty good structure (aside.. another approach superior in more general circumstances would be to convert the Map to a TreeMap which is always sorted)

The _ is an interesting character in Scala and is used in a few different ways. Programming in Scala calls it, in passing, "filling in the blank" which generally holds true. Ultimately, we are "filling in" the sortBy with the key of our List. Finally, we flatten the list, providing a flatten with a means of combining the key and the Seq of values. mkString works perfectly here since it just concatenates everything. x => indicates x as a placeholder for each of the List items and is analogous to _, which won't work here since we need to reference it twice.

OO Design tradeoffs

So I'm using Action Composition to take a Request, validate the HMAC key, and first class functions to pass on Response responsibility or otherwise respond with BadRequest if there is a problem with the signature or params.

A more Java-influenced design might throw Exceptions on error and in the delegated chain of command. This has the benefit of being more abstract.. if this function isn't called in response to an HTTP request, the invoking Exception handler could do something other than responding BadRequest as we do here. The cool thing about Scala is that I could just as well do that, but I make a more opinionated choice since I know how my class will be used in all cases.

Scala keeps OO, but turns the use on it's head and makes it always possible to ask: "gee, does this REALLY need to be abstracted?" More often than not, a more elegant and concise answer is to reach in the functional bag of tricks.

Zabbix 1.8.9 Debian Squeeze Backport

2012-01-18T20:18:00-07:00

I was beginning to get hit by many bad things in the Debian Squeeze zabbix 1.8.2 package. If you aren't aware, zabbix is a nifty data center monitoring system and is only slightly annoying compared to most other systems which are very annoying to set up and use.

Most notably, this package will safely run on PostgreSQL 9.1 from squeeze-backports and contains many performance improvements. It should be a drop in upgrade for the distro package.

Get it here:
http://kev009.com/files/zabbix-1.8.9-squeeze.tar.gz

Configuration Management Software Sucks

2012-01-10T06:14:00-07:00

Yes. Configuration Management Software Sucks. Horribly.

The main problem is that n-th order tweakability is preferred over convention It's just stupid. There are a core set of things that just about everybody needs to do. Those should be dead simple. Ready to uncomment and run. The set operating systems used in the enterprise is fairly small: RHEL5, RHEL6, Debian 6, Ubuntu LTS. A configuration system should be opinionated and have complete out of the box support for these platforms. Simple rulesets for the basics that nearly everyone uses should be ready to go.. package management, process initialization, file management, ssh, sudo, DNS, Apache, PAM, PostgreSQL, MySQL, OpenLDAP, etc. Keep it simple. Keep it simple. Keep it simple. Resist all urges to add complexity.

That's not the case.

You'd think after 30 years of Unix, BSD and Linux network deployments this would be pretty well trodden ground. Wrong. It's a complete crapshoot and everybody does things differently. Pick your poisons and reinvent the stack ad infinitum.

This is one of the few areas I'm green with envy of the Microsoft side of the fence. Between Active Directory, Group Policy, and maybe a third party tool or two for cloning and installs and such, Microsoft environments can easily be set up and managed well by complete morons (and often are).

Puppet

Puppet seems to have potential. Of course, out of the box you're pissing in the wind with a blank slate and most books and sites will have you following tutorials to rewrite rulesets that thousands of other people before you have similarly cobbled poorly together. As a Ruby project, it unsurprisingly has vocal hipster fanboys. Unfortunately, they forgot to parrot their DRY principle to each other.

It centers around a domain specific convention which isn't so bad.. but in no time flat you'll start seeing full blown Ruby programs intermingled. Ugh. But it's not so bad if you stick to the basics.

If you look around you can find reasonably complete module sets, i.e. http://www.example42.com/. It's not all gravy as these are heavily interdependent and kludgy. If you want a clean, simple solution you're back to rolling your own with some healthy copy and paste.

Since it's a Ruby project, aside from the annoying fanboys, you're also going to run into scalability problems past a few hundred nodes. There are mitigation strategies, but it's a joke compared to something like Cfengine.

Due to hype, you'll find decent versions in the Debian and Ubuntu backports repos. RHEL 5 and 6 are covered by a Puppet Labs repo. 2.6 and 2.7 are therefore readily available and as long as your master is running the later version you shouldn't have interop problems.

All things considered, Puppet is probably the best choice at the moment. It sucks, but it's got a lot of momentum behind it. There are mountains of docs, books, and tutorials to get you going and nothing is too foreign or hard to grasp.

Cfengine 3

I really want to like Cfengine. It's incredibly light weight and hardcore ROFLscale. It's got serious theory behind it and older versions have been used in massive deployments. But it's not just a blank slate. It's even lower level and incomplete compared to the others.

You really need to add a promise library to get features that should be included by default. These are all stagnate though, and still leave much to be desired.

There's a company behind it doing something or another, but the open source version is raw. If you have more than one Linux distribution, I'll pretty much guarantee the packages are incompatible.

The repo choices aren't great either. Uncle Bob's PPA on Ubuntu, out of luck on Debian. RPMs in the EL repos look out of date. You can of course get source and binaries from the Cfengine company, but it's not my preferred way to install things and makes bootstrapping harder than it needs to be.

I haven't tried the latest release, but quickly gave this one up when I found severe incompatibilities between point releases. Madness. You'd think people inventing something like promise theory could handle something as simple as version stability.

Ping me when a corporation backs Cfengine with a good promise library, some standard tasks, and repos for the common operating systems.

Bcfg2

Bcfg2 made the most sense to me out of the box. XML is yucky and out of fashion these days, but Bcfg2 manages to use it acceptably. Consequently, most things are declarative, easily read, and overall easy to mimic. Beyond that, you can tap into some Python template and generator stuff. But yes, these guys finally didn't put n-th order above the common cases! Installing packages and ensuring services are on is a snap.

They've got their own repos for many distros so installation isn't bad.

The client and server are Python so you'll have similar scaling problems to Puppet in large environments.

My biggest grievance with Bcfg2 is that the server needs intimate knowledge of each operating system version's package repos. You'll fumble around writing a good bit of XML definitions for this in a heterogeneous environment.

The main thing Bcfg2 is lacking right now is community momentum. Including repo definitions by default and some more doc work.. I think this would be a great system for small to medium deployments.

Conclusions

The lot of this stuff is really terrible. End to end system management under *nix is a major pain point. On top of this, you'll need a fairly free form monitoring framework (these also all suck) and directory service. Mix and match an impossible array of projects and eventually you'll find your own recipe that sort of works. Except everyone does it differently so you'll constantly be learning and redoing the same things over and over anyway.

It's not fun. What we need is end to end integrated thinking. This area is still ripe for picking. Oh RedHat, where art thou?

GUNNAR Optiks Computer Glasses Review

2011-07-19T23:00:00-07:00

Intro

As one might expect from a computing professional, I spend a lot of time in front of LCD screens. This is coupled to the modern assault of Compact Florescent Lights (CFL). These are both heavy on the blue spectrum. Depending on what you read and believe [1], this can have a number of health implications.

For the past few weeks, I've been swapping yellow lenses into my sport sunglasses while at the computer. I've been pleased with the result and noted an easier time going to bed at night and subjectively my eyes feel less tired.

Swapping the lenses so often became a bit tedious. I had to clean them every use. I began looking for a dedicated pair of glasses. GUNNAR Optiks has a decent marketing machine that swooned me into purchasing their purpose-built product. I was eager to try them out and see if there were any benefits to the specialty computer glasses.

Pros

Amber tint reduces blue shift as expected.
Look decent. I would comfortably wear these at my desk in an office environment. The GUNNAR lens coating gives off a bluish reflection that tones down the impact of the yellow lens to onlookers.
Comfortable frame. No problems here, I could wear the frames all day.

Cons

Magnification. Deal breaker. I wasn't expecting a product like this to alter my eye's focus, especially without ample warning. I wasn't sure about the magnification until reading another review [2].
Quality. Chinese made, feel flimsy, and other reviews say the finish can flake off.
Cost. For the relative quality, these are retailing for 2-3x the price they should be.

Conclusion and Verdict

I'm not afraid to spend money a decent product, even if a good chunk of that is marketing and brand name. However, the GUNNARs fall short in both quality and execution and I would advise looking elsewhere.

The magnification is a deal breaker for me. I don't need it, I'm not used to it, and it gave me a headache. I find it borderline negligent [3] to not mention this beyond fine print about "diopters" (note: I'm not an Rx wearer; this wasn't a familiar term) on the bottom of the packaging. It made taking eyes off the monitor and looking around awkward -- a tried and true ergonomic tip.

I'm going to stop by an optometrist and see if they can do a neutral real glass lens (superior optics and scratch resistance) with a slight yellow or orange tint and anti-reflective coating for a reasonable price. Alternatively, sport, safety, or shooting glasses from a reputable vendor can be cheaper than the GUNNARs and do a fine job. Prescription glasses wearers should consult their optometrist but I saw passing discussion that the anti-reflective coatings are great and of course they have the expertise to adjust the power and focal length directly to your needs.

The GUNNARs are getting returned.

Something good about every language I used in 2010

2011-01-01T03:38:00-07:00

Inspired by Samuel Tardieu's post, I want to do a year in review of all the languages I have used this year. A lot of times we prima donna programmers complain about anything and everything. I really enjoyed the positive outlook of Samuel's post and want to take note of my experiences with similar attitude.

Bread & Butter

Python - My language of choice for the year. Whether prototyping, experimenting, developing a Facebook application, maintaining a test framework I wrote for my workplace, or implementing cryptographic algorithms for a security course, Python continued to serve me well. Between a copy of "Python Essential Reference" and PyPI, I feel there are very few problems beyond my means thanks to the power of this beautiful language and its surrounding community.
Java - As a student, I pounded out many a line of Java throughout the wee hours of the morning in my capstone classes. Java seems to be the New Age language of academia and I can speak it universally to my classmates and professors as a lingua franca. I've noticed that my Java programs tend to structure themselves well without much effort thanks to the strong object influence forced by Java and expansive standard library and Collections classes. Also used at two collegiate programming competitions which is an entirely different experience than normal software development where the large standard library again came in handy.

Good Progress

C - I launched a good sized networking project in C as my first big project in the language and contributed a number of portability fixes to the libevent project. Fast to compile, fast at runtime, and full low level control, C is a great language for Unix Systems Programming. I greatly expanded my knowledge of the POSIX interfaces this year and really enjoy programming at this level. I've noticed that some principles from other higher order languages have rubbed off on my C style; namely, data hiding and well formed/adaptable interfaces (see the post right before this one).
C++ - Been putting this one off because of all the FUD and intimidation at the sheer size of it. C++ is pretty much the Latin of our field and is used in everything from safety-critical Jet aircraft systems, to GUIs, to games, to JITs, to cutting edge research. As some of the pundits say, C++ is the language for "Demanding Applications". If you consider Java as the Flight Engineer of a large aircraft, C++ is definitely in the Pilot seat. You have full control and high visibility of what is going on, but if you aren't careful you can crash and burn. I've probably progressed to the advanced beginner stage where I can use it as a better C but haven't endured the trials and tribulations of an expert in the art of C++, nor read important references like Scott Meyers' "Effective C++" series. I really like the power and efficiency of the STL and plan on knowing enough C++ to use it when called upon.

Breaking New Ground

VHDL - After a required Electrical Engineering course, I was exposed to the entirely different paradigm of programmable hardware (FPGAs). This was an eye opening experience. Fundamentally, digital design is concurrent. There may be valuable lessons here for both academic and professional Computer Science and I need to explore more here. In 2011, I'd like to buy my own FPGA development board and work through the design of a simple CPU to gain further appreciation of hardware and VHDL or Verilog.

Back Burner

PHP - The first language I seriously learned and used some 12 years ago (I dabbled in Perl before that at the ripe age of 8, and probably Lego Logo a year before that :-P ). I've been keeping an eye on it and it seems some of the Framework movement that stole a lot of developers away to other languages has sprouted mature analogues in PHP land. No longer just C for the web, PHP 5.3 continues the lineage of the 5-series as a serious object-oriented language for web development that is basically universally available and dead simple to scale. The extent of my PHP coding in 2010 was limited to maintaining some programs I'd written in years past (aside from merely installing/using PHP products like this blog).

On to 2011

D - D2 has me really excited. For some intents and purposes, it seems like an evolution of C++ with a healthy removal of backward compatibility. Embracing fast compile times, integrating concurrency and message passing, allowing easy interfacing to C libraries, and more mean this is a language capable of "Demanding Applications". Perhaps most intriguing is the use of the language proper for metaprogramming and compile-time programs. I have Andrei Alexandrescu's book on my shelf and have thumbed through it a few times. The fact that he is involved speak volumes of D's potential and his book looks superbly written. 2011 means working my way through the book and working on at least one sizable project in D.
Erlang - Erlang has been on my radar for a couple years now. The fact that the OTP has roots in the demanding and critical realm of telecom means this is a serious language and seems to deliver interesting take on concurrency. Erlang has already proven itself effective for XMPP servers and Message Queues. This may yet be one of the best languages around for scalable networking applications and I'd like to get some hands on experience with it in 2011.
Haskell - I don't know much about Haskell other than playing around with TryHaskell. What I do know is that Haskell has a fairly mature Software Transactional Memory and that alone interests me. I've also heard the optimizing compiler is pretty good. Through investigation is due in the second half of the year.

No Nonsense Logging in C (and C++)

2010-12-09T13:08:00-07:00

A lot of times people do zany things and try and reinvent wheels when it comes to programming. Sometimes this is good: when learning, when trying to improve state of the art, or when trying to simplify when only Two-Ton solutions are available.

For a current daemon project I need good, fast, thread-safe logging. syslog fits the bill to a tee and using anything else would be downright foolish -- akin to implementing my own relational database. There's one caveat. For development and debugging, I'd like to not fork/daemonize and instead output messages to stdout. Some implementations of syslog() define LOG_PERROR, but this is not in POSIX.1-2008 and it also logs to both stderr and wherever the syslog sink is set. That may not be desired.

So, the goals here are: continue to use syslog() for the normal case as it is awesome, but allow console output in a portable way. Non-goals were using something asinine like a reimplementation of Log4Bloat or other large attempt at thread-safe logging from scratch.

Using function pointers, we can get a close approximation of an Interface or Virtual Function of Object Oriented languages:

void (*LOG)(int, const char *, ...);  
int (*LOG_setmask)(int);

These are the same parameters that POSIX syslog() and setlogmask() take. Now, at runtime, if we desire to use the the "real" syslog:

LOG = &syslog;  
LOG_setmask = &setlogmask;

If we wish to instead log to console, a little more work is in order. Essentially, we need to define a console logging function "inheriting" the syslog() "method signature" (or arguments for non-OO types).

/* In a header somewhere */  
void log_console(int priority, const char *format, ...);  
int log_console_setlogmask(int mask);

And finally, a basic console output format:

/* Private storage for the current mask */
static int log_consolemask;

int log_console_setlogmask(int mask)
{
  int oldmask = log_consolemask;
  if(mask == 0)
    return oldmask; /* POSIX definition for 0 mask */
  log_consolemask = mask;
  return oldmask;
}

void log_console(int priority, const char *format, ...)
{
  va_list arglist;
  const char *loglevel;
  va_start(arglist, format);

  /* Return on MASKed log priorities */
  if (LOG_MASK(priority) & log_consolemask)
    return;

  switch(priority)
  {
  case LOG_ALERT:
    loglevel = "ALERT: ";
    break;
  case LOG_CRIT:
    loglevel = "CRIT: ";
    break;
  case LOG_DEBUG:
    loglevel = "DEBUG: ";
    break;
  case LOG_EMERG:
    loglevel = "EMERG: ";
    break;
  case LOG_ERR:
    loglevel = "ERR: ";
    break;
  case LOG_INFO:
    loglevel = "INFO: ";
    break;
  case LOG_NOTICE:
    loglevel = "NOTICE: ";
    break;
  case LOG_WARNING:
    loglevel = "WARNING: ";
    break;
  default:
    loglevel = "UNKNOWN: ";
    break;
  }

  printf("%s", loglevel);
  vprintf(format, arglist);
  printf("\n");
  va_end(arglist);
}

Now, if console output is what you desire at runtime you could use something like this:

LOG = &log_console;  
LOG_setmask = &log_console_setlogmask;  
LOG_setmask(LOG_MASK(LOG_DEBUG));

LOG(LOG_INFO, "Program Started!");

In about 60 lines of code we got the desired functionality by slightly extending rather than reinventing things or pulling in a large external dependency. If C++ is your cup of tea, it is left as a trivial reimplementation where you can store the console logmask as a private class variable.

Some notes:

You should still call openlog() at the beginning of your program in case syslog() is selected at runtime. Likewise, you should still call closelog() at exit.
It's left as a trivial exercise to the reader to define another function to do logging to both stdout and, using vsyslog(), the syslog. This implements LOG_PERROR in a portable way.
I chose stdout because it is line buffered by default. If you use stderr, you should combine the loglevel, format, and newline with sprintf before calling vprintf on the variable arglist to prevent jumbled messages.
Of course, make sure you are cognizant that the format string is passed in and do not allow any user-supplied format strings as usual.

Why VIM is not my favorite editor

2010-11-21T14:15:00-07:00

UPDATE:
clang_complete is what the people want and what the doctor ordered:

https://github.com/Rip-Rip/clang_complete

let g:clang_snippets=1
let g:clang_conceal_snippets=1

C^X, C^U, profit. Another awesome development for LLVM!

It sucks for C and C++ development.

Popup code completion ("IntelliSense") is a godsend. Instead of flipping back and forth between an API reference and your code, a non-invasive popup of available functions, method signature, struct members, instance variables, etc is right at your fingertips. It's especially useful when it contains the declaration's comment/Doxygen/JavaDoc.

Building a 'ctags' file of my system libraries takes ten minutes and weighs 1.5GB for VIM's integrated omnicomplete. Any time the headers are updated, it has to be manually rebuilt. The project's tags need to be rebuilt per change. Unbearable.

I also can't get inline function/method signatures or automatic struct member completion without a three year old script, omnicppcomplete. Yes, this is true for plain old C too.

The one editor I've found that provides the level of introspection I expect yet otherwise stays out of the way is KDevelop 4.1. It basically takes the Kate text editor, with awesome syntax highlighting and standard editing features, and adds some of the best auto-completion I've seen for C or C++. It's fast too, and doesn't require a ridiculous manual scan or gigabyte symbol database. It just works - automatically.

Well, editing from the console is pretty convenient, especially on slow remote SSH connections. Lazyweb, am I missing something that VIM gurus know and I don't? Does emacs provide the level of completion I'm looking for on the console out of the box?

I know the llvm devs have some code brewing that uses llvm for syntax completion. Maybe there's light at the end of the tunnel.

My .vimrc looks like this if anyone has any suggestions:

filetype on
filetype plugin on
filetype indent on
set nocp
set expandtab
set smarttab
set textwidth=80
set softtabstop=2
set shiftwidth=2
set omnifunc+=syntaxcomplete#Complete
set completeopt+=longest,menuone
set tags+=~/.vim/systags

autocmd FileType make set noexpandtab

augroup csrc
au!
autocmd FileType *        set nocindent smartindent
autocmd FileType c,cpp    set cindent
augroup END

syntax on

Stop Social Distributed Version Control Diaspora!

2010-10-29T23:08:00-07:00

A ton of people use github, gitorious, and bitbucket these days. Aside from the obvious benefits of dVCS, these sites have excellent features such as:

Code review
Merge requests
Repository "forking" (in a good way)/cloning and easy methods to let upstream know about your branch
Project following/notification

Basically, the buzzword they billet this as is social coding. The idea isn't entirely new considering mailing lists with patches have been in use for ages, but the new interfaces are intuitive and visually appealing.

The problem? Walled garden. One system doesn't work with the other. Worse if you want to run your own project infrastructure. You'd think as FOSS programmers we'd know better than this and not fall into the same pitfall that social networks are currently in.

I plead with you, social coding providers, come together and create a universal API in which we can uniformly share and exchange informationsuch as followers, branches, notifications, and merge requests.

I believe this situation is important enough to warrant the attention of community leaders. I have sent email to the Free Software Foundation and The Linux Foundation asking for guidance. If you would kindly pass this along to them and members of Canonical, Debian, and other large FOSS groups we will stymie community fragmentation and develop a way to further the openness that has allowed the creation of such fantastic tools.

Sincerely, The Undersigned Free Software Advocates (place comment on this entry)

Edit #1: Response to a couple frequently asked questions:

The idea is to keep the information open, free, and interchangeable as information is what these sites add. We've had public VCS repos and web interfaces for ages. The fact that gitorious is open source does not address this.
The value of these sites is the social factor, and for the most part they bottle it up. If you prefer github, but a project you want to follow uses gitorious (like Qt), there is no means of exchanging the social factor. This is nearly identical to facebook->Diaspora.While some of these sites do have extended APIs, we need to come together and make sure that they inter-operate and also work with self-hosted systems like trac and redmine.

Edit #2: A good summary of what needs to be done on reddit.

Edit #3: A response from Richard Stallman, Atlassian (bitbucket), and github!

An email from RMS:

> There is a chance to increase openness and productivity by making these
> systems as well as self-hosted solutions such as trac, redmine, and
> reviewboard inter-operate. Developing a standard protocol for the social
> features such as clone notification, merge requests, and commit comments
> would also make integration into the commandline and GUI tools easily
> possible.

It seems like a good idea to me, to the extent it is feasible. I
don't know how these tools work, but I think that most of them use
communication protocols based on their data formats, which are unique
to each program. So we could only unify the aspects of the protocols
which don't depend on those internal data formats. But that much
could be useful.

Both Atlassian and github saw this as a valid point and were not opposed to such a standard but did not indicate they were working toward anything. Can anyone advise on how to start a working group or RFC to define such a protocol?

Software Freedom Day 2010 - Gentoo Linux

2010-09-18T08:37:00-07:00

I'm doing a presentation/Q&A for Software Freedom Day 2010 in Charleston, SC. CSCLUG is putting on the event.

Software Freedom Day 2010 - Gentoo Linux Presentation - .odp / Open Office Impress
Software Freedom Day 2010 - Gentoo Linux PDF - .pdf / Portable Document Format

Here is the presentation. Feel free to use or modify this under the terms of CC-SA3.0.

Sun/Oracle Ultra 27 Workstation Discontinued?

2010-07-22T19:30:00-07:00

I just noticed that the Sun/Oracle Ultra 27 is no longer listed on the Desktops section of Oracles products page. This is a shame because I'm quite pleased with mine.

This sends a couple of messages:

Oracle doesn't think Solaris/OpenSolaris is viable on the workstation
Oracle can't deliver low margin hardware (the prices on these boxes skyrocketed after the acquisition)

It could be a purge while they bump to a new model featuring 6-core Xeons. Yet more than likely, another victim of the merger.

But First, Write No Code

2010-05-27T23:28:00-07:00

Something I see often in person and online are programmers constantly implementing common solutions, reinventing wheels, or embracing NIH.

Before you do this, please consider the Kev009's Oath - "But First, Write No Code". This is a solution to a variety of problems in software development, but today's article is specifically on using external code.

I've found that programmers who follow a system similar to mine (detailed below) develop systems that are more stable, maintainable, and sane. They likely write better code because it means they understand their tools and also read others' code. They examine the problem first rather than going in guns blazing.

Steps to decide whether to use an existing solution or write your own implementation:

Scan the area. Google, Freshmeat, SourceForge, standard library, OS libraries, etc. are your friend. See if the problem you are trying to solve has been solved. I don't care how long you've been programming or how much you think you know. The ecosystem of a language is constantly changing. Make a list of hits that look similar to the problem you are trying to solve. Try and get a quick sense of the idiomatic methods of using your language, OS, etc.
Do research. Are the solutions you found in step 1 suitable to the problem at hand? Consider the pros and cons of each item. Now, carefully evaluate how idiomatic the items are to your language and environment. If the item is open source, does the community seem active? If it doesn't fully map to your problem, does it look like you can modify it to do so?

Even if you end up developing a solution from scratch, you should at least now have some good references. Keep in mind, extending an existing project may be considerably less work. You might even be able to offload maintenance of that component. 3. Consider the license. This isn't just for the legal department. What kind of project you are working on will weigh in heavily. Commercial or open source? As a software professional, you need to be abreast with the various licenses in the wild. As an open source developer, you need to consider how licenses will affect your work being packaged by distributions. An open source library licensed under the GPL is not acceptable for static linking to commercial software. However, you can link to an operating system provided copy or bundle the dynamic library with your application. LGPL does not have this restriction. With both of these, you must supply your changes upon request from end users among other things.

BSD, MIT, and Apache style licenses allow you to make changes and redistribute under completely different licenses. Some just want credit in your documentation. These are very compelling even in commercial development.

Commercial components may have a per-copy fee associated which may dissuade their use by your organization. If you don't get the source, you won't be able to effectively change or maintain it so you will also be at mercy of that developer.
Make a decision. By now, your list should have been pared down based on licenses and research. Perform extensive evaluations of the remainder and eventually hone in on the one you think fits best. You're going to have to rely on your experience and intuition while making the critical decision. Perhaps the hardest part: weighing it against a mythical home-grown solution in your mind.
Implement the decision. Self explanatory. This either means bootstrapping your own project or fully integrating the external one. If you are extending an open source solution, consider submitting the patches back to the community for feedback and perhaps integration. If you are bootstrapping your own solution, you've got your work cut out. Is this only suitable for an internal project, or perhaps it would have its own merit as a new open source project?Be sure to reevaluate early and often. That library you chose might turn out to be a can of worms, just as the "easy" new solution you had in your head might require years of development.
Subscribe to the announce mailing list. Only if you used an external solution. Does the project have an RSS feed for releases or a low volume announcement list? Don't be like Adobe. Avoid embarrassing security problems. Also consider how enhancements and bug fixes to the external project might make your own project better, more stable, and more efficient. This is where the real lasting dividends of using an external solution come from.

This list is widely applicable. You've got a seriously high bar to reach if you are developing containers of <T>, sorting methods, GUI frameworks, parsers, text and binary file formats, and much more so try and follow it the next time you code.

KDE SC 4.4 - Steady, Incremental Improvements

2010-02-09T01:13:00-07:00

I haven't noticed any killer features in KDE SC 4.4 and I've been running it since Beta 1. I've noticed a lot of subtle improvements. Things like app stacking and selection in the task bar seem much more responsive. All around, plasma looks subtly better and my favorite KDE apps seem to just keep getting better.

KSysGuard is really impressive and now has the ability to connect to remote hosts for monitoring. However, the biggest change is in the greater ecosystem. It seems all the external apps like Amarok, K3b, and digiKam are coming along to fruition.

Other than that, this is a smooth release and shows that the platform is starting to mature. I think the Summer release distros will be able to do a good job delivering a nice desktop experience based on KDE 4.4. I'll end with my obligatory "try KDE 4.4 if you had previous bad KDE4 experiences".

2010 SpamAssasin Public Service Announcement

2010-01-01T14:34:00-07:00

If you run a public mail server, there is a good chance you run SpamAssassin.

There is a New Year's SNAFU in which any dates 2010+ are marked "grossly in the future". That is a problem since it is now 2010.

Run sa-update and restart SpamAssassin ASAP, or install the following rule to local.cf:

score FH_DATE_PAST_20XX 0.0

Thanks to LWN for pointing this out to me before too much damage was done.

Sun Ultra 27 Review - The Ultimate Linux Workstation

2009-12-17T03:51:00-07:00

Sun Microsystems has a powerful and favorably priced entry in the x86 workstation space with the Ultra 27. Such is the power of this workstation and its natural fit for *nix workloads, combined with a compelling price tag, I pronounce it the Ultimate Linux Workstation in its class. He's a short overview and review as well as a video entry at the end.

The Ultra 27 is a single socket system but packs a powerful quad core punch. With the Nehalem Xeon (similar to the consumer Core i7), markets that used to demand dual socket systems can now comfortably look to cheaper single socket alternatives. With the return of HyperThreading, the chip handles 8 hardware threads and unlike its previous guise in the Pentium 4, HT contributes noticeably and favorably to performance. Due to the extreme performance and number of execution threads, I think the Nehalem Xeon represents a paradigm shift where most high end systems will no longer carry two discrete CPUs.

The Ultra 27 competes in the same league as the Lenovo ThinkStation S20, HP Z400, Mac Pro Quad and various other boutique workstation manufacturers. Through channel vendors, all but the Mac can be comfortably configured for around $2000.

The price of this system is attractive and affordable to anybody who makes their livelihood off of professional computing. The components on a workstation like this are well engineered and thoroughly tested together. 3rd party vendors will also commonly certify software and hardware against popular workstation offerings. The Ultra 27 is comfortably expandable and can be equipped with Intel's best CPUs on the market, plenty of disk, and high end graphics. With two PCIe2 16-lane slots and ample cooling and power, the Ultra 27 has a lot to offer anyone in digital content creation, CAD, CAM, CAE or other visualization activities. With the speedy Nehalem chip and 4 disk bays programmers doing large builds or staging large applications should be more than content.

The Ultra 27 comes in a robust light gray case. The Spartan design is free of all the unsightly plastic that usually adorns HP machines and looks more substantial than either the Lenovo or HP competition. The interior isn't quite as beautiful as the Mac Pro's, but there is plenty of room and the internal drive bay keeps any need for user cable management to a minimum.

My configuration is as follows:

Intel Xeon w3570 Quad Core at 3.20GHz (has intel Turbo Boost to 3.46GHz)
6GB RAM ECC DDR3-1333
nVIDIA Quadro FX 380 graphics
500GB 7200 RPM SATA hard disk

I am upgrading from a previous IBM IntelliStation A Pro which had two physical single core AMD Opteron CPUs. The increase in performance is remarkable.

It is noteworthy that this system is much quieter than my previous. Indeed, the large fan is seldom audible even under heavy load. The heat output is negligible and the components have never been more than warm to the touch even after hours of maximal CPU usage.

[/caption]

The case is for the most part toolless and very clean and spacious inside. Every aspect of the system can be easily upgraded so the Ultra 27 should be a good investment and grow with time and requirements.

With well engineered hardware and plenty of power, the Ultra 27 is awesome as a general UNIX-class workstation. I am happily running Gentoo and even the largest builds are a cakewalk for this box. I push my computer harder than most people. Commonly running several browser instances loaded with tabs, several terminals, email, music, the Eclipse IDE, and more, interactivity has not diminished even when running parallel compiles.

I'll conclude that anyone looking at the Mac Pro should consider the alternative in machines like the Sun Ultra 27. This computer represents a latest in a powerful linage of Sun workstations and is ideal for just about any demanding workload you can throw at it.

Video of Windows 7 Black Screen of Death

2009-12-14T23:18:00-07:00

Here's a video to accompany my previous blog post Windows 7 has definite backlight problems ("Black Screen of Death"). As mentioned before, I cannot reproduce this in Arch Linux. The problem began after a recent "Patch Tuesday".

Microsoft has been denying any problems like this in the media. Comment if you have had similar experience on Nvidia hardware or Intel graphics.

Windows 7 has definite backlight problems ("Black Screen of Death")

2009-12-11T18:03:00-07:00

There is definitely a problem in Windows 7 RTM with the "Black Screen of Death" despite MS propaganda stating otherwise. This one has nothing to do with 3rd party antivirus software.

The particular problem I am having and see lots of reports on forums is that the backlight is not turning on after an LCD DPMS standby. What's particularly bad is that some people are blaming their hardware and buying new inverters for their displays. My problematic system dual boots Arch Linux and no amount of LCD power on/off sequences will trigger this in Arch.

I suspect though cannot easily prove that the symptoms came after recent "security patches". No MS, I don't have any virus scanners installed so blaming others isn't going to cut it. It has gotten particularly bad in my case after the most recent Patch Tuesday.

I can trigger it readily with my LCDoff utility, though it will also happen organically with Windows' automatic power management.

The question is, to whom or how do I report this? I don't want to be told condescending things by a support drone. With FOSS, I have clear and easy communications with developers. I'm not aware of an easy way to get that level of interaction from MS.

So, Microsoft, I invite YOUto contact ME if you want to take care of this bug on an easy to reproduce testcase.

DRBD merged with kernel 2.6.33

2009-12-10T06:31:00-07:00

DRBD has been a long standing external patch in many distribution kernels. It has finally been merged in the 2.6.33 window. Colloquially the "Distributed Redundant Block Device", this piece of code allows you to mirror blocks of storage across multiple nodes.

This is primarily useful in high availability setups. By synchronously mirroring storage across two systems, you can run an active-passive cluster where the backup machine will take over if the primary fails. Using a more advanced clustering file system such as GFS2 or OCFS, you can even do active-active setups although there are certain considerations there.

This is exciting because it alleviates the need for specialized hardware like SAN storage. Standard Linux servers with direct attached storage may be used and indeed even give appreciable performance. In practice, redundancy will be even greater than all but the highest end SAN equipment due to the lack of single point of failure.

DRBD also allows for asynchronous mirroring, that is, writes to the primary do not wait on completion to the secondary. This is useful for cold site backups and perhaps meeting legal compliance in certain industries as well.

Take a look at the DRBD site for more information and use cases.

Getting Beautiful Fonts in Gentoo Linux

2009-12-04T01:29:00-07:00

Here's my easy and modern guide to getting pleasant looking fonts on Gentoo with minimal effort.

Wikipedia rendered with the end result of this guide

USE Flags

Enable the following USE flags:

euse --enable truetype type1 cleartype corefonts

and make sure everything is built with them enabled:

emerge -uDNa world

Emerge Fonts

X.org and other apps should pull in some common fonts. Here are some additional fonts useful for OpenOffice and other programs. Dejavu provides excellent default fonts which we will enable in the next section.

app-admin/eselect-fontconfig
media-fonts/corefonts
media-fonts/dejavu
media-fonts/font-bh-ttf
media-fonts/font-bh-type1
media-fonts/freefonts
media-fonts/ttf-bitstream-vera
media-fonts/unifont
media-fonts/artwiz-aleczapka-en

Enabling fontconfig options

Use 'eselect fontconfig list' to see a list of available configuration options.

Try enabling the following with 'eselect fontconfig enable <num from list above>':

10-autohint.conf
10-sub-pixel-rgb.conf
20-unhint-small-dejavu-sans-mono.conf
20-unhint-small-dejavu-sans.conf
20-unhint-small-dejavu-serif.conf
25-unhint-nonlatin.conf
57-dejavu-sans-mono.conf
57-dejavu-sans.conf
57-dejavu-serif.conf

This will give you very nice hinted fonts suitable for the great majority of RGB LCD displays.

The 57- series rules enable dejavu fonts as the default Serif and Sans Serif fonts. This will improve the look of your desktop environment and programs like Firefox immediately.

~/.fonts.conf

This file controls your user fontconfig settings. We will reiterate RGB hinting and disable it for bold fonts so they are not overly bold. There are plenty of other tricks you can perform in this file to get more Windows-like text, but I'm quite satisfied with the following and find it very easy to read.

<?xml version='1.0'?>
<!DOCTYPE fontconfig SYSTEM 'fonts.dtd'>
<fontconfig>
 <match target="font">
 <edit mode="assign" name="rgba">
 <const>rgb</const>
 </edit>
 </match>
 <match target="font">
 <edit mode="assign" name="hinting">
 <bool>true</bool>
 </edit>
 </match>
 <match target="font">
 <edit mode="assign" name="hintstyle">
 <const>hintfull</const>
 </edit>
 </match>
 <match target="font">
 <edit mode="assign" name="antialias">
 <bool>true</bool>
 </edit>
 </match>
 <!-- Disable autohint for bold fonts so they are not overly bold -->
 <match target="font" >
 <test compare="more" name="weight" >
 <const>medium</const>
 </test>
 <edit mode="assign" name="autohint" >
 <bool>false</bool>
 </edit>
 </match>
</fontconfig>

Conclusion

Please share any thoughts and tips in the comments. I recommend browsing the X.org Font Guide on Gentoo Wiki, though some of the information there is out of date or more complex than the method I just outlined.

boo2pdf Update

2009-11-27T07:48:00-07:00

I did some minor updates to boo2pdf. Graphics should now be within the page margins. Please let me know if there are any other common formatting mistakes.

Unfortunately, IBM's "transmogrifier" utility doesn't work very well in Wine, so you should preprocess older books in Windows before running them through the boo2pdf web service (download is on that page).

Source code is now available from boo2pdf github.

Turn Windows Monitor/LCD Off (Assembly Language Version) -- Announcing LCDoff

2009-11-02T02:11:00-07:00

I recently upgraded my trusty ThinkPad T42 to Windows 7. Unfortunately, the screen-off keystroke no longer works. Some quick Googling showed that people made their [own handlers in C#][]. Well, that sucks if you don't have the .net CLR. This is not a problem on Windows 7, but I also wanted to try out my Assembly skills since they were all just calling standard Win32 APIs.

Download LCDOff.exe

This should work on fairly old systems, back to Windows NT 3.1 and Windows 95, unlike the C# versions. It is also 7.5kb vs 85kb.

I wrote it in MASM using the Irvine library for convenience. Optimized with http://upx.sourceforge.net/

TITLE Monitor Off (main.asm)  
; Description: This program turns off your monitor (power saving mode)  
; Revision date: 2 November 2009

INCLUDE Irvine32.inc  
INCLUDELIB Kernel32.lib  
INCLUDELIB User32.lib

SendMessageA PROTO :DWORD,:DWORD,:DWORD,:DWORD  
GetConsoleWindow PROTO

WM\_SYSCOMMAND = 0112h  
SC\_MONITORPOWER = 0F170h  
OFF = 2

.data  
msg BYTE "Monitor off. Program from Kev009.com",0dh,0ah,0

.code  
main PROC

call Clrscr

; Get our window handle to send the event to  
INVOKE GetConsoleWindow  
; The handle is now in eax  
INVOKE SendMessageA, eax, WM\_SYSCOMMAND, SC\_MONITORPOWER, OFF

mov edx,OFFSET msg  
call WriteString

exit  
main ENDP  
END main

I've set up a project page at if there ever need to be updates.

Xen 3.4.1 on RHEL/CentOS 5.4

2009-10-22T13:04:00-07:00

I'm happy to report that the updated Gitco Xen 3.4.1 repo is working well on CentOS 5.4.

If you are doing link bonding and bridging in accordance with my previous post "Xen 3.3 in RHEL/CentOS 5 and more Link Aggregation Fun", you no longer need to patch the network scripts as RedHat fixed the initscripts package in RHEL 5.4.

Upgrade procedure for CentOS 5.3 to 5.4:

yum clean all
yum update glibc\*
yum update yum\* rpm\* python\*
yum clean all
yum upgrade
reboot

Updated Xen Install Guide From My Previous Article:

Head over to http://www.gitco.de/repo/ and grab the repo for your arch. (Most likely wget http://www.gitco.de/repo/CentOS5-GITCO_x86_64.repo in /etc/yum.repos.d/ for the uninitiated).

If you already have Xen installed, you may need to remove and readd it.

yum groupremove Virtualization
yum groupinstall Virtualization

You'll also get some updated tools like Virtual Machine Monitor 0.7.0 that make it easier to install newer guests such as Fedora 11 or Ubuntu. Sweet!

Double check /etc/sysconfig/kernel. It should be set to kernel-xen. Likewise, check /boot/grub.conf and make sure that the Xen kernel is the default if the aforementioned was not done beforehand.

Reboot!

Linus and Windows 7

2009-10-22T11:25:00-07:00

This is awesome.

Announcing boo2pdf

2009-10-20T00:35:00-07:00

I've just uploaded a beta of boo2pdf, an IBM BookManager to PDF conversion app & web service. I'm currently experimenting with the HTML to PDF backends and would like feedback with book files I haven't tried. Once the code is cleaned up, I will dump it on my site.

Motivation

I have a large collection of old IBM machines and documentation. I want this documentation indexed by my own search facilities and Google for easy retrieval. PDF is widely read, while BookManager requires proprietary software and no search engines I know of parse it. This will probably be useful to Mainframers as well.

Take the web service for a spin here: http://ps-2.kev009.com/boo2pdf/.

Java: The Good Parts

2009-10-14T22:31:00-07:00

A while back, a book entitled JavaScript: The Good Parts made waves on the internet, especially social networking sites. This book purported to show the inner beauty of a language that was long considered second or third rate, coming of age. With the advent of toolkits like JQuery, Javascript/AJAX development has become easy and even fun.

I aim to do the same by showing "Java: The Good Parts" here at a high level.

When I was younger, I used to despise Java for political reasons and bad memories of early applets and applications. I suspect many users and developers (especially Libre software devs!) are on the same boat. By the end of this article, I hope I swayed your opinion or at least caused you to reevaluate your bias. I also wish to encourage further discussion about these points and ways we can improve any deficiencies.

Rough and Tumble Upbringing

When Java first started gaining popularity, it was loudly hyped as the end all language. It was expected that Java would take the "rich client" by storm, and applets would be the go to solution for enhancing web pages. What happened was a bit different. Java floundered and struggled to find a niche. On the client side, AWT apps looked horrendous despite using native widgets. Then Swing came about and despite easing development, it looked equally bad on all platforms (by default). Applets were basically a stillbirth. The ugly gray box, loadtime sometime measuring in minutes, and no coordination with the DOM and web browser made the average user hate Java.

One area Java was able to develop and secure a foundation, however, was the back end of large web applications. The Virtual Machine approach provided a marked advantage over the CGI and interpreted scripts of the day. Java's rich networking libraries, clean Object Oriented design, and safety made this the language de jour for large web applications.

Open Source Matters

In my opinion, the open sourcing of Java during its early infancy would have had little impact on most of the teething issues. The Virtual Machine, JIT, and Garbage Collection required many years of tuning to get acceptable performance and Sun did an acceptable job keeping it under wing. The relatively limited CPU and RAM of the mid '90s also made these concepts a bit ahead of their time. Somewhere in the 1999-2002 time frame, though, Sun really dropped the ball. An Open Source Java would have led to ubiquity on the booming Linux platform and a chance for all sorts of cross-platform software.

Open Source matters, and not just for the source code. Open Source projects naturally bring about very pragmatic and intelligent developers. These are the folks that thoroughly enjoy their hobby, work, and tools. The marketing guys and pointed haired bosses have much less pull here. On one hand, a vibrant community built itself with the many Apache Software Foundation projects. However, most of these were squarely focused on web applications or low level things such as build tools, testing frameworks, and message buses.

Due to the void, interpreted languages such as Python rose to the challenge while C and C++ remained the mainstay for applications programming. Microsoft started dominating Windows development with their .NET CLR languages. The glib/Gtk+ and Qt toolkits brought about a renaissance in cross-platform development with C and C++ respectively [though not limited].

It wasn't until the open-sourcing of SWT that GUI development in Java became attractive. The obvious killer apps here were the Eclipse IDE and the Azureus (now Vuze) bittorrent client.

Sun's closed grip of Java really stagnated any chance of abundant expansion in these middle years (2001-2006). Microsoft leveraged this weakness to create the excellent .NET platform and associated languages to maintain their closed platform and market dominance. The counterbalance that would have been Java was thus left playing catchup.

We are just beginning to see the fruits of this labor from 2006 through today. The OpenJDK project is now distributed with popular Linux distributions such as Ubuntu and Fedora. We finally have decent browser plugins and Java Web Start applications across 32 and 64-bit machines. The Java deployment problem will slowly fade from memory.

Application Development

SWT made Java apps beautiful. OpenJDK should make them ubiquitous. We finally have an Open Source platform that is widely deployed. The strong built in standard library and clean OO design patterns of Java make it a very pleasant host for developing rich client apps. Obvious areas for improvement here include better layout/form design tools and closer integration with upstream Linux distributors.

Somewhere along the line, Trolltech/QT Software (now owned by Nokia) released Jambi -- the complete Qt bindings, GUI framework, and incredibly rich library -- for Java. Oddly, this bombshell received little of the community and fanfare I thought it would or deserves. Indeed, QT Software demoted Jambi from their teir-1 platforms and hopes the community will pick it up. I hope this project isn't allowed to stagnate as there is a lot of potential here.

Web Apps

Along the "Enterprise Web Application" lineage of Java, we wound up with some disgustingly overcomplicated and bloated frameworks for building web apps. Ruby on Rails and Python Django came about and put a new spin on the development of rapid and robust web apps. The learning curve of these frameworks is much less than Java EE and I will go as far as saying they are more capable because of it.

By using Java, JSP, and Servlets directly on top of a light Model-View-Controller, I believe Java is just as compelling as some of the more popular scripting languages. Developers need to know they can trim the fat and that there are many advantages to developing in Java, namely because of the next topic...

Dynamic Languages, its all about the VM

It's all about the JVM stupid! One of the best features of Java and .NET are the underlying Virtual Machines. By using JIT compiled VMs, Java code has a distinct advantage over the common interpreted languages such as Perl, Python, and PHP. In the case of Java, the resultant is even naturally crossplatform.

The really interesting developments here focus on extending the JVM to syntax and paradigms other than the statically typed C++ lookalike. Clojure and Scala deliver innovative new techniques while Jython and JRuby bring these excellent languages to the Java software platform and virtual machine.

In short, Java provides everyone with a counter to Microsoft's .NET CLR. The Java VM has been around the block and tuned by giants such as Sun, IBM, Oracle, SAP and more.

I call on the community to discuss how we can encourage use of the JVM for languages other than Java and build this into a defacto runtime. Continued tuning and integration with Windows, Mac OS X, and Gnome/KDE *NIX systems is paramount. Research for easy multi-core development is also worthwhile. Meanwhile, distributions need to continue packaging the JRE and make it a default. Individual developers need to be made aware of "Java: The Good Parts" and myths debunked.

Applets, Rich Media, Native Code!?

With the release of Java FX, widespread deployment of the JRE, and better browser integration, Java has set the stage for a comeback to its roots.

During the late '90s and 2000s, Adobe Flash became the tool of choice for web animation and interactive pages. It really exploded with the advent of Youtube and other internet streaming sites making use of the Flash video format. Unfortunately, Flash player is notoriously insecure, resource intensive, and crash-prone. It is also not widely available for the millions of smartphones that have become more accessible than computers.

Luckily, there seems to be a shift back to the browser with new developments in AJAX, JavaScript, and HTML. The \<video> tag will hopefully make video as easy and portable as graphics are today in the browser. Clean JavaScript libraries and fast JIT JS engines make it practical to use this paradigm for many domains.

Yet one must acknowledge that somewhere along the line, manipulating a DOM/markup language with a scripting language isn't the most effective development platform for everything. Google even thinks it poignant to run x86 machine code in a sandboxed environment in your browser. I personally fail to see the logic behind this. Java provides a well evolved, cross-platform solution. Java can run on your ARM powered Android. Requiring an x86 CPU just seems like the wrong track in this modern age. Hopefully JavaFX will pick up the slack and return Java to its roots. I would love to see the demise of the terrible Flash plugin.

Future and Conclusion

It's time we considered Java for The Good Parts.

I hope some of my points caused you to reevaluate any bad preconceptions or past experiences you may have had with Java. Java has undergone great change since its birth and I think it is capable of becoming the premier development platform for applications programming of all types. Particularly interesting are some of the new languages such as Scala and Clojure. Java has long been a staple in web development, but has traditionally scared away amateur coders. If you cut the fat, Servlets and JSP are not much harder to set up than common place scripting languages. Frameworks such as Grails bring it to parity with Rails or Django.

Java underwent a sea change in 2006 with the releasing of the source code and opening of the development process. Java and the JVM should be championed by Libre software developers and users alike!

Mobile phones spur cross-platform applications. Open Source is mainstream.

2009-09-24T01:42:00-07:00

I had an interesting day today. At school, we had a social event with an industry governing board and several local software companies in the Charleston, South Carolina region. Aside from meeting a lot of new people, I was able to ask some of the industry leaders present about the platform and languages they used.

Perhaps the most interesting was that many Microsoft shops are moving away from fat client apps to web apps, and not simply because it is a buzzworthy thing to do. The primary driver is the proliferation of advanced mobile devices, namely iPhone, Palm Pre, and Blackberry. Two of the companies I talked to were VB.NET or C# shops and used to do traditional fat client software. Due partially to the smartphone craze, they are moving SaaS. They also mentioned the Mac as a rising popularity, but no Linux or netbooks or anything like that. It's funny how venerable HTML has become.. now the medium of choice for displaying cross-platform applicaitons. I doubt anybody ever imagined just how important HTML and JavaScript would become during their infancy.

Surprising and delightful to me was the talk of Open Source in the enterprise. I spoke with two gentlemen from different large defense contractors and they were spot on with there assertions that Open Source software is superior in many ways. Both were large Java EE shops and mentioned how they could check and verify FOSS for security much better than any proprietary software would allow. They mentioned that the US Government is one of the largest purchasers of software but even then working with COTS vendors is difficult and FOSS solves many of these problems by allowing them to commit back changes. The thing that made me the most happy was when one rep said that active participation in an Open Source project was a surefire way to boost a resume to the top of a stack.

Another interesting tidbit was that virtualization is synonymous with VMWare here (everyone specified this by name) even today. I'd go as far as to say that I'd probably have received strange looks if I had mentioned KVM or even a heavyweight like Xen or MS Virtual Server. Aside from Windows, the defense guys talked a lot about Solaris. I didn't get much reaction when mentioning Linux to anybody, sadly. (somewhat comically) the non-profits were largely the 100% Microsoft shops.

Mirroring Fedora

2009-06-23T04:11:00-07:00

Introduction

This post details setting up your own private mirror of Fedora's repos. There are many ways to do this, but this method is by far the best for heavy usage. By using MirrorManager, clients in your IP range need no custom configuration. Roaming laptop users automagically hit your mirror while on the premises, yet use the public infrastructure elsewhere. Setup isn't exactly hard, but it isn't well documented so I'll write about my experience here.

Some background info.. we have at least 50 Linux desktops, laptops, servers and VMs running about half Fedora 10 and half Fedora 11 at work. Due to the number of systems, breadth of packages used, and desire to quickly update when new releases are out, I decided on a full mirror setup. If you only have a handful of systems, you may be better off simply using a general purpose caching proxy like Squid, perhaps telling MirrorManager to point to it.

This guide should be used in addition to http://fedoraproject.org/wiki/Infrastructure/Mirroring which has some background info.

Initial setup and mirror

First, get prepared by installing MirrorManager-client, which contains the report_mirror script you will need. If your mirror isn't running Fedora, you can clone the source of this app from their GIT repo.

yum install mirrormanager-client

You'll be using rsync, a sysadmin's best friend, for efficient mirroring.

Set up a shell script like mine below (d0mirror.sh) one level up from where your mirror will be accessible (http, ftp, rsync, nfs - covered later). This one mirrors against kernel.org. Choose a mirror close to you on the Internet.

rsync -vaH --exclude-from=fedora-excludes.txt --numeric-ids --delete --delete-delay   
--delay-updates rsync://mirrors.kernel.org/fedora-enchilada fedora-mirror
report_mirror

And a text file (fedora-excludes.txt) excluding things you don't want/need. Take a look through a public mirror and decide if you want to eliminate anything else. You may want to remove the *.iso line below if you want users to be able to pull disc images from this box. Otherwise, this is probably a good list for most people. You can exclude all of linux/updates/testing/ if you don't enable the testing repo on any of your machines.

**/debug/**
**/alpha/**
**/source/**
**/SRPMS/**
**/*.iso
**/ppc/**
**/ppc64/**
linux/core/**
linux/development/**
linux/releases/7/**
linux/releases/8/**
linux/releases/9/**
linux/releases/test/**
linux/updates/8/**
linux/updates/9/**
linux/updates/testing/7/**
linux/updates/testing/8/**
linux/updates/testing/9/**

Run your shell script and sit back for up to a day or two depending on your connection speed. My current mirror weighs in at about 80G.

Internal distribution

While you wait for sync, decide how you want to run the service internally. HTTP is nice because it is easy for users to browse and decently quick with keep-alive. Using NFS, rsync, or FTP may be a bit more efficient if you are worried about this. You can list several URLs in MirrorManager for the best of all worlds.

Add the following to your Apache configuration if you decide to use HTTP:

Alias /fedora/ "/mnt/ar1/fedora-mirror/"

AddType application/octet-stream .rpm

<Directory "/mnt/ar1/fedora-mirror">
    Options Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

<LocationMatch "\.(xml|xml\.gz|xml\.asc|sqlite)">
    Header set Cache-Control "must-revalidate"
    ExpiresActive On
    ExpiresDefault "now"
</LocationMatch>

Set up any other services of you choice to push that directory out in addition.

Working with MirrorManager client and server

Next, open up /etc/mirrormanager-client/report_mirror.conf. Take notice of the site name, password, and host name. You will need to set these up in MirrorManager in a bit. The paths here are all local and used by report_mirror to check what you have available.

# if enabled=0, no data is sent to the database
enabled=1
server=https://admin.fedoraproject.org/mirrormanager/xmlrpc

[site]
# if enabled=0, no data about this site is sent to the database
enabled=1
name=<yoursitename>
password=<yourhostpassword>

[host]
# if enabled=0, no data about this host is sent to the database
enabled=1
name=x345-a2.internal
# if user_active=0, no data about this category is given to the public
# This can be used to toggle between serving and not serving data,
# such enabled during the nighttime (when you have more idle bandwidth
# available) and disabled during the daytime.
# not passing it means leave it alone in the database.

[stats]
# Stats are only sent when run with the -s option
# and when this section is enabled.
enabled=0
apache=/var/log/httpd/access_log
vsftpd=/var/log/vsftpd.log
# remember to enable log file and transfer logging in rsyncd.conf
rsyncd=/var/log/rsyncd.log

[Fedora Linux]
enabled=1
path=/mnt/ar1/fedora-mirror/linux

[Fedora EPEL]
path=/var/www/html/pub/epel
enabled=0

# lesser used categories below

[Fedora Web]
enabled=0
path=/var/www/html/pub/fedora/web

[Fedora Secondary Arches]
enabled=0
path=/var/www/html/pub/fedora-secondary

[Fedora Other]
enabled=0
path=/var/www/html/pub/alt

# historical content

[Fedora Core]
# if enabled=0, no data about this host is sent to the database
enabled=0
path=/var/www/html/pub/fedora/linux/core

[Fedora Extras]
enabled=0
path=/var/www/html/pub/fedora/linux/extras

Log into https://admin.fedoraproject.org/mirrormanager, creating a new account if you need to. Add a new site with the same name as the config file from above. You'll set the site password here, and make sure to check the 'private' box if this is only for internal users. Now, add a host under this site. The name here should probably be a FQDN of your actual mirror, even if it is internal only (i.e x345-a2.internal from my example above). Once that is done, add a "site-local netblock". This is your public IP network/netmask or network in CIDR notation. If you only have one public IP, it will be in the format nnn.nnn.nnn.nnn/32.

Almost done. Now, click Add Category. "Fedora Linux" is the only one you are concerned with if you followed all the values in this guide so far. Add the others if needed. Tell them your upstream source (rsync://mirrors.kernel.org/fedora-enchilada from above) and then your internal URL (http://x345-a2.internal/fedora/linux for my setup).

Conclusion

Once your rsync is complete and report_mirror is done, you should see clients start hitting your box. Don't forget to add your mirror script (domirror.sh from above -- rsync and report_mirror) to cron! You may wish to join the private 'fedora-mirrors' mail lists to be informed of new releases and changes.

The best thing is that it works across all package requests, including new machines, roaming users, 'preupgrade', etc. All in all, pretty nifty! Your users will love you when their upgrades are almost instant! The Fedora infrastructure is set up very well for mirroring, public and private, and this is how the project copes with the huge demand for new releases. Comment away if you need clarification or help.

Kernel 2.6.30 is a Go

2009-06-14T17:46:00-07:00

I initially thought this would be a rather uninteresting release, especially when we learned Xen dom0 didn't make the cut. Following the changelog line-by-line, this one still didn't seem very interesting to me. But analyzing the sum of parts, I have to consider 2.6.30 a 'golden' kernel -- certainly the best in a while.

There is solid improvement top to bottom here. A lot of the new KMS/DRM stuff from Fedora 11 has worked its way up stream. File system work is too much to mention, but highlights include relatime, writeback by default for Extfs, NILFS2, Btrfs development and more. FSCache works as advertised. Also some groundwork for NFS 4.1, which will eventually bring us pNFS.

Boot speed seems fast as ever, but I haven't taken the time to do any empirical analysis. Your results here will be hardware dependent but async initialization of certain subsystems is a welcome move in the right direction.

Basically, a solid release with a good balance of new stuff but mainly refinement of existing systems and merging of longstanding patches.

Kernel Newbies has, as usual, a great change summary: http://kernelnewbies.org/Linux_2_6_30

Kernel developers don't get Xen

2009-06-05T22:55:00-07:00

The recent bruhaha surrounding Xen on LKML (http://lkml.org/lkml/2009/6/2/475) is really disheartening. Essentially, the Linux kernel devs are at a disconnect with users. Some are proposing narrow-minded ideas such as DROPPING software paravirt or merging Xen as a whole into the kernel.

I use Xen for a few primary reasons: it bar none has the best speed -- full software paravirtualization pays dividends here; it is mature; it works on perfectly good machines that don't happen to have the latest chips; it does hardware passthrough on these same systems; it has great live migration that actually works.

Ingo Molnar wants you to send all your perfectly good enterprise iron to the landfill even though these systems will last 10+ useful years without boneheaded software decisions such as this.

These same FUDsters want to strip the crossplatform nature of Xen dom0 out too. Xen dom0 runs on NetBSD and Solaris. It is a true hypervisor and will plug into exisiting architectures, and not force you to use Linux for everything.

I have to admire all the hoops Jeremy Fitzhardinge has jumped through to date, as I know my patience is wearing thin.

Xen powers huge sites such as Amazon and services like linode.com/slicehost.com. By not having dom0 in the kernel where distros such as Ubuntu and Fedora can easily integrate it, kernel devs are doing a disservice to users.

I use KVM, VMWare, and Virtual Box at work in addition, but Xen is firmly entrenched in my toolbox. The roadmap they have looks great, and I just don't see a reason for decline in Xen popularity. High availability in Xen 4.0 is what I've always been waiting for.

Jeremy has gone to great lengths to work with upstream but keeps getting shot down and asked to do something else when he meets one requirement. The solution is to merge Jeremy's conservative dom0 patch set and work on a technical solution to the patches that the FUDsters consider bad. It's what the users want!

El Reg Humor and Java in free software

2009-05-08T22:45:00-07:00

The Register has a good article on Sphinx search with some entertaining pop-shots at Java and "enterprise software" that got a rise out of me:

Solr is popular with the enterprise crowd, who love its Java. Being a Java program, Solr includes no shortage of technology whose acronyms contain the letters J and X.

This tickles the enterprise pink, because these sorts of developers love nothing more than hanging out around a whiteboard drawing boxes and arrows and, from time to time, writing XML to make it look like they're doing real work. Solr thrives in this environment, being an Apache Foundation project, the Apache Foundation, of course, widely known as a cruel experiment to see what happens when bureaucrats do open source.

Having a bit of experience with Java from academia and a few open source projects I make use of, I can't help but laugh at how comically and concisely the editor summed it up.

By and large, successful open source projects tend to be written in languages other than Java. The entire GNU/Linux OS stack is primarily C, with some components using C++ like KDE, OpenOffice and Firefox. On the ever popular web front, PHP, Ruby, and Python lead the pack.

I think it turned out this way for a multitude of reasons. When working on the OS stack, the power and control of C and C++ are hard to beat. The plethora of libraries and raw speed of these compiled languages set the bar high for any newcomers. Java exists as a kludge, mildly useful for desktop apps and mildly useful for web apps while historically having a lot of problems. Native look and feel have long been the layman's complaint, though SWT has done a pretty good job there. Of course, omnipresent Java in the Linux world is relatively new. I think Java would have been the darling language of client apps had it been open sourced sooner, but this came about 7 years too late to have a large impact on shaping the common FOSS userland.

It is interesting how the open source projects built with Java tend to be highly bureaucratic and abstract. I think the bottom line is that FOSS programmers do what they do because it is fun and demand pragmatism. The "enterprise software" attitude/baggage that many Java apps and libraries carry are a big turn off to pragmatism and the hacking culture. The barrier to entry for Java web programming is also much higher than its "scripting language" competitors, which carry light and simple frameworks that focus on results, not procedure.

Java itself isn't that of a bad language. I actually enjoy working with it in school (...though I think it really isn't appropriate as an introductory teaching language, shielding important concepts from students. Maybe a future post?..). When it comes time for real work though, I consider Python, C, C++ more pragmatic depending on the job at hand. That, and the fact that most of the common scripting languages are gaining JIT compilers may accelerate Java toward status as a legacy language.

Your thoughts?

To users that miss xorg.conf and complain about it

2009-05-07T23:38:00-07:00

I get requests from users and see questions all the time for "where did my xorg.conf go in the latest Ubuntu or Fedora?", though it is usually a bit more of a flame.

The quick answer... press Ctrl+Alt+F2 or similar to log into a TTY console, or type 'init 3' into a root X terminal.

If you haven't already, log in as root and kill X or type 'init 3' if you want to be heavy handed. Then run:

X -configure
mv ~/xorg.conf.new /etc/X11/xorg.conf

xorg.config in two commands. Run the 'init 5' command to get back to your GUI login (or kdm or gdm or startx, etc if you know what you are doing. Worst case remove the .conf and restart.)

If you are advanced enough to edit an xorg.conf, the above should be a cakewalk and you shouldn't complain about it.

Regardless, you should investigate 'xrandr' which makes it simple to do runtime adjustments.

If you are a newbie, look into a gui. KDE has KRandRTray which makes controlling outputs and resolutions a breeze. Don't forget to toggle the output on with the Fn key if you are a laptop user.

Needless to say, Xorg is moving in the right direction. Stop complaining about it.

FS-Cache merged in Kernel 2.6.30

2009-04-03T22:22:00-07:00

FS-Cache has been merged into the upcoming kernel 2.6.30. This allows for a generic caching interface in the kernel for other file systems. For example, you can use local hard disks to cache data accessed via NFS, AFS, or CD-Rom. Since these tend to be high-latency while the disks are low latency, it should provide for a nice speedup.

Of particular interest to me, I contacted maintainer David Howells who is a Redhat employee. I asked whether this infrastructure would help with large disk image files stored on NFS -- a common though not particularly efficient case for VMWare, Xen, KVM, etc. His exact response was "Quite feasible. As long as you have a local disk on which to cache the files."

I am quite happy as I run this setup at work for some production VMs since it allows for easy migration and backup without the complexity and cost of a SAN or cluster FS. I look forward to testing when 2.6.30 hits the stable tree.

Good Linux File System Developments

2009-04-01T23:25:00-07:00

ext4 has sparked good controversy on the LKML. Aside from the recent delayed alloc and fsync issues, the whole FS stack is getting some much needed attention. Indeed, Linux file systems are starting to feel like first class citizens again with ext4 and Btrfs (merged in 2.6.29 for testing!) and the surrounding infrastructure being worked on. A lot of long overdue problems are being mitigated. Jens Axboe claims 8% single drive and 25% array speedup with some recent pdflush patches. This is very good news for all users since disk I/O has had a fast growing gap with CPU and main memory bandwidth, even with SSDs. The fruits of this labor are quite visible with recent boot speedups in distros like the upcoming Fedora 11.

Mandatory reading:

Arora, a refreshing new Qt/WebKit browser

2009-02-26T18:40:00-07:00

The Gentoo Qt maintainers have been doing a fantastic job of getting cutting edge Qt software into shape with the qting-edge overlay. I've been running Qt 4.5 since beta1 and am pleased with the direction it is going. Recently the devs bumped the Arora ebuild to version 0.5. Arora is a lightweight browser based on Qt and WebKit.

I must say, I am impressed. This browser is lightning fast. The interface is pleasing, yet minimalist. You get all the benefits of recent WebKit, and the nice cross-platform nature of Qt. It feels much faster than Konqueror4.2, and the recent WebKit engine means better compatibility.

It works with all the sites I've thrown at it so far and performs great on the Acid 3 test.

All in all it reminds me of Firefox's early days. Very mean and lean. Firefox 3.1 beta2 has served me pretty well, so I don't think I'll abandon it yet, but WebKit browsers are becoming quite compelling.

The WebKit devs are also pushing some interesting new ideas out with CSS animation. This example gives Flash-esque falling leaves. Maybe we will see less reliance on Flash in the future with the new video/audio tags and work such as this. It would be nice to move presentation back into structured [X]HTML and CSS, which is portable, easier to parse, and light weight.

If you are on Gentoo, bump to Qt 4.5rc1 and take a look easily (both are in portage):

emerge -av arora

I've seen it hit Fedora Rawhide recently as well.

Only one problem: icon is a bit conspicuous :-P

Good open source journalism

2009-02-21T23:40:00-07:00

I have to hand it to the guys at The H Open (formally Heise Online Open Source Edition). They have excellent coverage of the latest happenings in FOSS software. They also have great coverage of software releases and break it down to the important new bits. Finally, it is not overburdened with legal BS or thoughtless benchmarks like many other Linux news sites.

In short, the content is very relevant and professional. It is worth picking up their RSS feed.

KDE 4.2 on Gentoo

2009-01-27T15:18:00-07:00

KDE 4.2 is out officially. The ebuilds for Gentoo have been ready for a while. This is a truly fantastic release. If you've ever made an opinion about KDE in the past, I encourage you to give it another go.

My beta1 review back in December sums up most of my thoughts on the release. There isn't anything significantly changed since, just lots of polish and bug fixing. Everything has been stable and functional since I've been using it in the RC phase. This is a worthy opponent to KDE 3.5, GNOME, Windows and OS X.

Thanks again to the Gentoo KDE team. The ebuilds are in great shape!

One Small Step for QT, One Giant Leap for Free Software

2009-01-16T09:57:00-07:00

QT Software, under the graces of Nokia, has released the superb QT cross-platform toolkit under the LGPL.

This. is. HUGE.

For the libre software purist, this still benefits you, if indirectly. Companies that make changes to the toolkit must still submit patches. More influential, GPL incompatible software may now readily use QT for free. This will likely foster more QT centric developers, boost adoption of the underlying stack (Linux, etc), and lower the barrier for vendors to release cross-platform tools.

From a Nokia business perspective, it makes perfect sense and makes the whole thing that much more beautiful. "QT Everywhere" is really a possibility now. And, it's beneficial to Nokia as well as the ecosystem they are enriching. The more QT developers, the bigger the talent pool for Nokia software. The more contributors, the better the toolkit. Win. A small company like Trolltech could not afford to do this, but to a big dog like Nokia, the revenue from commercial licensing is insignificant and unimportant compared to device sales.

I know the company I work for, Analog Rails, will be able to take advantage of the license switch. Being previous commercial QT customers, it was expensive to juggle around machines to maintain compliance. For companies like VMWare that deploy cross-platform software and maintain their own cross-platform extensions, this surely must be compelling. I say, the more the merrier!

What a great day for free software, computing, and life in general :-) .

Ars Technica has outstanding coverage of the news: http://arstechnica.com/news.ars/post/20090114-nokia-qt-lgpl-switch-huge-win-for-cross-platform-development.html

Welcome back, Palm! The New Palm Pre

2009-01-08T20:10:00-07:00

I've always been a fan of Palm hardware, and even had a soft spot for PalmOS until it was left to rot for so long.

Indeed, my Palm 800w is a fine piece of hardware even though it was quickly eclipsed by the Treo Pro, but Windows Mobile has been really painful. It is slow, bloated, and crashes frequently. It has zero intuition and single-handed operation is out of the question. Worst of all, there seems to be a lack of useful or quality apps that were abundant in PalmOS. I am a bit mad that I paid $500 for such a lemon.

Enter the Pre, which looks like just the thing to get me back on board with Palm and get them back in the game.

The hardware design looks beautiful, giving a large screen but keeping the tactile and speed of a keyboard (this is the iPhone's biggest weak spot IMHO). It looks like typical form factor for this style of phone, similar to the new Blackberry, so nothing earth shattering there, but the OS looks top notch and is based on Linux. I think if Palm is able to deliver, they will once again be competitive, and they will keep me as a customer.

Take a look at these links for a nice report:

http://arstechnica.com/news.ars/post/20090108-palm-launches-new-handset-pre-operating-system-at-ces.html
http://www.palm.com/us/products/phones/pre/index.html

I dream of pervasive virtualization...

2009-01-02T12:09:00-07:00

I dream of a day where virtualization is pervasive.

Instead of thinking about services in terms of servers, CPUs or directly mapped resources, I should be able to to add virtual machines in terms of guaranteed throughput rate over a whole grid. Scaling out should be as easy as adding a blade or racking another server.

At the low level, I should have the option of running N+N redundancy. That is, the VM should run in lockstep across multiple machines - so if it is running on 2 vcpus, 4 in total would be used. This would allow for any node to fail. And the VM should be an aggregate of the low level hardware - e.g. a VM grid across 4 8-core servers should scale near-linearly when a single OS instance is running 32 processes.

Current solutions only attempt to do some of the tasks above, and most fail miserably. IBM mainframes have been doing it for ages.

If I had the time, I know I could build software to do this better than anyone else. All the puzzle pieces are there, especially the tough ones like hypervisors and Infiniband. This could have been done at least 3 years ago. I bet it will take the industry 3-4 years yet to get anywhere close.

This is a real virtual datacenter.

Xen 3.3 in RHEL/CentOS 5 and more Link Aggregation Fun

2009-01-01T07:12:00-07:00

RHEL 5 includes the now ancient Xen 3.0 hypervisior. A lot has been improved since then, especially in the current 3.3 release. Additionally, RedHat now owns the company behind KVM, so it is unlikely they will spend much time backporting Xen stuff for RHEL 5.3 or the likes.

Why Xen?

Xen is a proven hypervisor. It works well on lots of hardware, including servers without hardware virtualization and older 64-bit Opterons that wont run 64-bit guests in the likes of VMWare. Since the OS is usually paravirtualized, performance is top notch. By making an OS aware of the environment it is running in, you can optimize it for virtualization. KVM is playing catchup here, realizing that paravirtualization is still ideal for many things.

How..

Okay, so we are using or want to use Xen. Others have already built the packages we need, thankfully!

Head over to http://www.gitco.de/repo/ and grab the repo for your arch. (Most likely wget http://www.gitco.de/repo/CentOS5-GITCO_x86_64.repo in /etc/yum.repos.d/ for the uninitiated).

If you already have Xen installed, you may need to remove and readd it.

yum groupremove Virtualization
yum groupinstall Virtualization

You'll also get some updated tools like Virtual Machine Monitor 0.6.0 that make it easier to install newer guests such as Fedora 10 or Ubuntu. Sweet!

Double check /etc/sysconfig/kernel. It should be set to kernel-xen. Likewise, check /boot/grub.conf and make sure that the Xen kernel is the default if the aforementioned was not done beforehand.

Reboot!

Xen 3.3 and Link Bonding

See my previous post for general information, but it gets harder.

This one is a nightmare. In my previous post, I detailed how to get Xen to work with link aggregation with Xen 3.0. Well, it doesn't work in 3.3. Xen decides that it still owns eth0 and completely destroys your bond0 setup.

Like these people, I've come to the conclusion that the integrated network scripts suck. This is alarming since you'd think link bonded setups would be the norm for Xen setups.

The quick fix is to let the OS handle networking. We do that like so: add a br0 interface and tell the bond to bridge with it.

File /etc/sysconfig/network-scripts/ifcfg-br0

DEVICE=br0
ONBOOT=yes
BOOTPROTO=none
IPADDR=10.0.6.201
NETMASK=255.255.255.0
GATEWAY=10.0.6.1
NO_ALIASROUTING=yes
TYPE=Bridge

Then, edit your /etc/sysconfig/network-scripts/ifcfg-bond0 and add "BRIDGE=br0" and comment out any IP related information (since you are now defining that in the bridge. Head over to /etc/sysctl.conf and add:

net.ipv4.ip_forward = 1

Now, edit your Xen VMs in /etc/xen/ or /etc/xen/auto and change xenbr0 to br0:

vif = [ 'mac=ee:cc:aa:88:66:44, bridge=br0', ]

Okay, now disable the Xen networking garbage. Open /etc/xen/xend-config.sxp and comment out anything that looks like (network-script ....).

Almost done, but wait! RHEL 5.2 has a bug that prevents the bridge coming up on a bonded interface. Hopefully this will make the 5.3 cut or be pushed to 5.2, but until then go here. Download the new patch into /etc/sysconfig/network-scripts/ and run patch -p0 \< ifup-eth.patch for instance.

Finish

Reboot. You now have Xen 3.3 goodness on a big Ethernet channel! Post a comment if you have any trouble or questions.

Link Bonding Craziness in RHEL/Centos 5

2008-12-30T03:27:00-07:00

I just went through hell in a handbasket trying to get 802.3ad Link Aggregation set up on a Centos 5.2 Xen box. Setting up link aggregation itself isn't that bad - http://wiki.centos.org/TipsAndTricks/BondingInterfaces for a simple guide (after your managed switch is configd) - but what ever I did, I was unable to get both interfaces simultaneously active.

About the only useful debugging info I got was that the MAC was in use. I was puzzled because as far as I know, link agg takes over the primary MAC and sets that up for both NICs. Furthermore, the same exact hardware was working great on Fedora 10.

bonding: bond0: Warning: the permanent HWaddr of eth0
 - [MAC ADDR]- is still in use by bond0. Set the HWaddr of eth0
to a different address to avoid conflicts.
bonding: bond0: releasing active interface eth0
bonding: bond0: making interface eth1 the new active one.
bonding: bond0: Removing slave eth1
bonding: bond0: releasing active interface eth1
ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave eth0.

Luckily, I stumbled across this bug report. If you scroll down to the last comment, this appears to be a Xen specific issue. By default Xen tries to set its bridge up on eth0, and I assume this prevents the kernel bonding driver from taking over the NIC. By opening up /etc/xen/xend-config.sxp and adding:

(network-script 'network-bridge netdev=bond0')

Xen will bridge to the bond0 interface, and everything will work as expected.

Another trick I had to do was add a start delay to the networking scripts. This is useful if your hardware is crap (cough Broadcom), you need a dhcp lease and it fails, or you are running STP, link aggr., etc. On Fedora, RHEL, and derivitives this is accomplished by adding the NETWORKDELAY directive to /etc/sysconfig/network:

NETWORKING=yes
NETWORKDELAY=31

If you need more granularity, you can set delays to specific adapters in the /etc/sysconfig/ifcfg-{x} files with the LINKDELAY directive.

Just a couple of hard lesssons from the trenches, hopefully this will save someone else some time.

How to upgrade to ext4 in place

2008-12-24T05:19:00-07:00

Here's how you upgrade to ext4. The process is pretty easy, but requires an fsck which means unmounting or rebooting if the file system is in use.

Make sure you are using at least e2fstools 1.41.3 and kernel 2.6.28 (or a vendor kernel with latest ext4 patches applied)! Also, it's probably a good idea to have proper backups (really!). ext4 has just been declared stable, but what that really means is that the battle hardening has just begun. I've done several heavily used systems without fault so far though, so it's probably good enough for your desktop.

WARNING: DON'T CONVERT YOUR /boot PARTITION. Right now, there is no stable version of grub with ext3 support. Even if there was, it really won't gain you anything :-).

Run tune2fs, e.g.:

tune2fs -I 256 -O sparse_super,filetype,resize_inode,dir_index,ext_attr,has_journal,  
extents,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize /dev/sd[x][n]

Those are the default options for an ext4 file system if you were to create it with mkfs.ext4 (e2fsprogs 1.41.3 - see /etc/mke2fs.conf). I'm getting pretty damn good performance with this! The '-I 256' option sets 256 bit inodes, which most recent ext3 FSs use already. If this is the case, and you get a message telling you so, remove this option. Note that extents will make the FS backwards INCOMPATIBLE with ext3.

Next, edit /etc/fstab, e.g.:

/dev/vg/home /home ext4 defaults 0 0

Either unmount and mount or reboot your system. tune2fs marks the fs as dirty and performs a fsck and conversion.
NOTICE: distros with initrds may need to be regenerated or you won't be able to mount your root file system. In Fedora (replace kernel version with your own):

cd /boot
mv initrd-2.6.27.7-134.fc10.i686.img initrd-2.6.27.7-134.fc10.i686.img.old
mkinitrd initrd-2.6.27.7-134.fc10.i686.img initrd-2.6.27.7-134.fc10.i686.

That's all there is to it. Stay tuned for future ext4 developments like online defragmentation.

Also, ext{2,3,4} reserve 5% of space for root in case the drive fills up. On large modern drives, this can be excessive (e.g: 50GB on a 1TB disk). Consider running 'tune2fs -m 1 /dev/sd[x][n]' to reduce this to 1%.

For more information and tweaking:

Documentation/filesystems/ext4.txt from the latest kernel sources
http://ext4.wiki.kernel.org/index.php/Main_Page
man tune2fs
http://e2fsprogs.sourceforge.net/

NIN Echoplex live rehearsal

2008-12-19T04:12:00-07:00

One of my favorite blogs, Create Digital Music, ran a piece on NIN doing a rehearsal for Echoplex.

It's neat to see the Lemur, a touch screen MIDI controller, used live by the drummer. Overall the song structure is relatively simple but it is amazing how well they keep together. Notice the dude manning their monitoring setup full time and the in ear monitors 8-)?

I need to get a hold of an old touchscreen PC and see if I can convert it into a Linux MIDI controller with Open Sound Control and maybe some custom Java. It's sad because I had something perfect for the task a few years ago, an IBM Industrial panel PC, basically a laptop in a bulletproof magnesium case with touchscreen :-(.

If you are interested in this sort of thing, click through, otherwise enjoy the tune below.

http://createdigitalmusic.com/2008/12/17/how-they-work-nin-echoplex-rehearsing-live-with-lemur/

Also, check out Linux Journal's Java Sound and Music three part series.

NIN - Echoplex - Live at Rehearsals July 2008 from Nine Inch Nails on Vimeo.

Linux and the PC Mentaility

2008-12-17T22:41:00-07:00

Here's a minor thought dump of the moment: no hard numbers, just some general observations over the past year or two.

I've heard the comparison between Linux and other UNIX like OSes a few times in the past. Of course it is a bit hard to compare "Linux" - which is a kernel - and the others which most often are complete operating systems. Many times they even share a lot of GNU userland tools, so I will disregard userland.

The general feel I get from looking at some parts of Linux and its development process is that of the PC world. Build cheap, throwaway solutions and rip and replace later. Looking at things like the scheduler, FreeBSD was able to grow similarly advanced solution in a much more organized fashion without ripping and replacing this critical component several times and has far less developers. *BSD has a more traditional UNIX process from the days of the minicomputer and professional workstation, while Linux is deeply rooted in PC culture.

Aside from the scheduler, there seem to be other places where development has tarnish before shine. hrtimers, dyntick, CFS, and SLUB are some of the more controversial moves in the past year or so. Indeed, kernels 2.6.23+ didn't "feel" as polished as the few leading up to it and "felt" less responsive to me when playing audio or video concurrent with other tasks like compiling - something CFS was supposed to fix, but I never had a problem with before. Going into 2.6.28, these big changes are starting to settle down. Things once again "feel" better, more stable, and better performing - so I guess it is just a phase.

From a software engineering standpoint, it is interesting to ponder how to improve the process. One thing Linux has that many other projects don't is an enormous amount of contributors. A lot of people and companies are trying to different things with the kernel. It is important for them to get their code into the kernel as soon as possible, because it is such a fast moving process, but new features lead to bugs. Linux-next appears to be helping with keeping in progress trees so that developers can explore ideas before pushing to mainline, and Adrian Bunk's idea of the "long time support" kernels for those of us that don't run large distro kernels seem like a step in the right direction. I do wish he would forgo 2.6.27 and base it on 2.6.28 for the next LTS though. 2.6.27 has had quite a lot of major patching done since release and as I said earlier, 2.6.28 just looks better so far.

Another point: technically superior solutions don't always get favored. I've said before that I think there were better choices for file systems than ext3 for the past 8 or so years. CFS vs. RDSL is another example. The latest is SCST vs. SGST (http://lwn.net/Articles/311344/). SCST is clearly the only solution for the enterprise where Fibre Channel and Infiniband are likely to be used. And Ingo and Thomas are at it again reinventing the wheel with performance counters.

The counterargument is that Linux runs on most of the worlds supercomputers all the way down to small critical embedded systems. Most of these are running obsolete kernels with heavy vendor backports though.

Running the latest kernel is fun; after all I am a Gentoo user so I don't mind when things break, but for truly mission critical tasks I would not be afraid to look outside of Linux to BSD. Ironically, Solaris, once the stalwart of enterprise, seems to be embracing the Linux mentality.

USB 3.0 on Linux

2008-12-14T17:10:00-07:00

Take a look at this Intel developer's blog:

http://sarah.thesharps.us/2008-12-07-13-35.cherry

The video shows a USB flash drive transfering video at 125MB/s! To give you an idea of this speed, it is likely more than your hard disk can put out which is probably around 80MB/s. The rest of the article gives a good overview of USB 3.0 and she states that the bus should have about 400MB/s bandwidth in the real world. This is a breath of fresh air for external devices of all sorts, and I can't wait to see this on new computers.

The Linux drivers are currently under development, with the subsystem patches going into review. xHCI driver will have to wait until Intel finalizes and releases the specs. Anyways, it's safe to say that Linux should have USB 3.0 support as soon as products hit the market in the middle of 2009.

Bulletproof your server to survive Digg/Slashdot

2008-12-13T23:44:00-07:00

implementing scale up for web 2.0 sites with current practices

This blog was recently featured on Slashdot over the Thanksgiving holiday in the US. It was the perfect storm: commercial news organizations were mostly dormant creating a slow news day, and geeks like me were at home eager to get the latest technology scoop. What surprised me is how this relatively modest box, a Linode 540MB Xen Virtual Machine, withstood up to 100+ requests a second without even breaking a sweat. Furthermore, I had only performed some of the tuning I detail below. It scales to over 1100 requests a second after following my guide below!

I will detail how to tune your server for optimum capacity, or what I will call free scale up (as opposed to scale up by adding hardware or scale out - adding machines, database servers, application servers, load balancing - which may come in a future article depending on interest). Most of the ideas here are platform neutral - both OS and application server - assuming you are using a UNIX style OS.

The only tutorials I've found were dated and don't detail the latest practices like varnish or Passenger, so read on for a fresh look.

Audience

The intended audience for this article is anyone running a web site. Running your own web server gives much greater flexibility in choice of development environment. A dedicated server and certain virtual private server providers give much more predictable performance and wont cancel your service on a whim. (I'm looking at you MediaTemple... google for horror stories). A Linode VDS is much more flexible and very powerful for around the same cost.

Web Server

Most people use Apache. According to Netcraft, over 50% of hosts were as of November. For good reason, Apache has proven stability, scalability, and security. Some folks are quick to rip out Apache due to poor configuration and tuning. I personally find it to be an excellent choice for most sites because of the aforementioned traits and first-rate extensions. With proper setup, you will likely max your transfer or tax your application sever before it ever becomes the bottleneck.

Apache Tuning

The key to tuning Apache is to minimize RAM usage, especially on a limited machine like my 512MB Linode. Memory swapping of applications to disk is almost entirely unacceptable on modern servers. Disk I/O is very expensive and the biggest bottleneck on modern computers, which is why swap is so unappealing.

Therefore, you need to:

Limit overall Apache memory usage
Minimize per thread/process memory usage
Minimize disk I/O

Limit overall memory usage

Step one is very important. If your server begins swapping heavily, it can be very difficult to even log on and perform administration. You need to develop an idea of the RAM an average Apache process is using via top, ps, or another monitoring framework. Make sure you are looking at the RES column in top, since shared libraries will be used between all processes. Take this number and divide it by the amount of availalbe RAM. Available RAM should take into account RAM used by other processes including your database when under reciprocal load. Set the MaxClients directive to a number close to the resultant, and tune accordingly with benchmarks (see Benchmarking section).

Minimize memory usage

Step two determines how many child processes you can handle. This is important because the more children, the more in flight requests, and lower end user latency. This is also a lot more environment dependent than step one.

A good way to reduce memory consumption is to unload unneeded modules. Most server operating systems default Apache with a wide range of modules that are probably not used on your site including several basic authentication methods. Using shared objects rather than static modules will help memory usage as well, and most distributions ship this way.

If you use an Apache module for your application server (mod_php, mod_perl, mod_python, Passenger aka mod_rails), each child process will consume the memory of that module regardless of whether or not it is serving a static asset (images, css, etc.) or an application page. Mitigate this by using a proxy (see next section) or moving application serving to its own processes via FastCGI (PHP, most others), AJP (Java, Python), WSGI (newer Python), proxy (Ruby, all).

Disable logging

I should take a moment to step back and hit on an important topic. Hard disks have improved very little in regard to performance in recent years. Disk I/O is an expensive task and therefore the primary bottleneck you wish to avoid.

When Apache logging is enabled, a write operation must occur for every hit. If possible, consider completely disabling access logging. You can outsource web statistics to Google Analytics. If you require logging, make sure HostnameLookups is disabled (network I/O is even more expensive than disk!) and batch look-ups on another machine or during idle periods with a log analyzer. As your setups grows (scale-out), log files will become more cumbersome and you will probably be logging to database or a central server anyways. Varnish, a proxy/http accelerator detailed below has an optimized design for logging.

mod_cache

Apache has an integrated cache module that will keep frequently hit static assets in memory. For larger sites, forgo this and use a proxy which will be more flexible and allow easier scale-out.

MPMs

Apache makes use of MPMs, or Multi-Processing Modules, for its core functionality. The default on UNIX is prefork, which makes a separate process for each request. By switching to a threading MPM such as worker or event, you can cut down overhead and memory use. Some modules do not play well with threading (PHP), so you should research before changing MPMs. prefork works well for one and two core servers.

Alternative Web Servers

Lighttpd is the leading alternative FOSS web server. Users include A-list web sites such as Youtube and wikipedia. Benchmarks show impressive performance. Keep in mind Apache is by no means slow nor resource intensive and links on that page show that it is faster on some workloads.

When making comparisons, keep in mind that by design you will probably be using a FastCGI application server and most of the optimizations above will hold true for Lighty.

For sites with long connection times (download servers, AJAX keep-alive) or static content servers, I would definitely lean toward it (scale-out).

Nginx has also been picking up steam (pun intended) and is being used by large sites like Wordpress.com. I would consider it in the same class as Lighty.

Reverse Proxying

A reverse proxy is very useful for modern web serving. Even with just one server, a reverse proxy will keep common pages in memory - greatly reducing disk I/O. They will also keep static requests from using potentially heavy application server HTTPd processes. These are often very fast at basic HTTP since they are not concerned with all the features of a web server. When it comes time to scale-out, the proxy can be moved to a separate server. Proxies can direct traffic to different backend servers. Proxies can even be placed in geographically disperse areas (think CDNs: Akamai, Limelight - Youtube, Google). Logging, compression, and SSL can be offloaded to the proxy. In short, you want a proxy even on a single server (or at least mod_cache).

Varnish

Varnish bills itself as an HTTP accelerator. It was written from the ground up to perform reverse proxying, and this it does well. The Varnish design philosophy is enlightened and leaves a lot of the work like memory management to modern advanced operating systems. Logging is performed in a separate processes and is optimized. If you need an advanced proxy and accelerator, this is likely the way to go.

Squid

Squid has traditionally been used as the de facto FOSS forward and reverse proxy. Many large sites such as Wikipedia are extensive users.

Apache and Lighttpd

Both Apache and Lighttpd have modules that will allow them to cache and reverse proxy. For single server setups, it would probably be worth reusing the components of your web server (think: shared memory) if your application server is external. mod_proxy is very useful for forwarding ruby requests to a Ruby web server like mongrel or thin.

Application Server

The application server is where most of the magic happens in today's web 2.0 sites. Gone are the days of static HTML files. Most sites are now dynamically generated every visit, and customized per visitor. This is an order of magnitude more complex, and a lot of CPU time is spent on page generation. Therefore, tuning here is often one of the best things you can do to improve site scalability.

PHP

PHP is the most widely deployed language on the web. Many extremely popular applications are written in PHP, including: MediaWiki, Wordpress, Drupal, and phpBB.

Opcode Cache

By default, PHP breaks a script down into opcodes every time it is called. Opcode translation is necessary to simplify programs so they can easily be parsed by the Zend Engine. It is unnecessary for this to be done every time a script is called since the source code will rarely change once deployed. Luckily, a cache can be added that will eliminate this step. The net performance gain can be a factor of 2 to 10, very impressive for a simple install!

These days, you should chose APC - The Alternative PHP Cache. Once upon a time, there were several choices here. Turck MMCache was notably fast, beating even the commercial Zend Suite, but mysteriously died out (the original author is now a Zend employee. hmm.. coincidence??). Others have tried to revive it in the form of eAccelerator, but it isn't stable nor active. Any other arguments are moot point since APC will be part of PHP6 core as well as having PHP's founder as a developer.

Modules

Just as with Apache, removing unused extensions in PHP will help reduce memory usage. These can be commented out in php.ini.

Rails

Rails has gained a lot of steam (okay I'm wearing that one out) and is a favorite among many Web 2.0 startups including Twitter.

A lot of Rails scalability problems are due to the underlying Ruby language. The garbage collector, threading and memory allocator have been pinpointed to be particularly bad. Work is underway to fix these in Ruby 1.9 (bytecode) and 2.0(threading). In the mean time, consider Ruby Enterprise Edition in tandem with Passenger. Personally, I'd rather avoid Ruby and all you kool-aid drinkers (but I've done a large deployment of Passenger). Go Python :).

Python

Python is just a plain good language. With that out of the way, like all the other scripting languages, Python is supposed to be getting a bytecode implementation sooner or later. Psyco can yield an average 4x performance improvement and is available now. PyPy should be here sooner rather than later.

Java

Due to the Java language design, code is JIT (Just-In-Time) compiled and you don't have the compilation problem that the dynamic languages above do.

Java web apps are immensely complex, and aside from the latest JDK (1.6.0.10), your container will play a big role in speed. Jetty and Tomcat are always good choices.

Databases and Database Caching

A large portion of modern web applications are database driven. To keep your site running, this point of contention must be addressed. MySQL is ubiquitous and known for its speed. PostgreSQL offers some advanced features and is known as the DBA's FOSS database. If you need extreme scalability, consider DB2 but prepare to pay dearly :-).

MySQL

MySQL comes configured fairly well out of the box in most distributions. MySQL Performance Blog sums it up better than I can, so head that way for basic tuning info.

Probably one of the easiest things you can do is enable the integrated query cache. The good news is your application doesn't need to do anything to take advantage of this.

in my.cnf:

query_cache_size = 64M

For single server web workloads, this simple change can work miracles and prevent dreaded MySQL connection errors. This is especially true since web apps are primarily read oriented. The query cache isn't perfect in all situations, and in larger sites memcached is more appropriate but has its own disadvantages (see memcached section).

PostgreSQL

PostgreSQL should also be set up fairly well by your distribution. shared_buffers should probably be tuned, as well as max_connections. See the PostgreSQL wiki on tuning for a good overview.

There is nothing strictly akin to the MySQL query cache, for better or worse.

Applications and Application Caching

This is potentially the hardest step to implement, yet can also yield the greatest reward. Caching common database queries, objects, modules, or even writing static HTML versions of a page can cut server load to nothing. If you are using a common FOSS (free) or COTS (commercial) product, chances are the software already implements some of these options and they may just need to be activated or downloaded as an extension.

Keep in mind not all things are effectively cached, and you may need to perform a major rework to implement aggressive caching like this.

Generic Data Caching - memcached and APC

Many common applications contain backends for caching against memcached or APC. Mediawiki is a prime example of this, which integrates nicely with memcached or APC. If you are writing your own apps, using a memory cache can greatly reduce dependency on the database.

memcached

Realizing that databases have a lot of constraints, the folks at LiveJournal.com wrote a generic caching framework called memcached. Most large sites such as Facebook, Wikipedia, and Slashdot are all using this.

The bad news is you have to port your application to store and check against memcached. Database queries are a prime target, but just about anything can be stored here.

It is also handy for scale-out because you can add dedicated cache severs.

APC user cache

PHP APC users can manually store information in APC's shared memory. This is ideal for single server solutions. Take a look at this performance comparison vs memcached and files.

Application Caching

Although most pages are dynamically generated these days, a lot are needlessly so. For example, a content management system might include a header, content, comments and a footer. This output can be updated and written as a static HTML pages when an author updates them. Static pages are then served until a user comments on an article, which triggers a cache invalidation and the page is rendered and stored again. The output of generated menus, columns, and other objects can be stored in cache form as well.

Wordpress Cache Plugins

Wordpress has a couple of plugins that are mandatory for large sites.

WP Super Cache will generate static HTML files of posts on your blog. They are automatically served via some mod_rewrite magic, and will expire and update automatically. This can effectively reduce load to almost nothing - it completely eliminated database access and PHP execution.

WP Widget Cache is a nice addition that will cache output of widgets (sidebar elements such as menus) that don't commonly change.

Benchmarking

It is important to benchmark your site after making changes to see if it meets performance expectations. ab is a common tool for this task.

The following will run 10 concurrent requests for 3000 total against localhost:

ab -c10 -n3000 http://localhost/

Be very careful when benchmarking a live site. You could effectively Denial of Service your server while it is processing all those requests.

What do you think?

I'd be happy to hear your stories from the trenches. Please share your tuning advice!

Tux3 by Christmas?

2008-12-10T17:43:00-07:00

Development seem to be going well for Tux3.

Daniel Phillips of Tux3 just posted the following to the LKML:

The big goals for Christmas (this Christmas!) are:

SMP locking

Atomic commit

Posixly complete

Rudimentary fsck

With the following comical reference

With atomic commit, we will progress from "buggy Ext2 equivalent with missing features" to "buggy Ext3 equivalent with missing features".

Not a bad place to arrive at in five months, starting from scratch. Does anybody out there still doubt that the community process works, and is the best way to develop really complex software? Believe it.

See the whole post here: http://lkml.org/lkml/2008/12/10/358

Retrocomputing for Fun and Profit

2008-12-07T14:22:00-07:00

Buy Old Computers
???
Profit

What is retrocomputing?

I define retrocomputing [wikipedia] as the collecting and use of old computers. Why might one do this? Well, for one, enterprises cycle out machines fairly frequently. 2,3,4 and 5 year old systems are often sent out to scrappers in droves despite still being plenty useful. Top of the line systems for large companies often have more than enough power for small and medium sized ones at pennies on the dollar compared to new hardware. These machines are likely complete overkill for home use, but none the less are very useful for fun and learning.

IBM mainframe ops in the 1980s

Why?!

A lot of what I know about computers has been learned on old machines. Hooking up a couple of servers and desktops and trying to make something useful out of them is a great exercise for the aspiring system administrator. With open source software, it can all be done freely and easily.

Yes, you can run Linux, BSD, and Solaris from the comfort of your Windows desktop in a virtual machine (weak sauce...). Yet there is something much different when you cluster several high technology servers together, tethered to a Fibre Channel storage array and have them share a single distributed file system. The knowledge of setup, installation, and troubleshooting I've gained from mock scenarios like this I cannot compare to anyone else I've ever met. Breaking things here usually means digging deep and fixing it. If you were to screw something up at work like some of the things I've gotten into, it would probably cost you your job.

Retrocomputing is also fun. I am personally into old IBM hardware, though old UNIX workstations of all sorts are interesting to me. You can see my collection of IBM PS/2 and RS/6000 knowledge here: http://ps-2.kev009.com/. There is a particular thrill to booting up a machine that cost between $20,000 and $50,000 10 years ago. Knowing that these same machine models were used to design the Boeing 777, composed the famous Deep Blue machine, and were used in the largest automotive and shipbuilding firms not to mention some of the most important space craft to date also brings a sense of power and nostalgia. In some ways its similar to having a classic car, but different. Maybe if that classic car was a big ass bulldozer, tank, jet or some other well engineered piece of equipment :-P.

Some old systems I had at one time or another. Left to right: IBM PS/2e (first "green" environmental pc), RS/6000 43p (7043-140), Apple PowerMac 7100/80, RS/6000 7006-42W, RS/6000 7012-397, HP Visualize c360 (PARISC)

Nostalgia is one of the biggest things I get out of using particularly old hardware. I missed the mainframe days, the minicomputer days, the PC and DOS days, the Apple II days (well, actually I used these a bit at a very young age), and to a degree the early Windows days. Just like a history class, studying these old machines gives me insight as to why things are done the way they are today. It gives me appreciation for modern systems and makes me write clean and well optimized code. The old computer games that captivated me as a child (Sim City, Sim Tower, Sim Ant, Sim Farm, Gizmos and Gadgets, The Incredible Machine, Oregon Tail etc.) implanted a high degree of logic and understanding at a young age and it is heartwarming to revisit these. I grew up a Mac user as well, so seeing what I was(or: was not :>) missing on PCs is also interesting.

Old MIPS UNIX server booting and logging in

Some of the benefits of retrocomputing:

Enterprise class hardware
Cheap, possibly even free
Different design philosophies - not everything is x86 - a lot of this gear is quite different. For example, UNIX workstations integrated most of what we enjoy on our PCs years before it became available to consumers. SGI machines were doing A/V and 3D in the early 90s. IBM midrange AS/400s have an advanced integrated database, programming languages, and environment that make PCs look like a joke for business programming. WinFS, Object Storage Devices, etc are just now being talked about for PCs. The channel philosophy from mainframes is still pretty new to PC servers (fibre channel), not to mention virtualization.
If you break it, you can fix it and learn from it or toss it
The engineering and craftsmanship in some of these systems is downright astonishing
Old computers are works of art: they give you a window into the technology and culture of times past
You should never trust a computer you can't lift

It is interesting that we as humans produce such elaborate machines, only to discard them as scarp a few years later. It is humbling and shows you the incredible progress we are making.

How?

eBay is your friend, but also look for local scrapyards or businesses doing overhauls.

If you are faint of heart, plenty of good abandonware sites exist for games and operating systems that can be run on emulators or VMs. Check out this IBM mainframe emulator, Hercules. Some of the original IBM OSes are public domain.

If you don't want old PCs and big iron overtaking your house, there is plenty of good material on YouTube as well. The Computer Museum is a good start. Some of the consoles, offices, and outfits are hilarious.

Old SGI tech demo - pretty impressive!

Phoronix Benchmarking.. Statistically Significant? And Other Performance Concerns

2008-12-06T23:43:00-07:00

Phoronix has been cranking out a slew of benchmarks recently, pitting various different Linux distros against each other and even different operating systems with their own automated test suite.

What I would like to know is... are they bullshit? I'm no statistician, yet the proximity of the numbers and lack of error bars raise my own bullshit detection meter. See this URL for some background on statistical significance and error bars: http://www.graphpad.com/articles/errorbars.htm.

I spot plenty of outlandish things, such as FPS benchmarks for file system tests and JDK version changes through product life cycles, also to mention the somewhat unfairness of crappy binary graphics drivers across versions. JDK 1.6.10 is SUPPOSED to be faster; results from benchmarking against this are insignificant unless it is run across all versions. Yes.. GCC, glibc, and the kernel change between releases as well but these are not typically components that a user can swap out as easily as a JVM which should probably be bumped for security reasons anyways.

Furthermore, I realize all benchmarking should be taken with a grain of salt - one particular set of hardware and software will never map correctly to another set of hardware or software, but it should be possible to set up tests to gain some useful intelligence.

Can this kind of macro/micro benchmarking (depending on how you look at it) help weed out regressions? GCC 4.0 was noticeably slower on x86 than 3.x (See: http://www.coyotegulch.com/reviews/gcc4/index.html, http://people.redhat.com/bkoz/benchmarks/). At the same time I think PowerPC saw significant improvement due to auto vectorization and use of Altivec/VMX. But it also seems to be improving over time. I've heard 4.4 is supposed to be much better with a new register allocator (IRA). This probably the most important component of modern open source operating systems, so some of the blame might be placed here if the numbers have meaning.

All of this makes LLVM look more and more appealing. LLVM is able to do not only compile time, but also link and run time optimization. This is very appealing for commercial software where you are given a binary blob by the manufacturer that will likely that will not change through its lifetime. It also reminds me of Java and speedup through JVM upgrades, except this should work on any language.

"LLVM is... designed to enable effective program optimization across the entire lifetime of a program. LLVM supports effective optimization at compile time, link-time (particularly interprocedural), run-time and offline (i.e., after software is installed), while remaining transparent to developers"

One thing the Phoronix numbers do show is that things seemed to go down hill coincident with CFS (Completely Fair Scheduler), dyntick, and SLUB merging as well.

Evgeniy Polyakov of POHMELFS fame raised the alarm with some fairly significant networking regressions - how financial crisis affects tbench performance - that seem to support a general slowdown between 2.6.22 and 2.6.27. This resulted in noise on LKML and hopefully we will see improvements soon.

I guess what I am getting at is that compute power is so cheap that it seems stupid to not have automated tests against such things these days. Diego Petteno of Gentoo fame has been doing such things recently with Gentoo's excellent build system. I have set up Hudson, a Java Continuous Integration system, before to track commit regressions and such a system seems ideal for all modern software testing.

Anyways, I am interested in hearing your thoughts on benchmarking, software testing, and automation and how it can be used to improve modern software.

KDE 4.2 beta 1 on Gentoo

2008-12-05T15:59:00-07:00

KDE 4.2 is set for release on January 27th. Eager to see what is new and improved, I installed beta 1 on my Gentoo box. KDE 4.1.80 was tagged then released late last month after the feature freeze deadline. This is a snapshot of the current development tree that will eventually be released as KDE 4.2.

A quick blurb about KDE 4.1:
Busy with other things, I never wrote about KDE 4.1 on Gentoo. I've been using KDE 4.1 since 4.1.1 with the kde-testing overlay which has since entered portage and been bumped to 4.1.3. This is a good release and well worth at least testing if you are a KDE user. I've found it to be quite stable and use it exclusively for every day work. The Gentoo KDE team has done an excellent job creating ebuilds for it.

What's new?

Plasma received a lot of polish and is beginning to eclipse Kicker and indeed all other desktop and panels that I have used. Much needed features such as changing the panel height, auto-hide, and screen edge selection have been added. The task bar is highly configurable in typical KDE fashion, allowing you to define task grouping, sorting, filtering based on current desktop or screen or minimized windows only, as well as allowing manual grouping. The system tray also now allows hiding of unwanted tray icons.

Here's a screen shot cluttered with various plasmoids for demonstration. It's nice to see the community thinking up some fun and useful plasmoids.

I wasn't the biggest fan of the KDE 4 default menu. Luckily, the Lancelot menu has been accepted upstream and is now an option on stock installs. This menu is great for finding new applications (esp. new users) as well as thumbing through with the keyboard.

I'm a Firefox user, but occasionally will fire up other browsers for testing or to avoid restoring a large previous session if I am in a hurry. I'm happy to say that Konqueror feels much faster. It also seems to work much better on AJAX heavy sites such as Facebook. When I spoofed the user agent to report Firefox 2, Facebook chat worked fine, an improvement from 4.1. The continued merging of Webkit is clearly beneficial here.

One of my favorite KDE apps from 3.5, Ark, is also finally reaching feature parity. I missed shell integration with Dolphin/Konqueror quite a bit and am happy to say it has returned.

Notifications are displayed and stack nicely in the lower right corner. Operations such as downloading and moving files will show their status here.

Kontact gained usenet support by means of Akonadi, which now has support many for data sources. I think I will switch from Thunderbird/Lightning to Kontact with this release.

Kate has a new VI editing mode. This is quite a nice text editor.

Amarok is shaping up as well. This is 2.0 RC1, so it should be released on a date close to KDE 4.2. Take a look at the different Internet media sources. Last.fm support is now top notch!

Digikam 0.10-beta5, which seems to be stabilizing and evolving nicely, is another nice app. Bonus points if you can identify all the retro machines. The one on the left was probably the worlds first "green" PC.

Okteta, an easy to use hex editior, has also been updated.

I could go on and on showing the great progress. I hope I hit the highlights, but you can check out the feature plan for yourself here: http://techbase.kde.org/Schedules/KDE4/4.2_Feature_Plan. Noteworthy changes include improved multi-display, better desktop search with Strigi, and integrated power management.

Gentoo Installation

First, a shout out to the Gentoo KDE maintainers and testers. Creating ebuilds for fast moving snapshots and live sources for a project this large is not an easy task.

Installation on Gentoo is fairly easy if you have layman. 'layman -a kde-crazy' will add the KDE crazy overlay which has KDE 4.2 unmasked and ready for testing. If your box is ~arch, it should be as simple as 'emerge -av @kde-4.2' (see comments below for more info). I recommend using the kdeprefix USE flag if you wish to test development releases so you can fall back to stable if things aren't working correctly. This will slot 4.x releases.

If you want a stable and usable environment, I still recommend sticking to 4.1.3 at the moment. If you run a mostly stable Gentoo with KDE3.5, you can find a package.keywords file in the kde-testing overlay as well as some other minor goodies. These versions slot effortlessly so it isn't a problem switching back and forth.

Conclusions

KDE 4.2 has come a long way since 4.0 and is a nice steady improvement over 4.1. As I stated earlier, I use KDE 4.1.3 as my only desktop environment and am extremely pleased with it. I have had no major issues and have had uptime of over a month in the past without crashing/restarting KDE - so the good old KDE 3.5 stability seems to be returning. By trying the current beta out, I have no doubt that 4.2 should be just as stable by release.

Also, if you are an Nvidia user you owe it to yourself to try the latest 180.xx+ drivers. As people have long been saying, much of the performance problems they were describing were related to Nvidia cards and poor video drivers. With the new drivers, KDE is lightning fast.

Here's to an on time and successful KDE 4.2 release. I can't wait to see what QT 4.5 and KDE 4.3 will bring!

More Linux File Systems

2008-12-03T23:03:00-07:00

It seems I caught the wave of interest in Linux file systems. Here are some articles worth checking out:

LWN has done an excellent overview of Tux3: http://lwn.net/Articles/309094/
Phoronix subjects ext4 to performance tests against ext3, XFS, and reiser3: http://www.phoronix.com/scan.php?page=article&item=ext4_benchmarks
Btrfs performance analysis: http://btrfs.boxacle.net/

Of course, there are some other file systems that I haven't talked about that came up in the comments of my last post. Most of these are special purpose, still on the fringes, or different in scope than the first article which was about local storage.

The native flash file systems: UBIFS and LogFS. In theory file systems like these would be ideal for SSDs, but we need manufactures to stop putting FAT/Hard disk emulation and wear leveling into their drives. Windows is the culprit for this. These file systems also seem keyed toward embedded device flash memories at the moment and not general purpose storage. Neither are upstream yet. This could be positive, as the disk format and code can be readily changed to make them compatible with future SSDs if there is a change in manufacturing. Val Henson, a Linux file system authority, has some interesting thoughts.
The Log file systems: LogFS (flash centric, see above), NILFS. I believe ZFS and Tux3 share design philosophy from these. The idea has been around for a long time but none have ever really succeeded.
The shared disk and distributed parallel file systems: OCFS2, Lustre, GFS, PVFS. There
are a laundry list of these. Some implement entire disk file systems, while others add clustering or distributing properties to other file systems. SUSE [link] and I think Red Hat even considered using one as the default file system but booting (Grub) is an issue.
Network file systems: these add networking to other disk file systems. POHMELFS and CRFS (distributed extension for Btrfs) are interesting new ones here. Of course there are a laundry list network file systems for Linux. NFS, AFS, and CIFS are the old timers.
The others: These are more experimental or research oriented. chunkfs, spadfs, and many more. Many others just don't have steam behind them yet or are dead in the water.

My overview is clearly Linux oriented, though I mentioned ZFS in passing because I think it spurred a lot of these recent developments. That isn't to say the BSDs are sitting still with HAMMER, and FreeBSD is keeping UFS2 moving forward while NetBSD has LFS - a log file system.

P.S.:

I plan on performing some benchmarking soon after 2.6.28 goes stable. The list will include ext2/3/4, JFS, XFS, Reiser3 and Btrfs. The setup will include single and multi-disk configs. If you have any requests or suggestions for setup, please contact me.

On File Systems

2008-11-29T03:07:00-07:00

Update: More Linux File Systems

Introduction

Although the file system is one of the most important pieces of an operating system, we generally put little thought into them these days. Put bits in, pull bits out. It usually works well enough for desktop systems - until the power fails - but even that is usually pretty painless these days.

On Linux, there are many contenders in the file system arena. ext2 had been the standard for many years, though around 2001 and onward a few other choices become mainstream. Without delving into too much history: journaling support was added to ext2 in the form of ext3, ReiserFS is released, SGI ported XFS, and IBM ported JFS in no specific order. For a few reasons, mostly political, ext3 becomes the de facto file system for Linux.

Classic File Systems

In what I will refer to as "classic" file systems, the idea is basically the same. They essentially bolt journaling to the traditional file system UNIX layout. Here are the highlights from each of these:

XFS is hailed by some for its excellent support of large files and large file systems, and has some nice modern amenities like extents, delayed allocation, and online defragmentation. [http://en.wikipedia.org/wiki/Xfs]

XFS is not without a fair share of faults though. It's a somewhat half hearted port in my opinion. Mainly due to its IRIX roots, performance is usually comparable to the other era file systems but CPU usage is relatively high [http://www.debian-administration.org/articles/388]. Data atrocities after power outage or machine crashes seem to be common and even design decision. This LKML posting really struck a nerve with me: http://lkml.org/lkml/2007/3/28/316. A lot can go wrong on the hardware side of things, and if a file system is not privy to this it is a recipe for disaster. ZFS is a hero here with checksumming, which I will touch lightly later.

It is worth noting that XFS is still under active development and has a decent roadmap forward [http://www.xfs.org]
ReiserFS (Reiser3) is one of the first journaling file systems for Linux. It has some initial growing pains, but is quite a nice file system by kernel 2.6.

Performance is excellent with small files, but its scalability is questioned by many of the Linux elite [http://en.wikipedia.org/wiki/Reiserfs]. Hans Reiser then goes on to work on Reiser4 which is essentially a rewrite. SUSE Linux keeps a few developers on it, but it becomes pretty clear that this is a doomed file system [http://www.linux.com/feature/57788].
IBM's JFS is another of the UNIX ports. JFS traces its lineage back to IBM's AIX in 1990. An IBM team ports and improves it for use in OS/2 and later releases the code as free, open source software (FOSS).

The resulting Linux file system is noted for being scalable, resilient, and in particular easy on the CPU [http://en.wikipedia.org/wiki/JFS_(file_system)]. It also includes extents support. For whatever reason the kernel community and distros don't really latch on to it. JFS is basically just slowly maintained throughout its life cycle.
ext3 is a journaling extension of the Linux native ext2. Out of all the file systems in this generation, it is probably the least technologically advanced lacking features like extents. What makes up for this is the easy upgrade path for ext2 users, relatively simple code base, and broad upstream adoption. [http://en.wikipedia.org/wiki/Ext3]

Broad upstream adoption makes ext3 winner for most users and distributions, and it is now the most stable and supported Linux file system.

In hindsight it seems somewhat tragic that JFS or even XFS didn't gain the traction that ext3 did to pull us through the "classic" era, but ext3 has proven very reliable and has received consistent care and feeding to keep it performing decently.

Nextgen File Systems

In 2005, Sun Microsystems released the bombshell ZFS file system. This ushered in the era of what I will call "nextgen" file systems. As hard disks have gotten larger, strategies for backup, integrity checking, and support for large files have become much more important. These file systems also aim to ease management by blurring the traditional VFS line or offering tight integration with LVM and RAID. Silent corruption by bad hardware is also cause for alarm, and checksumming has been baked into some of these "nextgen" file systems to counter this.

In many ways, Linux was caught completely off guard and most developers weren't thinking very hard about the future of file systems prior to the ZFS release. Reiser4 explored some interesting ideas and aimed to be a killer file system (okay I'm really tasteless...) but Hans Reiser enjoyed a particularly bitter relationship with other kernel devs. Luckily, some even more advanced file systems have come into existence recently.

Reiser4 was the first effort for a next generation Linux file system. Introduced in 2004, it seemed to have some excellent new technology including transactions, delayed allocation, and an interesting plugin architecture for adding features like encryption and compression. Hans Reiser even advertised using the file system directly as a database with advanced metadata.

Reiser, the primary developer, often rubbed other kernel developers the wrong way when championing his new file system. Hans seemed to get defensive when questioned about code style and design decisions - particularly that of the plugin architecture. I think that a lot of this was due to misunderstanding and bad tempers, but to this day Reiser4 has yet to enter Linus' kernel tree. With Hans Reiser's murder conviction in 2008, the future of Reiser4 was frequently called into question. At this point it seems unlikely that Reiser4 will ever see upstream adoption, but some of the ideas explored have already been integrated into other "nextgen" file systems. [http://en.wikipedia.org/wiki/Reiser4#Integration_with_Linux].
ext4 was started as an effort to make a 64 bit ext3 to support large file systems. Later, others (Lustre, IBM, Bull - see [Theodore T'so comment below, #25 got involved and added extents, delayed allocation, online defragmentation, and more. [http://en.wikipedia.org/wiki/Ext4]

ext4 enjoys forward compatibility with ext3, and limited backward compatibility if extents are not enabled. Again the clear advantage here is that it improves upon the stable ext3 base, provides an easy migration path, and has many great developers working on it. However, it needs to be said that ext4 is still somewhat of a "classic" file system and doesn't have the level of features and scalability that the other "nextgen" file systems do.
Btrfs is clearly Linux's response to ZFS. Started by Oracle, this project has gained backing from all of the major Linux corporations. Traffic on LKML suggests that this will be the file system to carry on the torch from ext4.

Btrfs' key design feature is copy-on-write which will allow for inexpensive snapshots useful for backup and recovery. The goal is to completely surpass ZFS however, and many exiting features such as data/journal checksums, tight device-mapper integration, built in RAID, online fsck, SSD optimization options and even in place ext3 upgrades are being worked on [http://btrfs.wiki.kernel.org/].
On the tails of Btrfs, another advanced file system called Tux3 has been announced. The project's developers have been making use of FUSE to quickly prototype and test ideas. The initial work on a kernel port has just been posted [http://lkml.org/lkml/2008/11/26/13].

This project aims to do away with traditional journaling, instead playing back the logs (or recovering) on every mount. It will also feature inexpensive snapshots and versioning. The project's developers seem to be quite good at championing their ideas, but I predict it will be up to 3 years before we see this ready for mainstream use since coding has really just begun [http://tux3.org/].

Conclusions

With ext4 coming out in kernel 2.6.28, we should have a nice holdover until Btrfs or Tux3 begin to stabilize. The Btrfs developers have been working on a development sprint and it is likely that the code will be merged into Linus's kernel within the next cycle or two [http://www.heise-online.co.uk/news/Kernel-Log-Ext4-completes-development-phase-as-interim-step-to-btrfs--/111742].

It seems pretty clear that Solid State Disks (SSD) will be here for good. Theoretically these should blow magnetic storage away in terms of speed. We are already starting to see competitive write performance, and random access and IOPS are very impressive with the latest Intel SSDs. It is good to know that Btrfs plans to incorporate SSD optimization from the start, but these new devices may warrant yet another file system to achieve maximum speed. I personally think that wear leveling and FAT emulation are holding these devices back and would be better implemented by the file system.

P.S.:
I'd been meaning to write this article for a while, but things have been changing rapidly with the introduction of ZFS, Btrfs, and Tux3. I plan on doing benchmarks soon with kernel 2.6.28 against all the classic file systems, ext4, and Btrfs so subscribe to my RSS feed if you are interested. Any comments, corrections, or questions would also be appreciated!

P.P.S.:
Wordpress seems to mangle the format of this post so the bullet justifications are off.

KDE4 on Gentoo

2008-05-16T11:18:00-07:00

So I bit the bullet and installed KDE 4.0 on Gentoo. Version 4.0.4 recently hit the tree, and with some minor hackary to package.unmask and package.keywords I have a nice spartan KDE 4.0.4 desktop that I am typing this in.

My observations:

Like Gentoo's KDE3 packaging, you are given KDE4 pretty much as it is packaged upstream. It certainly lacks the polish that Fedora 9 and OpenSUSE 11 have been able to apply.

The desktop is somewhat of an afterthought at this point. Basic icon grid alignment and drag selection don't work. Icons on the desktop are treated as widgets which means you can rotate them and such which may be something to build on in the future. However this area basically needs work.

Oxygen is beautiful. Seriously, everything looks stunning and it does not get in the way. It's not gaudy. Very professional. Still room for improvement though, I.E. icon set isn't complete, consistency.

kwin's built in composting is awesome. This is exactly the kind of thing I've been waiting for. It doesn't have the heavy weight and gaudy appearance of Compiz or Vista. This will be a major productivity boost. Tabbing through windows, expose-zoom overview, and a heads-up of virtual desktops are the things I will use. The light shadowing and animation are also refreshing and professional. Nothing to get in the way here, it is stable and speedy.

The application launcher is a major pain in the ass. The integrated search is handy, but it just takes too many damn clicks to find an app otherwise. This will be especially unwelcome by new users since you might not know if the app you want is in 'Internet' or 'Development' or 'Multimedia' or even 'More Applications' under any of these. Every time you click the menu it starts from the favorites pane and does not remember the category you were in. I don't think the Windows 95 style start menu is the final or best UI answer, but it beats this Vista style menu and the OS X dock abomination hands down.

Basically, plasma as a whole needs work. Widgets seem pretty glitchy. Also responsible for the above desktop nuances.

Apps are hit or miss. Gwenview is great for thumbing through pictures. okular does the same for documents. Dolphin works very well for most tasks, while Konqueror is there for you otherwise. Many of the games have been updated and are pretty good. Marble, Kalzium - just plain cool. Ark was one of my unsung heros on KDE3. It is severely lacking in KDE 4.0. I hope that it gets some attention, and shell integration back too! A lot of other utilities are either absent or need work as well.

Amarok 2 has yet to be released, but this will be a major influence when it is.

Speed! KDE4 is quick. This is a welcome relief. Once KDE4 starts to gel, I'd imagine it will run at least as well as KDE3 on old hardware. Build speed was much quicker with cmake, which is very welcome on Gentoo.

Stability is much improved in this release. It still is not business-grade. Trivial apps seem to crash on exit and such.

The Weird? The default window size for just about everything seems unnecessarily small. I find myself having to resize pretty much everything before I can use it.

In conclusion, KDE 4.0 is a giant leap forward but it is a work in progress...

If you are migrating from KDE3 on an old install, best wait as there is some adjustment. KDE 4.0 as packaged in Fedora 9 and OpenSUSE 11 seem quite usable however. I wouldn't hesitate to install them on a new PC. I will certainly keep KDE 4.0 installed and re-evaluate with 4.1, but KDE3 has a better work flow for my use at the moment.

It reminds me a lot of Apple's OS 9 to OS X transition. I've been following the development for some time. From the alphas, betas, and release in January, I'm quite surprised by where we are at today. Things are much better than they were just six short months ago. A lot of work has taken place and we are really forging the foundation for a first rate competitor in the PC desktop arena for the next 10 years. KDE 4.1 should be be released near the end of the summer. I believe it will be enough to convert many KDE3 users over. Things should really calm down and align by 4.2 and it will no doubt in my mind be the best desktop environment whatsoever at this point.

Ableton Live 7 MIDI Problems

2008-05-15T03:59:00-07:00

A lot of people are having trouble with Ableton Live 7. This is especially true with MIDI functionality. It stems from the fact that Live 7 defaults to the DirectMusic MIDI interface. Apparently DirectMusic then has to do emulation because the standard Microsoft MIDI driver is not DirectMusic native (Microsoft, we will never understand you..). This is true for most USB MIDI devices.

This was affecting me by introducing high latency into MIDI I/O with my Novation Remote 61 SL and Akai MPD24. Initially all seemed well.
Gradually, hitting a note on my keyboard would get slower and slower until the point where it was unusable. We're talking hit a key and wait tens of seconds before the note plays.

Luckily the fix is pretty easy. Open up the MIDI preferences, and click the small arrow next to all your inputs and outputs. Now simply click the port type and it will change from DirectMusic to MME.

According to the forums, there are no drawbacks to this approach. It is unfortunate that Ableton decided to make this a default since so many people have had troubles with it.

Fedora 9, KDE 4.0 done right

2008-05-13T22:43:00-07:00

Wow!

I just downloaded the Fedora 9 KDE live spin. I am impressed. I tried a few of the betas along the way, but this is very polished. Much better than Ubuntu's try at KDE4.

It feels very responsive and works very well, even as a LiveCD under VMWare. This is KDE4.0 done right. I'm banking on OpenSUSE 11 having an even better build from the betas I've seen.

By KDE 4.1's arrival, I imagine KDE will be ready for prime time and most users will be able to migrate over.

But all in all, if you want a nice Linux desktop right now, Fedora 9 seems to have hit it pretty well.

My thoughts on software and complexity

2007-06-28T23:23:00-07:00

My thoughts on the growth of the Linux kernel and the status quo of using and developing software..

Prompted by discussion of this article: 1986 Mac Plus Vs. 2007 AMD DualCore. You Won't Believe Who Wins

[Ed: My response to accusations of Linux Kernel bloat]

The [Linux] kernel never has really been the problem. In 1 to 2 MB of compressed/compiled code on my computer (gentoo-sources + my custom .config and a couple of patch sets from future merges), there is some of the most advanced file system, networking, protocol, hardware and scheduling code ever conceived. Indeed, there are many areas that need work and are constantly being updated, but find me a kernel that supports NUMA, scales quite linearly with SMP, implements fair queuing of IO and CPU scheduling, has NO tick interval, virtualization and supports a wide gamut of platforms and hardware. It runs on systems as small as a microcontroller and as large as BlueGene/L. Did I mention it is free and I can learn from and hack on it?

The kernel isn't really expanding at a rate to be concerned with, because only a small subset ends up being needed for most users and systems. No, the problem really lies in user space on UNIX systems. Modern UNIX userland involves many many layers of programs interacting and building on top of each other. I really don't see it getting better in the future either. As higher and higher levels of programing languages are being used, more and more layers are added to the onion. This can make a programmer's life easier and allows more complex systems to be designed, but there are many drawbacks as well. Bug creep, feature creep, usability, complexity, and resource usage all come to mind.

Do I know the answer? Not at all. I don't think there is one. Software will develop organically in the wake of hardware progress for the foreseeable future. If and when this progress slows, perhaps things will change course. A sea change of compiler optimization, small is beuatiful engineering, and an emphasis on efficiency..

Linux: Btrfs, File Data and Metadata Checksums

2007-06-14T13:57:00-07:00

Chris Mason announced an early alpha release of his new Btrfs filesystem, "after the last FS summit, I started working on a new filesystem that maintains checksums of all file data and metadata." The Linux answer to ZFS?

Syncing Directories with Multiple Computers

2007-06-13T01:12:00-07:00

I have a laptop, a workstation, and a server at home that I use daily. I also have a collection of code, documents, and music that are useful to have locally on all three -- especially the laptop when traveling. For a while I would just copy files over the network (NFS and CIFS), or use the server but this has gotten tedious as the amount of data has grown. So I went looking for a syncing app.

I had two requirements:

It must not run as a service - I don't want yet another program loading at boot time or hogging RAM. Transfers should use NFS. Preferable since I am already running it and for speed.
It must be initiated from the client side - I don't want to SSH into computer X to push or pull an update

Unison

Unison has received a lot of hype. It certainly shows a lot of potential. Amongst notable features, it supports bi-directional syncing. You can have changes in the local and remote directories and interactively merge them. It also has a GUI for simplifying this and handling collisions. During my test, performance was terrible and it crashed during the merge. I read somewhere in the FAQ that using Unison with NFS isn't ideal. It prefers to use SSH or a socket, which goes against one of my requirements. In short, this is a program to watch and it may be suitable for smaller file sets -- but it simply did not work well here. The final strike is that it is no longer a research project and only lightly maintained.

rsync

rsync is a very powerful program. Its method of file transfer minimizes I/O and is fast. It is essentially unidirectional, so it lacks the power that Unison has in this regard. For my use case, this won't be a problem since updates happen on one system at a time and are usually pushed to the server. Gentoo users will be all to familiar with rsync when updating portage with 'emerge --sync'.

Using rsync is simple. 'rsync -av SOURCE DESTINATION' will be suitable for most jobs. A quick view of the man page will give a rundown on the options. Using a hub and spoke topology, I can sync all three in any direction: laptop <--> server <--> workstation.

I'm not sure why I waited so long to do this. It is incredibly efficient and easy and I recommend a similar setup if you have two or more computers. It nicely complements a version control system for things that don't need to be tracked over time.

Cheap fetchmail Trick

2007-06-06T21:11:00-07:00

I've had a pretty advanced email system running at home (Postfix, SpamAssassin, Dovecot, Fetchmail) for over a year. The original goal was spam filtering for some 10 year old POP3 mail accounts from my old ISP that were heavily spammed. I couldn't be more pleased with the system; false positives [not spam] are next to zero and very few false negatives [spam] slide through. I might even write a guide for doing a similar setup in the future, but not today.

One of the kinks along the way involved fetchmail. There was an email account that I needed delivered to two separate local user's inboxes. fetchmail and alternative getmail have no way of doing this natively from what I could tell. You simply can't have the same mail account twice. The question on IRC brought about strange solutions such as using procmail or aliases, but there were several disadvantage I could think of, especially involving the per user Bayes learning provided by SpamAssassin.

So I got to thinking, what if I simply fooled fetchmail into thinking that the account was actually two different accounts or servers. And here it is, .fetchmailrc.

poll bowling2-kev
uidl
proto pop3
auth password
via pop.isp.tld
user "bowling@isp.tld"
pass "supersecretpassword"
keep
is kev009

poll bowling2-bowling
proto pop3
auth password
via pop.isp.tld
user "bowling@isp.tld."
pass "supersecretpassword"
is bowling

There are two key things going on here. The first account uses the keep attribute, similar to the "leave mail on server" option in client software. The interesting bit is in the second account. If you look on the line beginning with 'user', there is a trailing period. I am simply taking advantage of the fact that fetchmail thinks this is a different host. In reality the final period is the FQDN (Fully Qualified Domain Name), including the root DNS zone so it is the same host.

I speculate that if entries were added to the /etc/hosts file, this could be scaled to more than two users. Also, uidl and keep across all the accounts could prevent the chance of a mail going to only the final fetchmail entry, if it arrives between fetches. The disadvantage would be overflowing mailboxes if they are not externally pruned, and longer fecthes.

So there you have it, a quick and dirty hack for fetchmailing one remote account to two local users.

I Love IBM

2007-06-03T03:13:00-07:00

Just taking a moment to express my appreciation for IBM. Yes, I'm a bit of an IBM fanboy.

POWER6 is geared for release, and it's badass. Check out this Ars Technica article for a good run down:
IBM's POWER6 flies the coop at 4.7GHz

Simply put, IBM is the only company competing with Intel on Silicon process technology. IBM is part of an alliance with Freescale, Chartered, Samsung, AMD and others so their innovation can and will trickle down to the consumer market.

This is what I'm talking about:
Made in IBM Labs: 10 Chip Breakthroughs in 10 Years

Thank you.

Gentoo 2007.0 Released!

2007-05-07T12:53:00-07:00

"The Gentoo project is pleased to announce the much-delayed release of Gentoo Linux 2007.0. This release met with several delays due to an abnormally high number of security vulnerabilities in large packages which had to be rebuilt using the newer, secure versions of the packages." A new installer, and much needed install media bumps round this one out.

Linux Kernel 2.6.21 and Tickless Kernel (CONFIG_NO_HZ)

2007-04-26T23:35:00-07:00

So Linux kernel 2.6.21 is finally out and all the buzz is about the Tickless Kernel patches. Wanting to drink the kool aid, I downloaded the freshly released gentoo-sources. Unfortunately, it looks like amd64 support has not yet made it in.

I pulled some numbers on my main workstation, a dual Opteron machine. The first command lists the number of interrupts. You'll notice that the 'timer' interrupt is primarily firing on CPU1. Below this is the output of `sensors`. Here you can see CPU1, again which 'timer' is firing, is running about 13°F higher than CPU0. These numbers are pretty consistent when I let the machine idle, so I am going to postulate that the timer interrupt is responsible for the thermal difference. It will be interesting to revisit the issue once amd64 support is added to CONFIG_NO_HZ, and I will post as soon as the results are in.

kev009@kev-ws-amd64 ~ $ cat /proc/interrupts
CPU0 CPU1
0: 969 9294483 IO-APIC-edge timer
1: 10545 28 IO-APIC-edge i8042
6: 0 3 IO-APIC-edge floppy
7: 0 0 IO-APIC-edge parport0
8: 0 0 IO-APIC-edge rtc
9: 0 2 IO-APIC-fasteoi acpi
12: 5 127 IO-APIC-edge i8042
15: 276725 718 IO-APIC-edge ide1
16: 746315 1666 IO-APIC-fasteoi ehci_hcd:usb1, nvidia
17: 59746 5990 IO-APIC-fasteoi libata
18: 282186 1581 IO-APIC-fasteoi ohci_hcd:usb2
19: 0 0 IO-APIC-fasteoi ohci_hcd:usb3
26: 280054 128 IO-APIC-fasteoi EMU10K1
28: 355905 111 IO-APIC-fasteoi eth0
38: 0 15 IO-APIC-fasteoi aic79xx
39: 160124 1762 IO-APIC-fasteoi aic79xx
NMI: 2068 1916
LOC: 9294591 9294562
ERR: 0
kev009@kev-ws-amd64 ~ $ sensors -f
k8temp-pci-00c3
Adapter: PCI adapter
Core0 Temp:
+120 F

k8temp-pci-00cb
Adapter: PCI adapter
Core0 Temp:
+133 F

My x86 server was a bit more evenhanded with its interrupts, so I can't do a thermal comparison like the above. So throwing caution to the wind I compiled a pair of 2.6.21 kernels: one with CONFIG_NO_HZ and CONFIG_HIGH_RES_TIMERS, and the other without. Armed with a kill-a-watt meter, it's time to see if there is any empirical evidence on power usage.

Tickless/High Res: 98-101 ~100VA

250Hz Tick/"Low Res": 99-103 ~102VA

With this hardware (IBM x330, dual 1GHz PIII, ServerWorks OSB4 chipset) I see about a 1-2 volt-ampere average drop.

I'd be interested in seeing others results, particularly on mobile hardware. Many laptops have a current discharge sensor that could be used to monitor and measure the difference. I suspect that the difference would be more noticeable if you run a 1000Hz tick as well. Please leave a comment or trackback with your thoughts and results.

Linux Kernel 2.6.21 Released

2007-04-26T23:27:00-07:00

Linux Kernel 2.6.21 has been announced. Linus writes: "So the big change during 2.6.21 is all the timer changes to support a tickless system (and even with ticks, more varied time sources). Thanks (when it no longer broke for lots of people ;) go to Thomas Gleixner and Ingo Molnar and a cadre of testers and coders."

Building NAS, Part 2

2007-04-15T17:13:00-07:00

Building your own Network Attached Storage server

In an earlier post, I discussed what was necessary to cable 1U servers up with power to run SATA drives. Next, you'll need some drives and a controller if you have not already purchased them.

Controllers

The SATA controller you select is one of the most important pieces of the NAS box. It is my recommendation you get a high quality card, even if you are just doing RAID-1 (Mirroring). There are basically three types of controllers: standard, fakeraid, and hardware RAID. A standard controller is just a host-bus adapter. Any advanced features like striping and mirroring will need to be performed by your OS. Fakeraid includes a RAID BIOS, but all processing is left to a software driver. Hardware RAID is a complete subsystem with a dedicated CPU.

Standard controllers and fakeraid cards are generally cheap and maybe even integrated on your motherboard. However, these are rarely a good choice for servers. Aside from requiring the host to perform a lot more work, these can also cause a lot of trouble down the road. For instance, if a hard disk fails with software RAID, the system may not be able to reboot without intervention. Fakeraid may present the disks to the machine as one logical disk to avoid this, but these cards are the most evil. They tend to use closed drivers and proprietary disk formats which basically means avoid these like the plague. The cards themselves are fine, just use them like a standard controller and use the software RAID of your operating system. That leaves us with hardware RAID controllers.

Hardware RAID controllers are the only logical choice for a server. Hardware RAID cards offload all RAID processing to a subsystem on the card. These cards have an integrated CPU to perform parity calculations for RAID-5, and usually a large amount of RAM to act as cache. The RAM can even be used as write cache, but it is important to have a battery ON THE CARD in case of a power failure or crash to avoid potentially much greater data loss. Hardware RAID controllers present their RAID volumes as logical disks to the computer and operating system, so faults are transparent to the host. They also tend to allow hot addition of storage. Finally, since the card is handling all the I/O, bandwidth requirements are heavily reduced. If I am writing to a hardware RAID-1 device, data only needs to be sent across the PCI bus once as opposed to software RAID which will need to write the data to each disk. This is a very important consideration for older machines that have limited bus bandwidth and large arrays.

All things considered, I went with a 3Ware 9500S. Cost: $100 on eBay.

3Ware has good drivers in the Linux kernel, and has been manufacturing SATA RAID controllers for some time. Other decent manufactures are Adaptec and LSI Logic. Be sure to check for OS compatibility before purchasing a card.

Hard Drives

Just because you are getting the price point of SATA, you should not disregard the quality of drives you are purchasing. Although most standard consumer drives will work fine, there is a much more attractive option. Seagate and others offer what they call "nearline" drives which are basically the consumer drives with a RAID friendly firmware and continuous duty cycle. What's best is that these usually only have a $10 or so premium over their consumer counterparts. When it comes to selecting a manufacturer, take a look at the warranty and technology integrated in the drives. Seagate and Hitachi are both good choices. Take a look at reviews too, especially StorageReview.

When most people purchase a hard disk, they make their choice simply on size. For a server (and even desktop!), you should also consider the spindle speed and size of the drive cache. 7,200RPM is enough for most servers, but 10,000RPM will deliver far greater performance under concurrent and random access. Of course, these usually come at a a heavy price premium and lesser capacity.

Here, I went with a pair of nearline Seagate Barracuda ES drives. The 320GB model was ample for my need, but these go all the way up to 750GB. They use perpendicular magnetic recording which increases density and speed, and feature a large 16MB cache. My initial testing shows nearly 80MB/s throughput! That level of speed has traditionally only been available on expensive SCSI disks. Cost: $100/drive.

Networking

Equally important to a NAS server is ample networking bandwidth. This is largely dependent on the scale of services, but a decent gigabit NIC should get you near disk performance. I picked up an Intel Pro/1000 MT Server Adapter, which features various offloading schemes to free the host's CPUs from networking tasks. If your needs are greater, consider aggregating several gigabit ports together. Cost: $20.

Final Thoughts

The total cost for storage hardware comes to about $320.

Cabling the drives up and configuring the array is a pretty straight forward task. If you chose software RAID, there are plenty of guides on the internet to assist you in setting up md and device-mapper. Try http://linas.org/linux/raid.html.

When it comes to RAID levels, the choice depends on the number of drives you have and the level of protection you need. RAID-1 is a great choice for most applications in that you get true redundancy and greater read performance, but at the cost of half the physical capacity. RAID-5 is also commonly used in which you lose the capacity of one drive, with a three drive minimum. It allows for the failure of a single drive. With the size of modern hard disks, RAID-5 is less attractive than it once was and also suffers from heavy write performance loss due to parity calculations. Try http://www.acnc.com/04_01_00.html for the lowdown on various RAID levels.

My next post will cover file system selection.

IBM NetVista 2800 Hacking

2007-03-29T21:38:00-07:00

Similar to the x330 SATA mod, I have some NetVista 2800s that I need 5v/12v accessory power from. I'm doing some experimentation with the Asterisk open source PBX and wanted to interface with the house phone lines. I purchased a Digium TDM400P for this task.

This is a modular card, with up to 4 lines. Any combination of FXO or FXS modules can be used. FXS modules to interface with analog telephones require 5v/12v power. The NetVista 2800 has a 4 pin power header, labeled "Hardfile Power" at J11.

The connector we need this time is a Molex 43025-0400. Here's the pin out:

I also found an IBM part for the xSeries 300. Note that this is not wired correctly for either the x330 and Netvista 2800. It will require reworking for both application. FRU 24P0622:

And that brings me to my final point. I found a really cool product today that would be useful for both the NetVista and 1U servers.

It takes power from the PCI bus to power a 2.5" notebook hard disk. There are a few different models with different SATA and PATA connection combos depending on your needs. The fact that this product uses a PCI slot is its greatest advantage and disadvantage. PCI slots are usually scarce on machines where this would be useful. None the less, at around $25 it is basically the same price as the IDE adapter you would otherwise need to use a notebook drive in a desktop machine.

IBM xSeries 330 (x330) SATA Retrofit

2007-03-28T21:31:00-07:00

This post documents the process of putting SATA hard disks into older 1U servers. At first, I thought it would be quick and easy but I will share some of the pitfalls I experienced along the way.

I acquired a pair of x330 servers this summer, basically missing only the CPUs and heatsinks. CPUs were not difficult to obtain, but it did take some trial and error to get correctly oriented passive heatsinks. Before putting these into service, I took the opportunity to bring some much needed larger NAS online for the home network by adding some large SATA drives.

After buying a good set of drives and a controller, I started running into some snags. The most obvious was the lack of standard 5v/12v accessory power. The x330 has a 24-pin header that provides power to the SCSI backplane. Using a multi-meter, I was able to locate the necessary contacts. If your server lacks molex power connectors as well, you will need to trace these down. My best advice is to triple check your readings and use an older drive to test the connection before you hook up your new drives. Nothing like all that work, followed by a puff of the magic black smoke flying out of your new drives :). The right eight pins are the ones we want.

To make a nice cable, you will need a Molex 43025-2400 connector and 4 to 8 pins. While I waited for mine to come in, I just pressed a power cable wire by wire into the connections to make contact.

The next potential point of struggle is getting the drives mounted in the trays. The x330 trays have a plastic SCSI SCA pass through on the end. These were held on with a pair of security Torx screws, so you may need to get creative if you do not have the correct bit. The drives fit nicely at this point, but the SATA power connection was blocked by a metal corner of the tray. A quick bit of work with a file fixed this.

Pull the old hotswap backplane and cable up the new drives. At this point you should power on and see how it goes. Hopefully you wired the power cable correctly.

Now your server is capable of accepting inexpensive, high capacity SATA drives! The basic process should be the same for any 1U server lacking power connectors, though you may need to dig deeper to get to the 5v/12v source (power supply is a surefire bet).

"Hello, World!"

2007-03-28T19:12:00-07:00

Well, I've decided to install WordPress and use it to run my personal domain. I've been putting off programming my own CMS for some time, and it really doesn't seem worth the time these days. WordPress recently released version 2.1, which has quite a bit of functionality and seems like a perfect fit for the purpose of this site. Time will tell, but so far I have yet to find any problems.

I will soon be posting a lot of computer related articles, especially some write ups on hardware hacking and mangling I've done. Following that, probably some programming/software/sys admin junk.

So, in lieu of actually writing any code, I'll start with the old programing adage -- "Hello, World!"

Yes, I realize I'm a little late to the blogging party. Better late than never.

Kev009.com

Lucent 7 R/E 5ESS Telephone Switch Rescue

FreeBSD on PowerNV

The Art of Software Development Management

Software Quality and the Table Saw

FreeBSD UEFI Root on ZFS and Windows Dual Boot

Thanks to

Install Windows 10 or other UEFI OS

Boot into a FreeBSD 11+ live environment

Enable sshd, if needed

Bring up a network interface

Partition the drive

Create a 4k aligned zpool

Boot environment compatible ZFS datasets

Perform a manual install of the distribution

Set a few things up

Install refind

Configure refind and add menu entries

Finish, reboot and enjoy!

GELI

Do one thing

Goodbye Wordpress, hello Pelican

Armed to the gills - Dell C6100 build out

Reusable Pagination in Play! 2

Model

Controller

View

View helper

Conclusion

Playing with Play Framework and Plivo

This is your brain on Functional Programming

Nested functions

for comprehension (syntactic sugar around map)

List fun[ctions]

OO Design tradeoffs

Zabbix 1.8.9 Debian Squeeze Backport

Configuration Management Software Sucks

Puppet

Cfengine 3

Bcfg2

Conclusions

GUNNAR Optiks Computer Glasses Review

Intro

Pros

Cons

Conclusion and Verdict

Further Readings

Something good about every language I used in 2010

Bread & Butter

Good Progress

Breaking New Ground

Back Burner

On to 2011

No Nonsense Logging in C (and C++)

Why VIM is not my favorite editor

Stop Social Distributed Version Control Diaspora!

Software Freedom Day 2010 - Gentoo Linux

Sun/Oracle Ultra 27 Workstation Discontinued?

But First, Write No Code

KDE SC 4.4 - Steady, Incremental Improvements

2010 SpamAssasin Public Service Announcement

Sun Ultra 27 Review - The Ultimate Linux Workstation

Video of Windows 7 Black Screen of Death

Windows 7 has definite backlight problems ("Black Screen of Death")

DRBD merged with kernel 2.6.33

Getting Beautiful Fonts in Gentoo Linux

USE Flags

Emerge Fonts

Enabling fontconfig options

~/.fonts.conf

Conclusion

boo2pdf Update

Turn Windows Monitor/LCD Off (Assembly Language Version) -- Announcing LCDoff

Xen 3.4.1 on RHEL/CentOS 5.4

Upgrade procedure for CentOS 5.3 to 5.4:

Updated Xen Install Guide From My Previous Article:

Linus and Windows 7

Announcing boo2pdf

Motivation

Java: The Good Parts