Andy Thompson

Getting chef running on your Raspberry Pi board (Debian Squeeze)

Sat, 23 Jun 2012 13:14:00 +0000

I received my new Raspberry Pi Model B board yesterday, and immediately started messing around with it (as you might expect).

Now I know that I might need to sometimes have to rebuild my installation on new SD cards, so I think it’d be good practice to store this in configuration format so it can be applied again to new builds.

Chef solo is good for doing server-less configuration management, as you can define all your apt package installations and configuration in one place, commit it to github and clone from it again when you need to set up your Raspberry Pi board again.

The steps I’ve taken to do this are:

First I resized my SD-card root partition to fill the entire flash space, and enabled the openssh-server so it can be connected to remotely. I found a good guide explaining these at:

http://elsmorian.com/post/23366148056/basic-raspberry-pi-setup

Next I took an existing knife bootstrap for Ubunutu 10.4. Knife is a tool for managing cookbooks and installing Chef Client and Chef Solo to a remote host. A knife bootstrap is a script that is pushed to the remote host which performs the latter.

I found an issue with this bootstrap, as it turned out the SSH call it was performing wasn’t running login profiles, so wasn’t adding the sbin directories to the PATH environmental variable. It was an easy fix to do, as the “bash -c” at the start of the bootstrap file just needed changing to “bash -l -c”

With the modifications, it was (for me) just:

knife bootstrap -d debian6-gems -x pi --sudo <ip address of R-PI>

This will SSH into the specified IP address as the pi user, and run the bootstrap under sudo for root permissions.

Given that there is additional knife configuration files needed to work out the path that debian6-gems is located in, this could alternatively written with the exact location of the bootstrap file:

knife bootstrap -t /path/to/debian6-gems.erb -x pi --sudo <ip address of R-PI>

I’ve added that knife bootstrap to a github configuration repository, so you can run it yourself by doing the following on your PC (assuming you have ruby and rubygems set up already):

gem install chef # for the knife command
wget https://raw.github.com/webtatic/configuration/master/chef/bootstrap/debian6-gems.erb
knife bootstrap -t debian6-gems.erb -x pi --sudo <ip address of R-PI>

Daemonising a PHP cli script on a posix system

Mon, 24 May 2010 13:38:00 +0000

I’ve been researching how to best write a long running PHP script executed on the command-line, and whilst there are Linux commands you can use to daemonise a command, these can also be written into a php script as well.

The easiest way to daemonise a command on a posix system is to run:

nohup command < /dev/null > /dev/null 2>&1 &

This simply calls nohup to execute the command, which in effect disables the SIGHUP signal (used to tell a child its parent terminal is closing), and backgrounds the process. All standard file descriptors are set to /dev/null to stop the command from reading/outputting anything back to the terminal, and so that when the terminal is closed, reads and writes from stdin/stdout/stderr do not fail.

This is a bit clunky, but the same can be achieved in PHP either by default, or via a command-line argument.

Daemonise

First, to achieve backgrounding, you must fork the process as the current process wont be able to background. Forking will create a new process with the same memory as the parent, which will carry on on the same line the parent forked on. Then detatch it from the terminal using posix_setsid().

switch (pcntl_fork()) {
    case -1:
        die('unable to fork');
    case 0: // this is the child process
        break;
    default: // otherwise this is the parent process
        exit;
}
 
if (posix_setsid() === -1) {
     die('could not setsid');
}

Next, we will replace the duplicated file descriptors (stdin, stdout and stderr) with /dev/null. When a standard file descriptor is closed, it can be replaced with one opened by php. The variables are not important, just the ordering of the fopen calls.

fclose(STDIN);
fclose(STDOUT);
fclose(STDERR);
 
$stdIn = fopen('/dev/null', 'r'); // set fd/0
$stdOut = fopen('/dev/null', 'w'); // set fd/1
$stdErr = fopen('php://stdout', 'w'); // a hack to duplicate fd/1 to 2

You may want to switch stderr to a log file instead e.g. fopen(‘/path/to/errorlog’, ‘a’). PHP errors will output to this file as they happen.

Lastly SIGHUP, along with a few additional terminal signals, can be ignored by the process:

pcntl_signal(SIGTSTP, SIG_IGN);
pcntl_signal(SIGTTOU, SIG_IGN);
pcntl_signal(SIGTTIN, SIG_IGN);
pcntl_signal(SIGHUP, SIG_IGN);

So that gives us the basic daemonising effect. The terminal can now be closed, and the process will continue to run until terminated either internally or externally.

Single Instance

Now with most daemon programs you want to make sure that there can be only one instance, and that instance can be tracked for termination later. To do this we can use a locked file that contains the process id. If the file is locked for write, the daemon is running, otherwise it isn’t. Also the file can be read by other scripts to get the process id, which can be used to send signals to the process.

This you would do before any other code.

$lock = fopen('/path/to/pid', 'c+');
if (!flock($lock, LOCK_EX | LOCK_NB)) {
    die('already running');
}
 
switch ($pid = pcntl_fork()) {
    case -1:
        die('unable to fork');
    case 0: // this is the child process
        break;
    default: // otherwise this is the parent process
        fseek($lock, 0);
        ftruncate($lock, 0);
    	fwrite($lock, $pid);
        fflush($lock);
        exit;
}

The lock will prevent another script from continuing. Once the script terminates, the lock will automatically be released allowing the script to be called again.

The full code

The end result is as follows:

$lock = fopen('/path/to/pid', 'c+');
if (!flock($lock, LOCK_EX | LOCK_NB)) {
    die('already running');
}
 
switch ($pid = pcntl_fork()) {
    case -1:
        die('unable to fork');
    case 0: // this is the child process
        break;
    default: // otherwise this is the parent process
        fseek($lock, 0);
        ftruncate($lock, 0);
    	fwrite($lock, $pid);
        fflush($lock);
        exit;
}
 
if (posix_setsid() === -1) {
     die('could not setsid');
}
 
fclose(STDIN);
fclose(STDOUT);
fclose(STDERR);
 
$stdIn = fopen('/dev/null', 'r'); // set fd/0
$stdOut = fopen('/dev/null', 'w'); // set fd/1
$stdErr = fopen('php://stdout', 'w'); // a hack to duplicate fd/1 to 2
 
pcntl_signal(SIGTSTP, SIG_IGN);
pcntl_signal(SIGTTOU, SIG_IGN);
pcntl_signal(SIGTTIN, SIG_IGN);
pcntl_signal(SIGHUP, SIG_IGN);
 
// do some long running work
sleep(300);

To see the replaced file descriptors have a look at the process’ proc entry.

ls -al /proc/`cat /path/to/pid`/fd
 
total 0
dr-x------ 2 root root  0 May 23 09:41 .
dr-xr-xr-x 5 root root  0 May 23 09:34 ..
lr-x------ 1 root root 64 May 23 09:41 0 -> /dev/null
l-wx------ 1 root root 64 May 23 09:41 1 -> /dev/null
l-wx------ 1 root root 64 May 23 09:41 2 -> /dev/null
lrwx------ 1 root root 64 May 23 09:41 3 -> /path/to/pid

Here is a script that will then kill this daemon:

$lock = fopen('/path/to/pid', 'c+');
if (flock($lock, LOCK_EX | LOCK_NB)) {
	die('process not running');
}
$pid = fgets($lock);
 
posix_kill($pid, SIGTERM);

How about Windows?

None of the versions of Windows support the posix api, however Microsoft have provided a service architecture in all versions of Windows since XP. There is a PECL module win32service, which can help you with dealing with setting up and controlling a PHP service. An example can be seen in the PHP.net documentation.

The Travelling Elephpant challenge, my two solutions

Mon, 17 May 2010 11:47:00 +0000

Ibuildings, a PHP development company with offices in the Netherlands, UK and Italy, have been running a series of challenges, with prizes such as iPads and tickets to the Dutch PHP Conference (DPC, of which they host). The latest completed challenge, the Elephpant challenge, was in the form of a code contest to write the fastest, shortest and least complex algorithm in PHP to solve a Travelling Salesman problem.

I’m happy to say that I was one of the winners for this task, getting 30 points (10 in each scoring method), which was the maximum points possible, for the medium category (people with 2 to 4 years experience), and winning a ticket to the DPC.

Here I will describe the two solutions I researched and completed. I only submitted the one which I thought would give me the biggest chance of winning, but each had their own specialities.

Picture under Creative Commons Attribution Non-commercial license by drewm on Flicker.

First, to understand the problem you must research TSP. There are two types of algorithms, exact and approximate, and a TSP can be asymetric (costs different depending on the direction of travel) or symetric. In the challenge I needed a exact symetric algorithm, of which Wikipedia briefly mentions several different algorithms.

I picked the two most promising algorithms, one based on dynamic programming, and one branch and bound algorithm. The former being shorter and least complex, and the latter being faster and more scalable.

Dynamic Programming

I wont go into too much detail of this one. Although it was the one I picked to submit for the competition it is in fact extremely simple, based on a single premise that a optimal tour can be calculated by finding the minimum tour length of each unvisited node added to the optimal subtour of the remaining nodes. It is recursive in nature, dividing up into smaller subtours until there is a single cost for traveling from one node to another.

Where i is the last landmark and S is the set of landmarks excluding the first and last.

This I managed to write in a total of 33 (PHP_CodeCoverage) / 38 (result) lines of executable code, a C.R.A.P index of 6 (as Ivo mentions in the comments, my result would have been in fact 4 for his calculations), and according to the Ibuildings test a time of 333 seconds. I however recorded the speed at 54 seconds on my laptop (not particularly high spec) with xdebug disabled.

Here is the solution, of which I have fully commented. Please consider this script with having a New BSD license if you wish to use a derivative of it.

Branch and bound

Wikipedia mentions branch and bound algorithms in relation to TSP as being the best algorithm short of cutting-plane algorithms. I ignored the latter due to lack of information found in my research, and the likelihood of it being much too complex for the challenge.

Branch and bound algorithms, it states as being an improvement in speed over other algorithms such as the dynamic programming one above. It can handle a lot more landmarks on top of this, however the challenge only needed to visit a total of 12 landmarks (including the start and end nodes).

When first testing the dynamic programming algorithm, I found it to be much too slow, and wanted to try a branch and bound algorithm to compete with it. This proved challenging and much harder to research and understand. I finally came across an extremely helpful web page describing a mathematical description of an algorithm.

When writing a branch and bound algorithm, it is important to select a good function to calculate the lower bound (something that is lower cost or equal to the best solution). If a tour is found to be the best calculated so far, all branches with higher lower bounds can be discarded.

The lower bound I used was simply based on the premise that the best tour is the sum of the edges of the tour, and therefore the lower bound must be equal to or less than this.

The rest of the algorithm is to choose the branches, add constraints that better select the lower bound, and check through them, picking each edge one at a time, testing whether to include or exclude it from the list of edges that make up the lower bound. Eventually enough edges will be included to get a tour. This I leave for you to read in the article I mentioned.

Now when I had written this, there were several slow areas, such as calculating the lower bound, and checking to eliminate subtours (tours that do not go through every point, but form clusters of smaller tours). As the article didn’t go into detail on how to do this, I had to come up with an inovative fast check. I researched subtour elimination, and came across this article, which although did not give the exact answer in a way I could understand, gave me an idea.

My previous subtour checks were scanning through the list of included nodes, tracing each path to see if it went back to the start or was too short. This was slow (although twice the speed of my dynamic programming algorithm). My idea was to instead track each cluster of joined edges as they get added, and check the constraints on the ones affected by the addition, making sure they didn’t hit the required edge count for each node without completing a full tour.

This proved to be much faster, totalling a time of 20 seconds to calculate the best tour, however the end result was a large amount of code, with many functions, loops and if checks. I’d have been lucky to get 10 points with an attempt at the fastest algorithm. Anyone who scored less than me in each section (speed, lines of code, and complexity) could have scored better. So it was back to optimising the dynamic programming algorithm above for me.

The solution can be seen here, which also I will license under New BSD. Sorry that there are no comments in the code at the moment.

Speed, lines of executable code and complexity

These were the categories of scoring, which award 10 points for each.

Speed varies a lot with algorithms, and the choice could lead to 0 points if choosing a brute force search, or 10 if choosing the best algorithm. I’d recommend anyone who tries to then optimise the code should make use of xdebug’s profiler to see which bits of code are sub-optimal. Trying to optimise the code without this is just a guessing game, and you may end up wasting time changing code that cannot be optimised further.

Lines of executable code I don’t consider the most important aspect of programming, but it does add to complexity in maintaining code, and shorter code can often be easier to understand. In a challenge, however, it can easily be abused at the expense of readability.

Complexity was calculated using a similar method to Change Risk Analysis and Predictions (CRAP) index. This was new to me as I had not come across it before. It is calculated based on the code coverage of a piece of code and the number of code branches in the code (functions, ifs, loops, logical ands and ors etc). This can also be abused to a point. As you can see in my winning code, when printing out a list of the landmarks for the best tour, I chose rather than to write a foreach to loop through and print out the names of the landmarks, I wrote a single line function to splice the landmarks names onto an ordered list of identifiers.

Conclusion

In the end I submitted my first solution, which ended up winning as I had hoped. The effort gone into the second solution I don’t consider wasted even if it was not quicker than the results. It has the capability to run on a much larger set of landmarks with a good time, although not good enough to tour Sweden.

As for the challenge results, you can see that this was a difficult challenge. None of the entries with the 0-2 years work experience fit the requirements, and even in my category many people failed the test.

One thing I must consider is that in a few months I’ll no longer be in the medium category, and challenges such as this will be much more competitive. I have already completed Ibuildings next challenge, and it may be the last before I can only enter into the senior category.

I congratulate the winner for the senior category, Michiel Brandenburg, winning the iPad, who scored 26. 10 for both lines of code and complexity, and 6 for being the 3rd fastest in his category (which was faster than mine as well).

Also, thanks to Ibuildings for running this challenge. I found it both interesting and challenging. Ibuildings have already started a new challenge you can enter into, based on Test Driven Development.

For any of you reading who are going to the Dutch PHP Conference, I hope to see you there :)

Now all I need is to somehow win my own stuffed elephpant to complete my own journey from London to the DPC.

Page-level caching with Nginx

Sat, 10 Apr 2010 22:22:00 +0000

Since my last post on using Nginx to cache proxied content, they have added proper cache handling via their proxy_cache* directives. These are much more suitable for use, as they capture the HTTP response headers and also use more advanced Cache-Control checks.

To start, install the latest stable Nginx avaliable at http://wiki.nginx.org/NginxInstall .

Next edit your nginx.conf and add the proxy_cache_path directive to define a named cache storage. These are independant of servers and locations, and can be reused inside each later on.

...
http {
    ...
 
    proxy_cache_path /var/cache/nginx keys_zone=anonymous:10m;
 
    include vhosts/*.conf
}

Next create the directory for the cache:

mkdir -p /var/cache/nginx

Next define your server configuration, which can be done for example in conf/vhosts/example.com.conf if you defined the include above.

server {
    listen            80;
    servername        example.com;
 
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  Host       $host;
 
    location / {
        proxy_pass    http://localhost:8080/;
        proxy_cache   anonymous;
    }
 
    # don't cache admin folder, send all requests through the proxy
    location /admin {
        proxy_pass    http://localhost:8080/;
    }
 
    # handle static files directly. Set their expiry time to max, so they'll
    # always use the browser cache after first request
    location ~* (css|js|png|jpe?g|gif|ico)$ {
        root          /var/www/${host}/http;
        expires       max;
    }
}

As we don’t want the nginx worker processes to have root permissions when in use, add to the start of conf/nginx.conf:

user nginx
 
...

Then sort out the user and permissions:

useradd nginx
chown nginx:nginx /var/cache/nginx /usr/local/nginx/{fastcgi_temp,logs,proxy_temp}

To start nginx on bootup, add the following to the end of /etc/rc.local:

/usr/local/nginx/sbin/nginx

Then also run this command to start nginx now.

That is all that is needed, no patches this time. There are several more proxy_cache* directives avaliable that you can use to tweak its behaviour, see the proxy module documentation for more details.

OData, a RESTful contender for your API

Wed, 17 Mar 2010 13:37:00 +0000

OData is a new API protocol that has recently been released by Microsoft, along with the launch of their new site on the 16th March 2010. It is a RESTful standard, which adds a lot of its own goods to the table.

As a RESTful standard, it exposes a web service in the form of resources accessible via discrete HTTP urls, representing actions via the HTTP methods. It fills in the gaps that the REST style of architecture leaves open, giving a full specification, from the request to the response (although omitting authentication and authorisation). OData surprisingly also allows RPC-style operations as well.

In essence, OData is as to RESTful service style as SOAP is to RPC. Before this, I had a hard time taking REST seriously, given the lack of protocol specifications meaning that each implementation of a RESTful service tends to handle things differently.

It also includes features that make it much more ready for the enterprise environment, supporting:

a service definition language
error message formatting – missing from the documentation at the moment
dataset paging
advanced queries
batch requests

There are two request and response body formats which are supported, AtomPub and JSON.

Features

Providing a web-service description language, it allows automatic creation of proxy classes for clients to quickly consume with little knowledge of the service. It also allows a level of client-side validation of a request before it is sent out.

If an error occurs, as with the REST architecture style, OData producers will use HTTP error statuses 4xx/5xx to notify of an error, however this information is not usually enough to help the consumer bug fix, so OData defines an error message specification which is sent in the response body.

A frequent problem faced when building a web-service that lists entries is that it becomes a problem when the number of entries gets large. Asside from the fact that not all HTTP servers are built for long streams of data, the consumer does not need all of that information at once, so paging is required to limit the response. OData defines this ability in its specification.

Web services in a REST architecture tend to have limited functionality in an individual request. As such, additional resources are needed to define complex transactions. This means that the web-service needs multiple calls to different resource actions. OData improves this by allowing multiple calls to be batched into one request, removing the HTTP overhead associated.

GData protocol

If you have ever used any of Google’s API’s you may think this sounds very similar to the Google GData protocol, which has similar features. Google released the GData protocol for access to it’s services, such as Calendar, Docs, and YouTube API.

GData is really only pushed as a protocol to be consumed for Google’s services, however OData appears to be an attempt to open the server side of the protocol for other companies to use, providing much more for developers who wish to develop a OData producer, including .Net libraries to help expose data in the protocol.

OData SDK

Microsoft has also provided as part of it’s OData SDK, along with the .Net server libraries, client libraries for most major languages (Javascript, PHP, Java, Objective-C, Silverlight, and obviously .Net), so is ready immediately for several of Microsoft’s services, including Azure.

Aside from the disappointment of no PHP server libraries for this (of which I may try to rectify myself by creating my own), the new web-service protocol appears to be the best thing since SOAP, if not better, and may even replace it as my prefered choice in web-service development.

See more information at:

http://www.odata.org/

Techniques for creating a secure shared web server

Mon, 18 Jan 2010 13:05:00 +0000

Here are several techniques for creating a secure shared web server.

Update 2010-03-14 – Revised opinion about APC and eAccellerator, which possibly do use memory-mapped files, also added detail about mpm_worker not working with mod_apparmor.

PHP open_basedir ini setting

PHP provides this setting in order to limit the access of functions which use PHP’s fopen to a specified directory. PHP extensions may not always use this, instead using the standard fopen, which wont be protected. Shell functions, such as exec, will also be unable to restrict external programs, so should be disabled.

If you are using mod_php, you will need to use php_admin_value rather than php_value, as .htaccess files can override php_value or php.ini settings:

<vhost *:80>
...

php_admin_value open_basedir /path/to/project
</vhost>

suEXEC CGI/suPHP

This technique involves the use of setuid, which allows the super-user to change to any other user. The way suEXEC does this is providing a program with the user sticky bit set. With the right configuration, Apache will call this, passing in a script to execute, and the user and group id for which to run from.

Suexec will check a few conditions against it’s compiled settings, such as the base directory from which scripts are allowed to run from, and directory permissions. If these pass, it will execute the script.

Using suExec in CGI mode, or when using suPHP, you will get the same disadvantages of using CGI normally, and so would not be suitable for a server.

suEXEC FCGI

suEXEC can also initialise a program supporting FastCGI, allowing better utilisation of resources. The FastCGI server can then be connected to from Apache. It does not have the benefit of Apache’s process pooling, and you may not be able to modify the FastCGI pool once running.

For more information check out this blog post on setting it up.

mpm_itk

mpm_itk is an Apache process model based on mpm_prefork, which instead runs its master process as root, giving it the ability to fork and setuid Apache itself. This gives many advantages over the above techniques, allowing use of any Apache modules and process pooling.

It however has a tradeoff. Firstly, the connection is established by a process running as super-user. This, the module’s creator mentions, would be a problem if there was a mod_ssl security hole.

Secondly, since processes using setuid cannot return to super-user, the only realistic possibility is to terminate the fork. From a performance viewpoint of forking and terminating, this isn’t a big deal, however there is no way to persist modified shared memory between requests, as forks copy-on-write. Memory-mapped file support should still work, as well as regular file caching.

To use, install the mpm-itk, and add the configuration to your vhost:

<vhost *:80>
...

AssignUserID myproject-apache myproject-apache
</vhost>

For more information checkout the mpm-itk project page.

mod_apparmor

This is an Apache module which comes with AppArmor. AppArmor is a kernel-level access control module, which allows processes to run under different configurations (called hats). mod_apparmor can change the hat a vhost is running under, giving kernel-level access control to all calls to files to all Apache modules.

This, unlike mpm-itk, isn’t restricted to a request fork and terminate processing model, so should work on more Apache2 mpms.

Mpm_worker, however will have problems with changing hats per process, as multiple requests to different vhosts share the same hat, so mod_apparmor shouldn’t be used. This doesn’t stop you from just using AppArmor by itself to provide a global policy.

<vhost *:80>
...

AAHatName MY_HAT_NAME
</vhost>

See the following link for more information on using mod_apparmor and configuring hats for it: 5.2. Configuring Apache for mod_apparmor

As mentioned, mod_apparmor needs AppArmor to be compiled into the kernel, which may rule out some Linux distributions. Distributions like Ubuntu have AppArmor integrated into their kernel, so if compiling your kernel is not an option, you could switch distribution.

Conclusion

mod_apparmor may be the best option for a secure shared web server, as it would seem the most secure whilst allowing shared cache modules to work correctly, which give languages like PHP a huge boost in performance.

Minimous – Posterous’ minimalist look in WordPress

Sun, 10 Jan 2010 13:30:00 +0000

If you’ve ever searched for a WordPress theme on WordPress.org, you’ve probably found that none of them are suitable for your personal blog. Most tend to be complex designs that you’re sure someone created with a specific look in mind, but they don’t suit you.

A friend from work mentioned something similar almost 6 months ago, so when she tweeted a suggestion to design something like the Posterous look (it’s a very nice simple but popular look), I offered to help, figuring it would take no time at all to create something similar in WordPress.

So I copied the Kubrick theme and stripped out the styles, adding new ones in place, and 15 minutes later I had something beginning to take shape to show. In the end it took me about 2 hours, having never written a WordPress theme before, to get it to what it largely looks like now.

I submitted it to the WordPress.org theme directory, and a couple of weeks later it was approved and many people started downloading it. WordPress.org lists new and updated themes on the theme portal front page, giving them a much needed boost in promotion until they fall off those lists (“Newest Themes” about 1.5 weeks at the time, and “Updated Themes” about a day or two).

Since then, people have tweeted, blogged, and even contacted me on Facebook about the theme. It has now had over 2000 downloads.

One thing that I regret is not having put a link in the footer in the initial release (missing almost 1000 of the first downloads). When I had added that, I could easily see who’s using the theme.

As for what the theme looks like, you’re looking at it right now. Unless of course you are reading this in the future and I’ve changed the theme for some reason, in which case you could preview the theme on WordPress.org. I’ve written a web page describing how to sort out the sidebar over here, so be sure to check that out if you are thinking of using the theme.

Justifying your choice in web service infrastructure

Sat, 09 Jan 2010 13:44:00 +0000

First up, there is no silver bullet in building a web service infrastructure. There are two prevailant types, however, that you should ideally be choosing from:

RPC (remote procedure call) – e.g. SOAP, XML-RPC
REST (Representational state transfer) – e.g. umm? REST?

What I strongly suggest is using one of these, and not designing your own protocol, or proprietry XML straight-up. As for which you should choose…

REST

One thing you’ll notice when developing a REST web service is that there isn’t an exact specification of the protocol and structure, however the suggested way is to use discrete urls as resources, and use HTTP (GET, POST, PUT, DELETE) methods to determine the action to perform.

Looking at the Twitter API, it instead forgoes the discrete resource urls and HTTP methods in favour of using verbs in the url to determine the action to perform on the resource, and just HTTP GET and POST to determine whether it gets or sets data. This hasn’t stopped it from being successful however, and its use of “REST” standards has attributed to the success of many of its third-party sites.

Would that have been true if they had used SOAP instead? Obviously not, as Twitter can also be used in javascript mashups, which SOAP would be a big pain to use.

Since the idea in REST is to use the HTTP specification in designing the implementation, there is only a small amount of error reporting that can be done using HTTP status codes, the HTTP methods may not be enough to represent complex actions, and the input protocol (application/www-form-urlencoded) is only a simple key-value structure (sure PHP improves on it with the use of [] to represent multi-dimensional data, but not all servers and clients are written in PHP, so this should factor into your decision).

However, the HTTP specification includes extra features such as caching that the web service can be coded to benefit from.

RPC

As for RPC web services, I’ll stick to talking about SOAP as its better featured than others. This instead entirely uses HTTP POST on a single url (endpoint), and uses verbs that determine actions and parameters. Yes it can have multiple endpoints, but for simplicity just consider them as other web services.

This is a protocol with a fixed specification, which can be relied upon to be the same. Both the request and response are simliarly structured with a header and body. It even has a specification for error reporting. Once you’ve seen one SOAP service, you’ve seen them all (well not exactly, different vendors sadly sometimes have their own nuances).

Your choice

So I’ve already mentioned that there is no silver bullet. Each may be better suited depending on the situation. But what situtation?

The simplist answer some may jump to conclusions is that SOAP should be used in inter-application communication and REST for third-parties. That may be mostly true.

Some points to consider:

Where a language has support, SOAP is easier to consume, with a standardised interface that can allow a language to make calls via a proxy that acts as if it were a local call.
A language only supplies a HTTP client more often than SOAP (e.g. Javascript), however a REST client which helps with resource locations and detecting and decoding the response is rare.
Any web service written in REST could also be written in SOAP, however the reverse is not as easily done due to the restriction on request complexity of resources and single-dimensional data
REST is only suitable for sending key -> value pairs as input due to the nature of its input protocol (application/www-form-urlencoded), and would need additional resource urls to represent multi-dimensional and batch data, meaning multiple calls to create, insert and commit as a single transaction, when SOAP can do that all in one call.
REST is tied to HTTP over the web, however SOAP can be used in different transports. A bit moot however if you only ever need to create/consume web services.
REST has additional transport specifications that can be used for caching and conditional gets. However SOAP can still code this into a RPC function, or a SOAP header.
SOAP has additional specification extensions such as WSDL and WS-* (too many to list)

Conclusion

In all, I favour SOAP myself, but then all the web services I have coded so far I believe to be necessarily complex enough to justify it over REST. In PHP it is extremely easy to create and consume SOAP using SoapServer/SoapClient, and it automatically throws SOAP faults as Exceptions, which I wouldn’t want to live without.

I can see the attraction of REST due to it’s language support and most services using it support JSON. Zend Framework also makes it easy to reuse your controllers, which should already be in resource structure, for serving REST views.

Recovering a broken Subversion working copy

Wed, 02 Dec 2009 22:28:00 +0000

There are times when a Subversion working copy can mess up. This is usually due to human error, for example due to permissions problems or moving files or folders incorrectly

These can usually be easily recoverable, although at times it can seem there’s no solution. Here are a few examples and their solutions.

Incorrectly deleting files/folders without using svn rm

This will usually result in subversion complaining when you attempt to commit. The symptom of this is the svn status giving something like:

!       path/to/missing/file

To fix this, simply issue the svn rm command for the file, and Subversion will fix and mark it as deleted.

Incorrectly moving files/folders without using svn mv

Sometimes someone may forget to run the subversion move command, and move files/folders manually. If you attempt to add a file after this, you will lose the history of the file. If you attempt with a folder, it will give an error:

svn: warning: 'path/to/moved/folder' is already under version control

The symptom of this is an svn status of:

?       path/to/moved/folder
!       path/to/old/folder

The best solution for this is to simply move it back and run the svn mv command instead.

Adding a folder with permission problems

Sometimes either the group or permissions of a folder will be incorrect. If you add the folder, it will usually issue an error like:

svn: Can't create directory 'new/folder/.svn': Permission denied

Whats worse is that the working copy now registers the folder in a half-added state. The parent folder metadata says the folder is added, but the child folder is missing its .svn metadata. The symptom of this is an svn status of:

~       new/folder

The solution is to first fix the permissions of the folder, then move the folder to a temporary location, and revert the add:

# first fix the permissions
# ...
 
mv new/folder new/folder2
svn revert new/folder
 
mv new/folder2 new/folder
svn add new/folder

An .svn metadata transplant

There are sometimes some cases that can’t be explained. The best solution I’ve found for this is to do a .svn folder lobotomy and restore with fresh .svn folders. To do this:

# backup your project in case you run into trouble
cp -Rp /path/to/project /temporary/location
 
# strip out the old .svn folders
find /path/to/project -name .svn -print0 | xargs -0 rm -rf
 
# check out a clean copy
svn co http://repo/location /temporary/location2
 
# move the .svn folders from the clean copy into the correct relative
# place in the <strong>broken</strong> copy
cd /temporary/location2
find . -name .svn -print0 | xargs -0 -I {} mv '{}' '/path/to/project/{}'
 
# remove the clean copy
rm -rf /temporary/location2

Get an "A" in YSlow with Webtatic Optimizer

Sun, 09 Aug 2009 22:14:00 +0000

The performance of a website is an important issue. Even fast responding dynamic pages can be hit with problems with sub-optimal static content such as high overhead on many HTTP requests and large javascript/css files. Tools like YSlow, and Google Page Speed help identify these problem areas.

Webtatic Optimizer is a tool that can be used to improve these areas, and can help get an almost perfect score.

Make fewer HTTP requests

JavaScript and Css files are naturally loaded individually. Having multiples of them being used across an entire site, e.g. including the JQuery library along with your own javascript, adds to the time the user has to wait for the files to download.

The Optimizer can concatenate these files and store in a new static file, so loading this instead will mean less requests, resulting in an “A” in this category. (images may decrease the score if you use many of them, but I will add in a future release the ability to easily create CSS sprites which rewrite the css file to accomidate the position of an image within its sprite).

Add Expires headers

Whilst you can force an expiry of X hours in apache, this means that changes to these files wont be reflected in the browser until the image has expired. However, browsers cache files based on their entire request url (including query string), so simply concatenating onto the referenced url a unique query string that changes when the contents are modified, is enough to be able to set a year’s expiry whilst changes are reflected immediately.

The Optimizer does this by rewriting css url() urls, calculating the last-modified date of the physical file, and appending this onto the url. There is also a php function helper to do this for static files included in dynamic pages. With this you can get an “A” in this section.

Compress components with gzip

Text-based http requests can be reduced in size significantly using gzip when the browser supports it as a content encoding. The Optimizer does this when creating the concatenated files, also creating a gzipped version at the same time.

This will give an “A” in YSlow.

Minify JavaScript and CSS

If a browser is not able to use the gzipped version of a file, optimized versions of the original JavaScript or CSS can be used instead. This is done at the same time as concatenating the files, using JsMin for JavaScript, and a simple CSS minifier, giving you an “A” for this category in YSlow.

Example

Using a configuration such as below, enables the above optimizations to be incorporated into the generated files:

[config]
version = 1.0.0

[javascript]
config.workingPath = http
defaults.output = site.js
files = "
  + javascript/jquery.min.js
  + javascript/my-javascript.js
"

[css]
config.workingPath = http
defaults.output = site.css
files = "
  + css/layout.css
  + css/global/**.css
"

The above configuration can passed to the optimizer command-line script:

optimizer-compile optimizer.ini

This will concatenate http/javascript/jquery.min.js and http/javascript/my-javascript.js, minimise it, and store it in http/site.js and a gzipped version in http/site.js.gz. It will also concatentate, minify, and rewrite url()’s for http/css/layout.css and any CSS files in http/css/global and sub-directories, and store it in http/site.css and a gzipped version in http/site.css.gz.

Advanced features

There are much more advanced features that can be configured, which you can find out more about in the documentation. The Webtatic Optimizer is a fully extensible library, using plugins and filters to achieve the above. You can enable/disable any features or can even write your own.

An example in use

Using each of the Webtatic Optimizer’s features above, I have managed to get the Fubra Passport, an authentication gateway and payment system for the sites Fubra Limited runs/is affiliated with, a score of “B”. It fails at getting an “A” as it doesn’t have a CDN, and cookies are passed for static content.

I will address the latter, along with making it easy to automatically rewrite local static file urls to a CDN, in a future release.