In mid-November, I made the decision to try out the online dating site OkCupid. This resulted in my first ever modern-North-American-style formal dating adventure. This post is something of a retrospective + lessons learned from my cross-cultural perspective. I have done some external research before, during, and after the adventure, but all of the information is highly anecdotal.

Before I say anything else, I will say that I think every TCK should experience their host culture’s most common dating practise sooner after re-entry. It’s a very eye-opening experience and can give you a whole new perspective on this decently-important aspect of the culture around you. If you, like me, find the structure of a formal dating experience crazy, you’re not very wrong, but you should try it anyway. You just can’t really understand anything you hear about it (probably including the contents of this post) until you try it.

The One Thing

If I have identified anything as the “most important” about the whole adventure, it’s that you have to really try. You can’t dabble, or hold back to protect yourself, or anything like that. You must be emotionally available, and willing to bend your normal system of existence in order to be in the other person’s world. This is basically the same as with any other cultural integration experience.

In my case, this involved things like more emotional availability than I normally afford to most friends, heavy Facebook usage, etc. While it was going on I felt like I was following this principle pretty well, though of course in hindsight there were tons of ways I could have done even better.

Online Dating

As I said, online dating was also a part of my particular adventure. If your biggest obstacle to this particular cultural experience is finding someone to try it with, then this may be a good option for you. I tried the site OkCupid, mostly because of their OkTrends blog (even though that has been defuct for ages, it’s still awesome) and because it’s free, so I could try it with no obligation. This is super useful if you’re not sure you want to try it out. The site does a very good job of being fun to use, and I found the various pseudo-gamified elements highly addicting. They do have some really crazy nonsense that’s part of the system (they come up with things to email you about all the time, just in case you weren’t planning to come back to the site. For example, everyone early on gets a “people on OkCupid want you so bad” email, and at some point I even got a “our system has determined that you’re hot” email, which may be the most hilarious email I’ve ever received), but in general that just adds to the amusement.

On the other hand, from talking to other online dating users, I cannot characterize my experience as normal. I played around with the site for only threeish days before someone awesome popped out of the woodwork with an expression of interest. I followed up, and things just sort of went well from there. I’m told this is the furthest from normal you can get, and I put it down to God’s purpose.

Being Creepy is Ok

The first super shocking thing I learned is that it’s considered totally acceptable to use all of your Google-fu, etc, to find out as much as you like about the other person, so long as you don’t use that information in an overly creepy way.

When making conversation, this can give you a tonne of fodder for what the right questions to ask are, so use it to your advantage.

Clarify Expectations

Something you should do by the end of the first date is make sure you’re both on the same page as to what this is and where it is going. Ask your date what they’re looking for, and on what sort of timeframe they usually operate. First dates are culturally constructed to allow these sorts of awkward questions to be asked of complete strangers with fairly little fear, and I can tell you that not knowing is way more stressful than asking the question will be.

Don’t go into this with the expectation “I’m just trying this out as an experiment”, though. That violates the One Thing.

Facilitate Conversation

You have a wealth of life experiences, but it’s way more fun to learn about someone else than to talk about yourself. Don’t be secretive about your past, but try not to use it as a conversation starter. Use what you know to draw the other person into talking.

Telling Your Friends

I paired this adventure with an experiment in being much more open about my personal life with friends and co-workers. This may enhance your experience, or it may bias your results, or both, YMMV. One thing that I discovered is that your friends bring their own cultural baggage into the conversation. The mere fact that you find your date worth talking about led, in my case, to a whole different sort of conversation (as if about a budding relationship instead of a formal date with a stranger) with many people. As above, if you plan to involve your friends, set expectations. You can never communicate too clearly.

Have Fun

This is not the same as the age-old “be yourself” advice. Be anyone you want to be. This is actually the perfect opportunity to try out minor behaviour modifications on an unbiased judge, if you like that kind of thing. I mean, don’t lie, but you don’t have to be just like you are with your friends.

Make a Move

This one I’m the least sure about, and comes mostly from research. I did not do it at all, and while it may have gone better if I had, I don’t really have enough information to say. The idea is that by the end of even the first date, you should be flirting. Be interested. This seems pretty in line with the One Thing.

The End

It is possible that your dating adventure will turn into a budding relationship with all the 1000 of pleasures and new experiences (learn more at https://poppersbuyersclub.com/). But let’s be realistic, you’re shoving yourself together with a stranger and then seeing what happens. This is most likely going to end before you get that far. However, you cannot operate as though this is the case, because that would violate the One Thing.

If you follow the One Thing at all, this is going to suck a bit. Unless you call it off, but why would you do that? That definitely violates the One Thing. Anyway, it’s not at all like a breakup, it’s not even as bad as every time you moved away from all your friends for the nth time, and being accustomed to people coming and going will certainly make it suck less. That said, you’ve invested some amount of your emotional existence in this, so when it stops you’ll probably have a few days of turmoil.

One thing I learned here, and I have confirmed this from many sources at this point, is that it’s not only acceptable, but totally normal to end a formal dating relationship by simply not replying to messages anymore. So if you’re texing or facebooking you just get dead air. If you’re calling they dodge your calls / send you to voicemail. If this happens to you, I’m not sure what to suggest. You can’t know right away that it’s over. I slowly escalated my attempts to contact the other person over several communications mediums until I did elicit a response. YMMV. Try to use other indicators to see if this is the case. For example, a few days after my date stopped messaging me, she changed her profile picture on OkCupid, which was a pretty dead giveaway that the adventure had come to a close. Use discretion, and if you’re anything like me and simply must know for sure, try to deal without hyperventilating.

Also, it may be your instict to say “no big deal, I never really expected this to end in a relationship” and file the other person as one of your extended contacts, or even move to try and pull them is as friends. That’s not a rediculous thing to want (you’ve found a person that it’s fun to hang out with, why wouldn’t you want to do more of that?), but there’s something about the nature of the adventure that means the other person is very likely not interested in that. Something to keep in mind.

Next Steps

So, that’s it. I had fun. I learned a lot. I met someone new. I have (I think) a better understanding of my host culture. I actually feel better about myself as a person, just because a stranger was interested enough to message me for weeks and hang out with me twice. I know that she dropped into my life at basically the perfect moment.

I wasn’t really intending to go on this adventure, so I don’t have a plan other than “back to business as usual”. Some people have asked me if I would use this mode again. As you can tell, I’ve never really been a fan, but I might consider it. It’s a terrible way to find a mate, but as long as expectations are clear it’s an ok way to spend some time, and you never know. So, not planning on it, but I still suggest you try it once.

This is a guide to building simple webapps using Haskell (modelled after this article on a different framework). We will use:

• WAI (Web Application Interface, and various utilty packages) for the backend
• mustache2hs for templating
• sqlite-simple for database access

Getting set up

There is a very useful utility for building Haskell called Cabal, which will allow you to track which versions of which dependencies you are using, and will tell you if they are not properly installed, etc. Create a project.cabal config file, like so:

name:            project
version:         0.1.0
cabal-version:   >= 1.8
category:        Web
copyright:       © 2013 Your Name
author:          Your Name <youremail@example.com>
maintainer:      Your Name <youremail@example.com>
stability:       experimental
synopsis:        My awesome todo-list app
homepage:        http://github.com/yourName/repoName
build-type:      Simple
description:
        A longer description of my awesome app

executable Main
        main-is: Main.hs

        build-depends:
                base == 4.*,
                http-types,
                wai,
                wai-util,
                wai-dispatch,
                yesod-routes,
                warp,
                text,
                path-pieces

source-repository head
        type:     git
        location: git://github.com/yourName/repoName.git

cabal configure will check that you have everything installed cabal build will build your project.

You may want to set up a Makefile, like so:

GHCFLAGS=-Wall -fno-warn-name-shadowing -XHaskell98 -O2

dist/build/project/Main: project.cabal dist/setup-config Main.hs
    cabal build --ghc-options="$(GHCFLAGS)"

dist/setup-config: project.cabal
    cabal configure

.PHONY: clean

clean:
    find -name '*.o' -o -name '*.hi' | xargs $(RM)
    $(RM) -r dist dist-ghc

Hello World

Save this as Main.hs:

module Main (main) where

import Network.Wai.Handler.Warp (run)
import Network.HTTP.Types (ok200)
import Network.Wai.Util (string)

main = run 3000 (\_ -> string ok200 [] "Hello, World!")

Now make and run it:

cabal build
dist/build/bin/Main

Go to http://localhost:3000 and you should see your Haskell site!

Routing

Our previous app gave the same response to every request. We will use the routeGenerator utility to create fast, compiled routes from a simple syntax:

cabal install route-generator

Add the following rule to your Makefile:

Routes.hs: routes
    routeGenerator -r -m Application $< > $@

You will want a file named routes with your routing information in it.

The router supports any possible HTTP method:

GET / => homePage
POST / => postPost
PURCHASE / => buyTheThing

Where the names on the right-hand side are the names of functions in your Application.hs module.

You can also capture parameters:

GET /post/: => showPost

Here’s an example of an Application.hs with handlers for these routes:

module Application where

import Network.HTTP.Types (ok200, notFound404)
import Network.Wai (Application)
import Network.Wai.Util (string)

homePage :: Application
homePage _ = string ok200 [] "Hello, World!"

postPost _ = string ok200 [] "You posted!"

buyTheThing _ = string ok200 [] "Bought it!"

showPost arg _ = string ok200 [] arg

on404 _ = string notFound404 [] "Not found"

And run the whole thing, with the proper 404, like so:

module Main (main) where

import Network.Wai.Handler.Warp (run)
import Network.Wai.Dispatch (dispatch)
import Application
import Routes

main = run 3000 $ dispatch on404 routes

Headers

Get a header:

import Network.Wai.Util (bytestring)
import Data.String (fromString)
import Data.Maybe (fromMaybe)
import Data.Monoid (mempty)

homePage req = bytestring ok200 [] (fromMaybe mempty $ lookup (fromString "User-Agent") $ requestHeaders req)

Set a header:

import Network.Wai.Util (stringHeaders')
homePage _ = string ok200 (stringHeaders' [("Content-Type", "text/calendar")]) "Not a calendar ;)"

Content types

Respond with the appropriate content type:

import Network.Wai.Util (handleAcceptTypes, string, json)

homePage = handleAcceptTypes [
        ("text/plain", string ok200 [] "You asked for text, here it is.")
        ("application/json", json ok200 [] ["A JSON", "array"])
    ]

Templates

There are many good templating systems. My favourites are blaze-html and mustache2hs, because:

1. They give you some type-checking of your templates at compile time.
2. They are super fast.

To use mustache2hs, first install it:

cabal install mustache2hs

You will need a module to contain the records that you will render out in your template (Records.hs):

module Records where

data HomePageData = HomePageData {
        title :: String,
        username :: Maybe String
    }

And an actual template to render (homePageView.mustache):

<html>
    <head>
        <title>{{title}}</title>
    </head>
    <body>
        <h1>{{title}}</h1>

        {{#username}}
            Welcome, {{username}}!
        {{/username}}
    </body>
</html>

Set up your Makefile to generate the code:

MustacheTemplates.hs: Records.hs homePageView.mustache
    mustache2hs -m Records.hs homePageView.mustache HomePageData > $@

And actually render it out:

import Network.Wai.Util (stringHeaders', textBuilder)
import MustacheTemplates

htmlEscape :: String -> String
htmlEscape = concatMap escChar
    where
    escChar '&' = "&"
    escChar '"' = """
    escChar '<' = "&lt;"
    escChar '>' = "&gt;"
    escChar c   = [c]

homePage _ = textBuilder ok200
    (stringHeaders' [("Content-Type", "text/html; charset=utf-8")])
    (homePageView htmlEscape $ HomePageData "My Title" Nothing)

Logging Requests

If you want to see a log of all requests on standard out, you’ll need to change Main.hs to use a middleware:

import Network.Wai.Middleware.RequestLogger (logStdoutDev)

main = run 3000 $ logStdoutDev $ dispatch on404 routes

Serving Static Content

If you want to serve a directory of static content alongside your app, you can use a fallback mechanisms from the wai-app-static package:

import Network.Wai.Application.Static (staticApp, defaultWebAppSettings)
import Filesystem (getWorkingDirectory)

staticRoot = staticApp . defaultWebAppSettings

main = do
    cwd <- getWorkingDirectory
    run 3000 $ dispatch (staticRoot cwd) routes

Or alternately use the middleware from wai-middleware-static:

import Network.Wai.Middleware.Static (static)

main = run 3000 $ static $ dispatch on404 routes

Sessions

Some apps need a way to store data between requests using cookies. wai-session is a package that provides a generic way of doing this, and has existing backends for in-memory storage, encrypted cookies, and tokyocabinet. The wai-session-clientsession package contains the backend for encrypted cookies:

module Main where

import Data.Default (def)
import Data.Maybe (fromMaybe)
import Data.String (fromString)
import qualified Data.Vault as Vault

import Network.Wai
import Network.Wai.Util (string)
import Network.Wai.Handler.Warp (run)
import Network.HTTP.Types (ok200)
import Control.Monad.Trans.Resource (ResourceT)

import Web.ClientSession (getDefaultKey)
import Network.Wai.Session (withSession, Session)
import Network.Wai.Session.ClientSession (clientsessionStore)

app session env = do
    u <- sessionLookup "u"
    sessionInsert "u" (show $ pathInfo env)
    string ok200 [] $ fromMaybe "Nothing" u
    where
    Just (sessionLookup, sessionInsert) = Vault.lookup session (vault env)

main = do
    session <- Vault.newKey
    store <- fmap clientsessionStore getDefaultKey
    run 3000 $ withSession store (fromString "SESSION") def session $ app session

Databases

For database access, use postgresql-simple or sqlite-simple:

import Database.SQLite.Simple (open, close, query, Only(..))
import Database.SQLite.Simple.FromRow (FromRow(..))

data Post = Post {
        postTitle :: String,
        postBody :: String
    }

instance FromRow Post where
    fromRow = Post <$> field <*> field

showPost :: Int -> Application
showPost postId _ = do
    conn <- open "./production.sqlite3"
    [post] <- query conn "SELECT * FROM posts WHERE post_id = ?" (Only postId)
    string ok200 [] (postTitle post)
    close conn

Of course, you shouldn’t probably re-connect on every request. Change your Makefile to have the router pass an argument through:

Routes.hs: routes
    routeGenerator -r -n 1 -m Application $< > $@

And do the connection from Main.hs:

import Database.SQLite.Simple (open, close)

main = do
    conn <- open "./production.sqlite3"
    run 3000 $ dispatch on404 (routes conn)
    close conn

And then you can use the connection:

showPost :: Connection -> Int -> Application
showPost conn postId _ = do
    [post] <- query conn "SELECT * FROM posts WHERE post_id = ?" (Only postId)
    string ok200 [] (postTitle post)

Deploying to Heroku

Deploying to Heroku is easy with the heroku buildpack.

First, our hello world app needs to change slightly. Heroku tells us what port to run on with the PORT env variable:

module Main (main) where

import System.Environment (getEnv)
import Network.Wai.Handler.Warp (run)
import Network.HTTP.Types (ok200)
import Network.Wai.Util (string)

main = do
    port <- fmap read $ getEnv "PORT"
    run port $ string ok200 [] "Hello, World!"

Then add a Procfile in your root dir to tell Heroku how to start your app:

web: ./dist/build/project/Main

And add a Setup.hs to build your app:

import Distribution.Simple
main = defaultMain

Then, assuming your project is a git repo:

heroku create --stack=cedar --buildpack https://github.com/pufuwozu/heroku-buildpack-haskell.git
git push heroku master

This year, for the first time, I attended the security convention and hacker party time that is DEFCON.

I had heard a lot of things about DEFCON, but it far exceeded my expectations. The convention is so full of things to do that most of the time I was not even aware of them all. Atendees are, mostly, intelligent, approachable, and respectful.

On Thursday, within 10 minutes of setting up an account for the secure wireless (by using the inscure wireless), my Blackberry Z10 went nuts. I took a device that didn’t have any data on it on purpose so that I could be reckless like this. I was impressed by the speed and violence of the attack. I could not turn my phone off properly (it just kept vibrating and making noises) and had to remove the battery. Booting in airplane mode was fine, but the moment I connected to the WiFi again the attack continued. Creating an actually secure account using a friend’s 3G allowed me to actually use my phone for the rest of the convention. Overall, it was an awesome experience.

I saw talks about quadcopters, Dust, ToR, SIM card exploits, physical lock exploits (as well as normal lockpick techniques). I attended workshops on evidence tampering and handcuff shimming. I spent some time trying to decode some of the crypto puzzles that were an integral part of the conference. I met some people.

The reputation DEFCON has for being full of misogyny and therefore having few female attendees is completely overblown. I was actually suprised at the (seemingly) huge number of female attendees, compared to many other tech conferences I have participated in. Hacker Jepordy has a stripping component, and that was the only anti-feminism I saw at the convention. I saw a lot more, but that was Vegas casinos, and not inside the convention space.

The weather was amazing. Hot and dry. The air conditioning way overcompensates, though (my hotel room was set to 19 when I arrived), so take a warm shirt.

Overall I enjoyed myself, and I plan to go back again next year.

Most people in the west are familiar with the idea of insurance. Most of us pay at least one insurance policy. What is it that we really hope to be accomplishing by this?

Insurance works by having customers pay fees in return for protection against some money-related disaster. When a specified disaster (such as a car crash) happens, the insurance company then pays (some of) the cost. The idea is that you would not be able to afford this unexpected disaster, and the insurance company takes that burden off of you. If nothing bad ever happens, then you just keep paying.

That is, insurance is a bet against yourself. The only way the insurance ever “pays off” is when you experience a disaster. Car insurance is a bet that you will crash. Life insurance is a bet that you will die sooner rather than later. The insurance company itself is actually betting on you. As long as nothing bad happens to their customers (on average), they make money.

So, if you have bad things happen to you all the time, insurance is a good deal. Otherwise, you will end up paying more than it would cost to recover from the disaster, just for the “protection”, in which case a savings account would be a better investment.

Why do we buy insurance, then? Why not just save? Well, the biggest reason is that if some disaster occurs before the savings account has filled up, insurance is a better deal. There is a fear that, if disaster strikes, we might end up in debt (or worse). We want protection. There has to be a better way to get it.

Let’s say I have a barn, and it burns down. I don’t have insurance, and I have not yet saved enough money to build a new barn. I cannot live without a barn (my livelihood depends on it), so I cannot just alter my lifestyle to a barnless lifestyle. A very traditional solution to this problem was/is the barn raising. A barn raising is when the entire community comes together, pools their resources, and rebuilds my barn. Why do they do this? Well, because if their barn burns down I’ll do the same for them. Essentially we are treating our pooled savings (and other resources, primarily manpower) as an on-the-fly insurance pool. This is a bet on the community, because if nothing bad happens, everyone keeps their money.

Barn raisings are great, but they have a problem: what if community members do not save well? What if they want to help, but the money is simply not around? This is where we get another strategy from: pooled community savings. A group of people open a fund together, and pay dues just like they would to an insurance company. Payments are made out of the fund (based on community vote, or similar) when this is a disaster. If the fund has enough money in it, then dues can be lessened or stopped. Members may even be allowed to transfer “their portion” to another fund if there is a good reason. PeerCover and others are doing pretty innovative work in this area.

What strategy to pick depends quite a bit on the situation, and there are a few places where insurance cannot be avoided (car liability insurance is legally mandated in many places), but there are a number of options available. Let’s stop betting against ourselves out of fear.

This is just a breif interlude to document something that it took me a very long time to find out. Some ARM cross-compiling toolchains default to thumb instructions. Thumb instructions are this nifty feature of ARM that allow most common instructions to be represented in 16 bits instead of 32 bits, which can save RAM and improve instruction fetching. Many of the instructions needed in the context-switch part of the kernel, however, must be in full 32-bit ARM. No problem, there are instructions for switching the CPU mode, just make it switch whenever moving from one to the other, right?

One of the downsides of writing C over raw assembly is that you don’t have full control over what instructions are output. I needed a way to make the toolchain realize that mode-switching instructions needed to be inserted. I could not (at the time) find such a way.

So, I used a different solution: tell the C compiler to always output 32-bit ARM, no matter what the default is. This worked, but presented another problem: the default and standard libraries for my gcc were also in thumb, and the interwork was not getting set up there either! So I could no longer use anything from libgcc or libc, which meant that not only could I not use their memcpy, etc (not neccessarily bad, since in an OS one might want to write one’s own of much of libc anyway), but I couldn’t use things like the % operator (which on ARM is implemented using a helper procedure in libgcc).

A helpful StackOverflow user finally found the answer for me recently, and I have confirmed that by marking procedure entry points as functions I can stop being concerned about -marm, which allows me to Use -lgcc and % and even use parts of -lc.

It’s no secret that I’m not impressed with the current state of University education. I’ve spent a long time griping about the various issues, and pointing out ways I would do it differently. None of that makes any difference, however, until I actually do something about it.

While my issues are with post-secondary education generally, my primary expertise (especially of late) is in Computer Science and Software development. So, I developed a curriculum, found a community, and started teaching a class.

Teaching one class does not constitute a solution, but it does provide a learning experience for me. The class is over half done now, and in the rest of this post I will be examing things I have tried and what I have learned.

Hands-on Learning

Since my approach is focused entirely on education, I did not assign any exercizes early on. I provided students with the tools needed to play with the concepts on their own, and set them free to hack. This turned out to be too open-ended. When encountering a new concept for the first time, many students simply had no concept of what direction they should take their explorations in.

I am now providing some concrete exercizes to the students, but really I need to find a way to integrate more hands-on learning into the class itself. This is made more difficult because of the pacing I chose for this version.

Pacing

Probably the biggest experiment in this first version of the course is the pacing I am using. The class is an overview of Computer Science from both the perspective of functional abstractions and low-level machine implementation. It is designed to give students a flavour of what parts of Computer Science they might find interesting for further study. I am doing the whole thing in 8 weeks.

The amount of material I am covering would normally be covered in two or more semesters at a University. Why am I doing this? For two reasons: so that students have less time to get bogged down on individual details (since, as an overview course, this is not about depth), and also to find out how fast one can reasonably progress without hopelessly confusing students.

While I cannot be sure without more experiments, I am also beginning to suspect that the pacing increases student engagement (at least for the sorts of students I have solicited). New material every single class means that students do not have an opportunity to tune out because “we already talked about this”.

While the pacing is definitely hurting the students’ ability to deeply absorb the subject matter, I conduct informal experiments periodically to determine understanding. Students in general seem to be grasping concepts, and find themselves coming back up to speed on items they failed to retain quickly enough to demonstrate a level of penetration.

Students

For this course, primarily because of the pacing, I solicited students with prior knowledge of computer internals and programming. I started out with a good mix of students from various backgrounds, but certain students (about half of the 10 I started with) dropped the course fairly early on.

Students who stuck with the course were precisely those students who had both enough knowledge of computer internals to handle the pace, and enough of a deficiency in prior experience with Computer Science-related material to be interested in an overview. I hope to run both slower and more in-depth courses in the future in order to serve other sorts of students.

While prior knowledge of the basics seems to be helping the students’ ability to comprehend new material, it also occassionally poses a distraction, since I cannot present any idea as strictly new. I need to improve my ability to communicate an idea that is “new” in the context of a course, without speaking as though no one present has ever heard of it before.

From the Top and Bottom

The concept of my particular curriculum for this course is to start at the “most” abstract and “most” machine-specific concepts, and work inward. This is in stark contrast to most first-year Computer Science courses, which start out in practical programming, move on to machine specifics, and then later on do algorithms and (maybe) more abstract (actual) Computer Science.

Students have indeed struggled with the seemingly-abstract concepts, especially early on when they may not yet even have a basis on which to understand “why they care” about the abstractions that are possible. This is partly because of my failure to spur adequate hands-on learning, and partly because of the ordering. On the other hand, the juxtaposition of abstractions and related implementation details has already more than once resulted in realizations about the nature of the abstractions (“that product type is just like a struct!”)

Next Steps

I still have to finish teaching this class, and will report more at that time. Some of my students (and also other members of the Kwartzlab community) have expressed interest in an Operating Systems implementation course, based on my Writing a Simple OS Kernel series of posts. I hope to run this course mid-summer, and structure it purely as a project course. This should give me more experience with the ways that hands-on learning can be effectively brought directly into a class.

I am also excited about the idea of running other more in-depth courses, based on community interest. From a “Computer Science for Programmers” side I would like to run a course I’m calling “Advance Abstractions” (dealing with advanced concepts in control flow and data modelling) and also a non-course structured as various talks by various contributors on specific data structures and algorithms.

I would also like to run the “slow version” of this same course, targetted at complete beginners. The problem with this version is that it requires participants to have more time to dedicate to the course. We’ll see where that goes.

piratepartyca: Two more arrested for spoofing MusicBox data: prtpty.ca/66b

jjdavis: RT @doctorow How long will the Canadian gov’t hold people without trial?

m0rty: This is my jam: t.co/462ab

acklas: @doctorow at least a year, remember? http://freebyron.org

wilkiz: @lucy2 They may charge them with computer crime, but that can’t hold

acklas: @wilkiz @lucy2 Crazier prosecutions have done. enwp.org/Aaron_Swartz

Our protagonist is at his company office. So are officers of the RCMP. They’re interested in his work on the so-called “MusicBox antivirus”. Nicnus is not affiliated with that project in any way, but you may recall that he wrote the code which originally made the blocker possible.

They’d rather he come with them. Does he have to come? Is he being charged with something? No. He’d rather stay here. Is that a problem? They confer. It doesn’t have to be. Good.

They want to know if he’s aware of the recent rash of activity. What activity? People spoofing MusicBox data, targeting the RCMP data collection. Oh, that. Of course he knows about it. Everyone does.

Is he also aware that his software may be used to perform this exploit? What exploit? The one where people spoof data! That’s not an exploit. Is sending a letter with a false return address an “exploit”? Yes, he is aware that his software makes it possible to create such a letter and send it to the RCMP.

Why would he create such software? Well, he wasn’t the first to do so, if they only care about the spoofing. They are not aware of the prior art. He can show them the code and the posts. It is not a secret. They would like that.

16:00 <nicnus> cops are here

Does he know why the earlier code was written? To demonstrate the unreliability of the data. Unreliability? Surely they understand that they cannot trust anything they get from a data stream that can be spoofed.

They take anonymous tips by phone. This is at least as good as that.

16:02 <acklas> Woah! Are you ok?

16:05 <nicnus> calm down. If I’m helpful, they’re happy 😉

They’ve never needed to track down pranksters leaving fake anonymous tips, because it’s hard to do en-masse. It can be done en-masse with VoIP? Well, no one has at the least. What would they do if that changed? They probably can’t apply computer crime laws to telephone calls. Probably. It’s still impeding the work of the police. Might even be fraud. There are laws in place.

16:20 <nicnus> They’re gone.

* Your connection has been secured

16:21 <acklas> Oops. Should have turned that on sooner.

16:21 <nicnus> meh. Those anonymous kiddies following your lead are in trouble, though

16:22 <acklas> Oh?

16:22 <nicnus> Yeah. At the very least, it’s some form of fraud

* acklas shrugs

16:23 <acklas> Their fault for not being more careful

Nicnus is waiting for a bus. He only went in to the office to meet with the officers. They came to his workplace and expected him to be there. So old-school.

There’s a chilly breeze blowing. Nicnus slips his hands up inside the sleeves of his hoodie. When his phone vibrates, he waits a few minutes before checking.

m0rty: Cops were just here. Something about my computer sending them messages?

Oh, man.

@m0rty your computer is probably infected with a botnet virus.

The bus is here. He gets on, but there are no seats available. There is barely a place for him to stand. And some people think no one takes the bus around here! He keeps one hand on a bar for stability, the other on his phone.

m0rty: @nicnus A virus? I have Norton and AVG both.

Nicnus stares at the screen for a moment. The bus lurches, throwing him against some blonde in her twenties. He plants his feet and switches to holding one of the provided handgrips on the bus.

@m0rty having two probably not helping. You can get it anyway. Then your system does what it is told.

People are moving. He slides his phone closed and looks around. Someone is trying to get off the bus. They should have made their way to the door long before now. The door is sitting open, and they are pressing through the crowd. There, they’ve made it. People flow towards the back as someone boards. Now he is wedged against some hefty guy who has a seat. Good thing he didn’t bring his backpack today. He manages to slide open his phone again.

jjdavis: @m0rty @nicnus is right. Running two antivirus not a good idea.

acklas: @m0rty You should probably just always assume your Microsoft systems are infected.

m0rty: @nicnus What should I do to get rid of it?

Nicnus cannot write a reply. No way to manipulate the phone properly just now.

jjdavis: @m0rty Only sure way is to wipe and restore from backup

acklas @m0rty might not be worth it. You’ll just get infected again

jjdavis: @acklas that’s really optimistic of you

The bus lurches again. He is face-to-face with the same blonde. Suddenly aware of his surroundings instead of his phone, our protagonist realises that this is probably awkward. He turns his head. No longer face-to-face, but now he’s staring at her shoulder. That’s probably not better. He looks over her shoulder and out the window. It’ll have to do.

m0rty: @jjdavis @nicnus I don’t really keep backups. Don’t want to lose my family videos! :S

16:31 <acklas> Who is this m0rty guy anyway? Where is he?

16:31 <jjdavis> Some guy nicnus knows. I don’t this he’s near any of us, though.

Now everyone is exiting the bus. Everyone? Oh, no, just most everyone. It’s a major stop, and this is where most of the people were going. Nicnus has two more stops to go, so he finally gets a seat.

16:33 <nicnus> friend from highschool. He lives out west.

@m0rty I wouldn’t worry to much about it. Let me know if you have more trouble

His stop is next. He pulls the signal cord and makes his way to the door. Phone goes into his pocket. Bus is stopped. Door is not opening. Oh, red light. Driver is waiting to pull ahead. Whatever. Open now. Getting off.

Our protagonist trudges towards his home. His mind is on the motion of his feet. Should he take longer strides? Maybe he should try to avoid the cracks in the sidewalk. Careful, there’s something on the ground. Probably don’t want to know what it is. Now he notices the people walking ahead of him. Two abreast and blocking the sidewalk completely. Not walking very quickly. He plods behind them for a few seconds before pulling over onto the grass and walking past.

Home. His phone is still in his pocket, but he doesn’t really need it. His desktop is already on, since he hadn’t expected to be gone long.

* Your connection has been secured

16:45 <nicnus> acklas: have the cops been to see that lady on your street again?

16:45 <acklas> nicnus: yeah. different ones. They were nicer this time. Figured out pretty quick she was not the actual source of the traffic.

16:46 <jjdavis> So someone is being more careful. It’s not all kiddies

16:46 <nicnus> It’s just kiddies getting caught

16:50 <acklas> the kiddies always get caught

16:51 <nicnus> the cops I talked to likened the MusicBox data to anonymous tips. So they don’t expect it to be admissible in court, I guess

16:53 <jjdavis> I guess that answers your brother’s concerns about evidence fabrication

16:55 <nicnus> yeah, maybe

Hungry. Food. First, music. What music? Something energetic. He opens an encrypted tunnel to a computer in the United States so that he can access a music service that only works for “people in the US”. He asks it to play energetic music, based on his past preferences, and it queues up what looks like a stream of evenly-mixed Dubstep and Power Metal.

Food.

His fridge is empty. Typical. Bread. He has bread. Good. Toast. What to put on the toast? Does he need to put anything on the toast? How hungry is he? Toast first, then re-evaluate.

What is he working on today? Oh, right, the discovery process for the encryption box. Some customers have reported that sometimes the boxes cannot find each other, even if they are plugged directly in to the same hardware. That doesn’t make much sense, but he is going to try to reproduce it.

One hour passes. Then two. Nicnus remembers he put toast in the toaster. It’s cold now. Whatever, eat it anyway. He isn’t really closer to finding the problem. Well, he’s a little closer: he’s pretty sure there *is* a problem. He just is not just quite what it is, or how to make the problem happen. It happened once, but then worked when he tried again.

00:10 <jjdavis> Taking the g/f out. Will not be very responsive for a bit

* jjdavis is away: out with the g/f

Nicnus is pulled from what he is working on. 00:10 UTC? What is that in local time? He looks out his window. Early evening, looks like. Where did his day go? He looks at the code that he has been working on. It might be the cause of the problem, but it is hard for him to check since he can’t surface the problem reliably. How will he know if changing this has fixed it?

03:52 <jjdavis> That went pretty well. Maybe I’m getting the hang of this

03:55 <acklas> or maybe she’s just being nice 😉

What time is it? Probably should think about dinner. What has he eaten today? Orange juice for breakfast, and toast for a late lunch. Now that he’s thinking about it, he is very hungry. Why wasn’t he hungry a minute ago? What was he working on? He can reproduce the problem now, but it is not what he thought it was. He has no idea how to fix it, but at least he can test it. Food. He should eat food. Does he have any food? He has some frozen yogourt in his freezer, and some peanuts. Dinner!

Someone sends him a link to a funny video. He watches it twice while he eats. It’s pretty good. The end of the video contains an obscure reference to some product from Suzuki. Now he’s reading Wikipedia articles about the Dutch East India Company.

06:01 <acklas> bed time

* acklas is offline

The issue with the encryption box might be in the hardware, and not in the software at all. How would he test that? Oh! They have an all-software emulator for it, which they built so they could test the software back before the hardware existed. It’s a bit out of date, since they don’t use it anymore. After some testing, he finds that it has the same problem. The problem is definitely in the software.

His music stopped at some point. When did it stop? Seems that his encrypted tunnel to the US has stopped working. He starts up another one.

09:00 <nicnus> Woo! I think I’ve got this bug nailed. Time for some sleep

Let’s say you have some code and you want it to be able to iteract in a very similar way with either a network connection or a named pipe. You might structure this using a sum type (also called a tagged union):

data Stream = Network NHandle | Pipe PHandle

def interaction(h)
	input = case h
		when (Network handle)
			readNetwork(handle)
		when (Pipe handle)
			readPipe(handle)
	end

	output = transform_input(input)

	case h
		when (Network handle)
			writeNetwork(handle, output)
		when (Pipe handle)
			writePipe(handle, output)
	end
end

Since “dynamic typing” is just the practise of giving every value the exact same type, which is a giant sum type, a dynamically typed construction would look like this:

def interactionDynamic(h)
	input = case h
		when is_a?Network
			readNetwork(h)
		when is_a?Pipe
			readPipe(h)
	end

	output = transform_input(input)

	case h
		when is_a?Network
			writeNetwork(handle, output)
		when is_a?Pipe
			writePipe(handle, output)
	end
end

This is basically the same, but it introduces a tradeoff. In the original, if we wanted to add another supported stream, we needed to add a case to the sum type *and* cases to the function, and in return we were guarenteed that we would only ever be passed cases we could handle. With the dynamically typed version, we can just add cases to the function, but any value can be passed to us and if we can’t handle it the code will just crash.

Neither of these solutions are very nice in this case. We really want abstracted operations:

def read(h)
	case h
		when is_a?Network
			readNetwork(h)
		when is_a?Pipe
			readPipe(h)
	end
end

def write(h,output)
	case h
		when is_a?Network
			writeNetwork(h, output)
		when is_a?Pipe
			writePipe(h, output)
	end
end

def interactionDynamicAbstract(h)
	write(h, transform_input(read(h)))
end

Much nicer! One issue here, though, is that these operations really are fundamental to Networks and Pipes, and we have defined them as part of our code. It is likely that other people using Networks and Pipes will end up writing similar code. What if we bundled up all the operations you could do with a Pipe right alongside the data?

def interactionDuckTyped(h)
	h.write(transform_input(h.read()))
end

Now the operations live inside the datatype. This has the added advantage that we don’t even need to write any code to support a new type of stream, as long as the stream comes packaged with a read and a write operation.

There are two issues here, though. One is very similar to the tradeoff we made for dynamic typing, which is that the compiler cannot infer from what we’ve written which operations are necessary and so cannot check that values being passed to our procedure actually implement these operations until runtime. The other problem is that the only thing we know about the operations for sure is their names. We cannot easily document that these operations should have certain properties, because the users of our function cannot easily what operations we are using.

Can we fix these issues while keeping the extensability and power? We could try bundling the operations up seperately:

def interactionRecord(h, ops)
	ops[:write](h, transform_input(ops[:read](h)))
end

Now we can know that these operations will be present, and we can document that the operations the user passes in must satisfy certain properties. This seems like a pretty good solution. We have, however, leaked the management of which operations to use back to our user. Can we keep this safety while regaining the conveniance of having the stream type author specify the operations instead of our user?

This is exactly what typeclasses do. Consider:

typeclass Stream a where {
	void write(handle, bytestring);
	bytestring read(handle);
}

hastypeclass Stream Network where {
	write = ...;
	read = ...;
}

def interactionTypeclass(h)
	Stream::write(h, transform_input(Stream::read(h)))
end

Now we have a natural place to document the properties operations must have (the typeclass definiton), and their relationships with each other. The compiler can look up the correct hastypeclass to figure out what code to run for each type, and while it does so it can check that the operations we need (the ones that are part of Stream) are in fact defined for the input type.

Safety, extensability, and ease of use, all in one solution.

Last time we got IPC working, and used it to build a simple path-resolution module. This time we’re going to write a fully-functional serial port driver.

Generic Interrupts

When we got interrupts working, we sort of hacked the kernel part in to support the timer, which is all we were interested in at the time. Let’s make that a bit more generic. We’re going to need one more syscall, since interrupts have to be handled by the kernel. The simplest one we can implement just allows a process to wait for an interrupt to fire on a particular line:

void interrupt_wait(int intr);

.global interrupt_wait
interrupt_wait:
	push {r7}
	mov r7, #0x5
	svc 0
	bx lr

#define TASK_WAIT_INTR 3

case 0x5: /* interrupt_wait */
	/* Enable interrupt */
	*(PIC + VIC_INTENABLE) = tasks[current_task][2+0];
	/* Block task waiting for interrupt to happen */
	tasks[current_task][-1] = TASK_WAIT_INTR;
	break;

We are already enabling the timer interrupt when the kernel starts. Here, we enable the interrupt that the process is asking to wait on, since if it is not enabled the task will block forever, so this seems like a useful place to do it.

We also want to unblock tasks when the interrupt fires. Get rid of the -4 case that we have hardcoded for the timer and put this in:

default: /* Catch all interrupts */
	if((int)tasks[current_task][2+7] < 0) {
		unsigned int intr = (1 < < -tasks[current_task][2+7]);

		if(intr == PIC_TIMER01) {
			/* Never disable timer. We need it for pre-emption */
			if(*(TIMER0 + TIMER_MIS)) { /* Timer0 went off */
				*(TIMER0 + TIMER_INTCLR) = 1; /* Clear interrupt */
			}
		} else {
			/* Disable interrupt, interrupt_wait re-enables */
			*(PIC + VIC_INTENCLEAR) = intr;
		}
		/* Unblock any waiting tasks
			XXX: nondeterministic unblock order
		*/
		for(i = 0; i < task_count; i++) {
			if(tasks[i][-1] == TASK_WAIT_INTR && tasks[i][2+0] == intr) {
				tasks[i][-1] = TASK_READY;
			}
		}
	}

You’ll notice we also need a new magic number for versatilepb.h:

#define VIC_INTENCLEAR 0x5 /* 0x14 bytes */

Code for this section on Github.

Output Driver

Let’s use this new ability to implement half of our driver: output. This will allow us to write to a specific fifo for output to the serial port instead of using bwputs.

A refresher of serial port related magic numbers:

/* http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0224i/Bbabegge.html */
#define UART0 ((volatile unsigned int*)0x101f1000)
/* http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0183g/I18381.html */
#define UARTFR 0x06 /* 0x18 bytes */
#define UARTIMSC 0x0E /* 0x38 bytes */
#define UARTFR_TXFF 0x20
#define UARTIMSC_TXIM 0x20

#define PIC_UART0 0x1000


Let’s make a generic driver that accepts parameters telling it what serial port to drive:

void serialout(volatile unsigned int* uart, unsigned int intr) {
	int fd;
	char c;

We need to create the fifo that processes send bytes on:

	mkfifo("/dev/tty0/out", 0);
	fd = open("/dev/tty0/out", 0);

We also need to tell the UART to trigger an interrupt when it becomes ready to send a byte:

	*(uart + UARTIMSC) |= UARTIMSC_TXIM;

And finally a loop to read bytes and send them to the UART:

	while(1) {
		read(fd, &c, 1);
		interrupt_wait(intr);
		*uart = c;
	}

This seems pretty reasonable, but there is a problem. The UART triggers an interrupt when it becomes ready to send a byte. If it is already ready (such as at boot time), the interrupt will not trigger. So we try this:

	while(1) {
		read(fd, &c, 1);
		*uart = c;
		interrupt_wait(intr);
	}

This is closer, but in both of these versions there is a race condition where the UART might not actually be ready anymore by the time we process the interrupt (depending on what other tasks are doing), so let’s add a check for that:

	while(1) {
		read(fd, &c, 1);
		if(!(*(uart + UARTFR) & UARTFR_TXFF)) {
			*uart = c;
		}
		interrupt_wait(intr);
	}

Now, however, if the UART is not ready, we will skip bytes. So we only want to read a new byte if we actually wrote the current one:

	int doread = 1;
	while(1) {
		if(doread) read(fd, &c, 1);
		doread = 0;
		if(!(*(uart + UARTFR) & UARTFR_TXFF)) {
			*uart = c;
			doread = 1;
		}
		interrupt_wait(intr);
	}

To test this out, let’s set up a new first:

void first(void) {
	int fd;

	if(!fork()) pathserver();
	if(!fork()) serialout(UART0, PIC_UART0);

	fd = open("/dev/tty0/out", 0);
	write(fd, "woo\n", sizeof("woo\n"));
	write(fd, "thar\n", sizeof("thar\n"));
	while(1);
}

Our current pre-emption speed will make this a bit of a pain to run, so you can set the clock value in main to something lower.

We can now print to the serial port by writing to a fifo!

Code for this section on Github.

Input Driver

Some new magic numbers:

#define UARTICR 0x11 /* 0x44 bytes */
#define UARTFR_RXFE 0x10
#define UARTIMSC_RXIM 0x10
#define UARTICR_RXIC 0x10
#define UARTICR_TXIC 0x20


The input driver will be another process that waits on the same interrupt. Unfortunately, because this one interrupt actually can signal one of several things, if two of them are true, and different processes work on processing them, the interrupt will fire constantly and prevent any work from getting done. We solve this by clearing the more specific interrupt in the drivers where we handle them. For example, in serialout:

*(uart + UARTICR) = UARTICR_TXIC;

Next, start with some setup:

void serialin(volatile unsigned int* uart, unsigned int intr) {
	int fd;
	char c;
	mkfifo("/dev/tty0/in", 0);
	fd = open("/dev/tty0/in", 0);

	/* enable RX interrupt on UART */
	*(uart + UARTIMSC) |= UARTIMSC_RXIM;

	while(1) {

We want to wait on the interrupt, and then immidiately clear the specific interrupt we handle, just like in the output driver:

	interrupt_wait(intr);
	*(uart + UARTICR) = UARTICR_RXIC;

And we want to make sure, not only that no race condition has changed the UART status, but that the interrupt was meant for us at all:

	if(!(*(uart + UARTFR) & UARTFR_RXFE)) {

And then just read the byte and put it in our fifo:

	c = *uart;
	write(fd, &c, 1);

This input server has a bit of a bug. If the fifo is ever full, it will block, and maybe miss input bytes from the serial port. Our processes are likely to be way faster than any serial port, but still, in a real kernel you would want to account for this by splitting the driver into two processes: one that waits on the interrupt, and one that buffers the bytes internally.

Code for this section on Github.

Echo Application

To tie all this up, we write a simple application that will echo back every character you type:

void echo(void) {
	int fdout, fdin;
	char c;
	fdout = open("/dev/tty0/out", 0);
	fdin = open("/dev/tty0/in", 0);

	while(1) {
		read(fdin, &c, 1);
		write(fdout, &c, 1);
	}
}

Code for this section on Github.

We’re done!

We now have a working serial port driver that respects the hardware by using interrupts instead of burning CPU time looping on flag checks. Remember, any interactios with the UART will affect these drivers, so using bwputs may have strange side effects now.

To be honest, this is about as far as I expected to get when I originally started this series. I could write a clock driver, to allow for sleep calls and similar, but you should be able to see how that might work. One big topic I have avoided thus far is enabling the MMU and doing virtual memory / memory protection. I may try to figure out how that works on this harware, but it will require quite a bit of doing. Suggestions for others things you would like to see in this series go in the comments!

Code so far on Github.