localStorage security

In my previous post, I wrote about a scheme to replace symmetric key login with an asymmetric key scheme where it keeps its keys in localStorage. I mentioned that the use of localStorage is certainly a questionable proposition, and looking around the web there are definitely a lot of OMG YOU CAN"T DO THAT!!! floating around. So it's worth going over the potential attacks on this scheme which so relies on the relative security of localStorage.

First off, what are the security properties of localStorage itself? Not much. The browser walls localStorage off from other origins so that other sites can't snarf up its contents, but other than that anything executing within the context of the origin site can grab the entire localStorage object and have at it. That sounds pretty scary to be saving rsa private keys. But taking the next step, I can really only see two vector that could be used to reveal those keys: somebody or something with access to, say, a javascript console could trivially grab those keys. The other vulnerability is XSS and other kinds of injection attacks where malicious javascript loaded into the browser could gain access to the keys.

Local Access to LocalStorage

So how much of a threat is malware or somebody with console access snooping on keys? Well, it's obviously a big threat. If somebody has unfettered access to your device they can do just about anything they want including going into your browser and opening up the javascript console, for example, and snarfing your keys. So that sounds fatal, right? Well yes it is fatal, but it's not uniquely fatal: they can also snarf up all of the stored browser passwords trivially, install keyboard loggers to grab your credit cards, and generally wreak havoc. So pointing out that bad guys 0wning your computer is bad doesn't say much about this scheme. Does this scheme make things even worse? I don't see it. You might point out that I can lock my browser with a password which assumedly encrypts its stored passwords. Assuming that that password hasn't been compromised (a big assumption), it provides some protection against snoopers. But there's nothing fundamental about my scheme that prevents using a local PIN/password to unlock the key storage as well. Indeed, some apps may very well want to do that to increase the security at the cost of more hassle for the user. Another possibility is that we can do some cat-and-mouse games with the keys. See below.

Evil Code and Key Security

Like most environments, executing evil code is not a particularly good idea. For javascript, once a piece of code has been eval'd, its just so much byte code in the interpreter and the browser cares the least about its provenance with respect to who can see what within that browser session. So the short answer is that evil code loaded into your site's browser window is going to give complete access to the user's private key. We can maybe make that a little harder for the attacker (see below), but at its heart I cannot see how you can truly avoid disclosure. That said, code injection is a serious threat period: if evil code is inserted into your browser session, it can snoop on your keyboard strokes to get blackmail material when you're on sites that maybe you ought not be, make ajax calls back to your site in the user's name to, oh say, buy those UGG's the attacker has been drooling over, and generally do, well, any number of evil things. So there is ample reason to protect against code injection attacks and the web is full of information about what one must to to protect yourself. So I don't see how we're making things worse by keeping private keys in localStorage. 

Storing Passwords vs Asymmetric Keys

It's worth noting that we're talking about asymmetric keys and not passwords when evaluating threats. As Linkedin shows, the real threat of a password breach is not really so much to the site that was breached, it's all of the sites you used the same password on too. That is, the collateral damage is really what's scary because you've probably forgotten 90% of the sites that have required you to create an account. So storing a password in localStorage gives a multiplier effect to the attacker: one successful crack gains them potential access to 100's if not 1000's of sites. That's pretty scary especially considering that the attacker just has to penetrate one site that's clueless about XSS to get the party rolling. Worse: revocation requires the user to diligently go to every site they used that password on and change it. Assuming you remember every site. Assuming the bad guys haven't got there first.

So for the inherent vulnerabilities of storing asymmetric keys in localStorage, it's worth noting that we gain some important security properties, namely the localization of damage due to being cracked. If an attacker gains access to a user's private key they only gain access to the site that the key is associated with, not the entire Internet full of sites that require accounts. And revocation is trivial: all that is required to stanch a breach is to revoke the individual key that was compromised on the individual browser that stored it. It's not even clear that you ought to revoke all of the keys associated with a given compromised user's account since they'd still need to have access to the individual devices with the two attacks above. But it might be prudent to do so anyway. And it seems very prudent when storing the public keys associated with a user account to store some identifying information like the IP address, browser string, etc so that users can try to get some clue whether new enrollments have been performed (which would indicate that their email password has been breached too given the way we do enrollment). 

Cat and Mouse 

One last thing, everybody with half a security head will most likely tell you that security through obscurity is worthless. Heck, I've said it myself any number of times. And it's still mostly true. However, it's also true that bad guys are as a general rule extremely opportunistic and are much more likely to go after something that's easy to attack than something that's harder -- all things being equal. And if an attack gives you a multiplier effect like getting a widely shared password, even better. So while we can't absolutely thwart bad guys once they're in and have free reign, we can make it a little harder to both find the keys in localStorage, and to make use of them once they find them. For one, we could conceivable encrypt the private key in localStorage with a key supplied by the server. Yes of course the attacker can get that key if it has access to the code, but the object here is to make their life harder. If they have to individually customize their attack scripts, they're going to ask the obvious question of opportunity cost: is it worth their while to go to the effort, especially if the code fingerprint changes often? Most likely they'll go after easier marks. 

Likewise, we could change the name of the localStorage key that contains the credentials and set up a lot of decoys in the localStorage dictionary to require more effort on the attacker. And server-side, we could be on the lookout for code that is in succession going through decoy keys looking for the real key. And so on. The point here is to get the attacker to go try to mug somebody else.

So is it Safe?

Security is a notoriously tricky thing so anybody who claims that something is "safe" is really asking for it. Are there more attacks than I've gone over here? It would be foolish to answer "no". But these are the big considerations that I see. Security in the end is about weighing risks. We all know that passwords suck in the biggest possible way, so the real question is whether this scheme is safer. Unless there are some serious other attacks against the localStorage -- or the scheme as a whole -- that I'm unaware of, it sure looks to me that the new risks introduced by this scheme are better than the old risks of massively shared passwords. But part of the reason I've made this public is to shine light on the subject. If you can think of other attack vectors, I'm all ears.

Asymmetric Key Login/Join

Using Asymmetric Keys for Web Join/Login

I've written in the past about how Phresheez does things a little different on the username/password front by auto-generating a password for each user at join time. This has the great property that if Phresheez has a compromise, it doesn't affect zillions of other accounts on the net. However, the password is still a symmetric key which has to be stored encrypted for password recovery. Storing any symmetric key is not ideal.

As I mentioned in the previous post, what we're really doing here is enrolling a new device (browser, phone, etc) to be able to access the Phresheez server resources -- either the first time as when you join, or for subsequent logins. So I've come up with a new method using asymmetric keys which neatly avoids the problem for storing sensitive symmetric key data on the server. The server instead stores public keys, which by definition are... public. So if Phresheez is compromised, they get none of the sensitive credential information, just public keys which are worth something to Phresheez, but worthless to everybody else unless they have the corresponding private key, which is supposedly hard to obtain. This takes this compromise isolation scheme to the next step, and is surprisingly straightforward.

For web use, I'm taking advantage of the new html 5 localStorage feature to store the asymmetric key. localStorage seems sort of frightening, but the browser does enforce a same origin policy to limit who can use it. If we believe that the browser protections are adequate (and that's worth questioning), then we can use it to store the asymmetric key. Note that although these flows do this in terms of web sites, there is nothing that *require* using web technology. The cool thing is that it works within the *confines* of current web technology, which is sort of a least common denominator.

Join

1.  To join, the app generates a public/private key pair and prompts the user for a username and   some recovery fallbacks (eg, email, sms, etc). The asymmetric key is stored in localStorage   for later use along with the username it is bound to. The app then signs the join information   (see below) using the private key of the key pair, and forms a message with the public key and the signed data.
2. The server receives the message and verifies the signed data using the supplied key. This proves that the app was in possession of the private key for the key pair. The server creates the account and adds the public key to a list of public keys associated with this account.

Login

1. Each time the user needs to log in to the server, it creates a login message (see below) and does a private key signature of the message using the asymmetric key stored in localStorage. The signed login message along with the associated public key is sent to the server.
2. The server receives the message and verifies the signed data. If the supplied public key is amongst the set of valid public keys for the supplied username, then the login proceeds.  See below for a discussion about replay.

Enrolling a New Display Head

1. When a user wants to start using a different device, they have two choices: use a currently enrolled device to permit the enrollment or resort to recovery mode using email, sms, etc
2. To enroll a new device using an existing app, the app can prompt the user for a temporary pass phrase on the currently enrolled app. this password is a one-use password and expires in a fixed amount of time (say, 30 minutes). It doesn't need to be an overly fussy password since it's one-time and timed out. The app sends this temporary password to the server with a login message (see below). The server saves the temporary password and timestamps it for deletion -- say less than one hour. An alternative, is that the app can generate a one-time password for the user and send it to the server. Either work.
3. Alternatively, if an enrolled device is not available, a user can request that a temporary password be mailed, sms'ed, etc to the user. The server stores the temporary and timestamped password as above. The user receives the temporary password and follows steps 4 and 5 to complete the enrollment.
4. The user then goes to the new device where they are prompted for a username and the temporary password. The new device creates a public/private key pair as in the join flow, and signs a json blob with the username and temporary password (see below). The new key pair is stored in the new device's localStorage
5. The server receives the signed json data along with the new public key and checks the temp password stored in 2a or 2b. If they match, the temp password is deleted on the server, and the new public key is added to the list of acceptable public keys for this user. Subsequent logins from this device follow the login flow.

Message Formats

Note that there isn't anything sacrosanct about json here. It could be done using a GET/POST URLEncoded form data too. I just happen to find json a nice meta language. And by Signed, I mean a sha1|256 hash over the data and signed with the private key. I suppose I could sign the pubkey as well, but that's just details, just like i'm not specifying the canonicalization of what's in the hash.

login/join message

{"pubkey":"--pubkey data--", "signature":"RSA-signature over login-blob", "body":"--signed login/join--"}

signed login

{"cmd":"login", "username":"bob", "timestamp":"unix-timestamp", "optional-temp-password":"otp"}

signed join

{"cmd":"join", "username":"bob", "timestamp":"unix-timestamp", "email":"bob@example.com", "sms":"1.555.1212"}

replies (not exhaustive)

{"sts":200, "comment":"ok"}
{"sts":400, "comment":"database down"}
{"sts":500, "comment":"bad encrypted data"}
{"sts":500, "comment":"timestamp expired"}
{"sts":501, "comment":"username taken"} // joins, but a re-join with a enrolled key is ok

Replay Protection

It's worth discussing replay protection. Here I have a timestamp which would assumedly need to be fairly well synchronized with the server time, and be relatively short lived -- say a few minutes. Alternatively, if it's acceptable to add a step, the client can request that the server send a nonce and add the nonce to the encrypted blobs instead.

In all cases, however, it should be assumed that the entire transaction is sent using TLS so that the server-client communication is private. Subsequent transactions may or may not be sent over tls... session management, etc is out of scope of this idea.

Multiple Accounts on One Device

A shared device with multiple account is possible if the username is stored along with the asymmetric key pair binding them to each other. Multiple entries can be kept, one for each credential, and selected by the current user. This, of course, is fraught with the possiblity for abuse, since you're enrolling the device potentially long-term. A couple of things can possibly be done to combat that. First, the user can request that the credential be erased from localStorage. Similarly, in the enrollment phase, a user could request that the key pair only be kept for a certain amount of time, or that it not be stored at all. Last, it's probably best to just not use shared devices at all since that's never especially safe.

About Public Key Encryption

I'm a little creaky on RSA right now so forgive me if I get some of the details wrong. I've checked on a newish linux box running Chrome and public key verifies are cheap while private key signatures from within javascript are more painful (~1s for a 1024bit rsa). I doubt that's a deal breaker, and in the long term giving native BIGNUM support to javascript may not be a bad idea. For hybrid apps like Phresheez, it could even reach out to the native layer to get keys and signatures if it's a big problem. Likewise, generating keys can be slow, but possibly backgrounded while the user is typing username, email, etc if they need to. And of course, there's the perennial question of the RNG. How good Math.Random() in js is certainly an interesting question. However, we have to keep in perspective what we're really changing from which is crappy megashared user passwords. A 512 bit RSA key with a not terribly good RNG is most likely still better that the current situation, and there's a pretty darn good chance that we can do better - maybe even much better.

Conclusion

In conclusion, this mechanism provides a way to finally break the logjam of pervasive insecure  shared secret schemes that are so prevalent not only on the web, but everywhere. The server never needs to keep a long term and potentially sensitive symmetric key, nor does it ever need to store anything that is not fundamentally public (ie, a public key). This wasn't really available for use one the web until we could use localStorage to store credentials in the browser. In conjunction with out of band recovery mechanisms like email, SMS, as well as currently enrolled devices, we can enroll new devices using that generate their own credentials so that a compromise of one device doesn't even compromise other  devices you own. 

The big question is whether we can make the user experience close to what people's current expectations are, but with a few twists -- like, for example, making clear that "recovery" isn't a moral failing, but the expected way you enroll new devices. UX is a tricky thing, and should not be discounted, but it seems there is at least hope that it could be successful.

Client vs Server Charts

Charts are a very quick way to view statistical data, and good charting packages can bring a lot of neat ways to slice and dice that data. Since Phresheez started out as a fairly typical server side web site, it was pretty natural to generate the charts server side as well. Back then, javascript was still pretty slow, and html5 canvas support nonexistent. A more serious problem in reality is that I hadn't taken the step to process and cache the statistics for a day, so the amount of data required to be sent to the browser could pretty big -- on the order 100kb typically. So I never really considered it.

Besides, the graphing package I used (jpgraph) is pretty complete and I've really had no complaints about it per se. It's biggest problem honestly is its reliance on an underlying library -- libgd -- which isn't the best. Ok, having done graphics kernels before, it pretty much sucks. In particular, the curve algorithm really sucks producing jaggies and that really bothers me (it's almost as if they're using a two direction ellipse step algorithm rather than 3 directions which didn't Bresenham figure out ages ago?) . And it can't figure out how to write text on the baseline when it's anything other than horizontal, which makes the graphs look rather amateurish. But they have served me well, and it's definitely useful because sometimes only an image will work, like when you need to post goodies to Facebook which doesn't allow arbitrary blobs of html and javascript for pretty obvious reasons.

In the past year, I had made some changes to the server side graphs to freshen them up. This included using a graphic artist's best friend -- gradients. Without getting ratholed about whether that's a good or bad thing, one noticeable effect of using gradients is that the size of the .png (.jpg's look horrible) goes up dramatically. No surprise, but the once 10-20kb graphs were now 30-40kb each. Given that they looked slicker, it seemed a decent tradeoff. A more pernicious issue, however, was caching. People jump between pages with various graphs all of the time. Since people are looking at the graphs, oh say, at lunch when they've been skiing, the images cannot reasonably be cached -- the GPS uploaders are all busy at work for both you and your friends, and you expect that the charts will be kept up to date. So in reality, that 30-40kb is multiplied by the number of charts, your number of friends, and the the number of times you look at the app again. While it was certainly server load, I was much more concerned about user experience since often the reception at resorts suck and trying to download a 30-40kb image each time seems... slow.

So I had long ago fixed the stat aggregation caching problem for it's own obvious benefit. I had been playing around with html5 canvas stuff and was generally impressed with how well it behaved cross platform -- even ie9 does a pretty good job from what I can tell. So I decided on a whim to start looking for a javascript package that does graphing. I'll admit that my research on the subject wasn't the deepest -- in the beginning I was mainly interested in just testing the waters -- but I eventually settled on RGraph. Since the server side graphs had been evolving for years, I was rather worried about how long it would take to just get to the baseline of what I had server side, but I was rather pleasantly surprised that it only took me a week, maybe two tops to get to parity. Better is that the rendering on browsers is much better than libgd, so goodbye jaggies. And it can do cute little animations. And since it's client side, I can attach events to the graphs more easily -- yes, I know that it's a hack since it's a Canvas rather than SVG, but still it's easier to contemplate than krufty image maps. 

I had been vacillating about whether to make the change for quite some time for one reason: it increases the size of the web widget by about 100kb, which was pretty substantial. What finally won me over was that I realized that I was being penny wise pound foolish: the cost of an image is say 30kb, and you might look at 3 of them for yourself at one sitting, several for your friends and then you may have several sittings as well. This all adds up, and as I mentioned it creates noticeable lag in the user experience. The client side graphs, on the other hand, all use the same cached statistics blob which is about 10k uncompressed, 3-4k compressed over the wire. So where you might be looking at 200-300kb or more of data transfer over a day, doing it client side is probably on the order 10-20kb, if that. And it appears almost instantaneously, especially if it doesn't need to refresh the stats blob. Compare that to the upfront 100kb code investment which is amortized over the life of the web widget which is generally every couple of weeks, maybe longer, it became obvious that this was a no-brainer. 

So I've managed to convert everything over and push out a release. Everything seems to be working, but corner cases on graphs are hard to ferret out (thinking labeling, grrr) so I expect there will be some futzing as they crop up. The support at RGraph was very quick and they're receptive to upgrades which I have a few smallish things that I've dropped the ball on. It's a client side world.

Phresheez Join Passwords

Phresheez requires that you create an account because in order to do anything interesting it needs a place to send points to on the backend. However, we had quite a bit of evidence that users abandon the app before ever signing up. There's probably a variety of reasons for this, but it probably boils down to one of two reasons: either they just don't want to have yet another thing that requires a username and password, or they find that it is too onerous to type all of the necessary information in. I read a very interesting piece on iPad Usability which mostly applies to phones too, and one not very surprising observation is that people really dislike typing on their phones. For Phresheez that's probably even worse because they are probably finding out about us through friends and are probably in a hurry to get out skiing.

So I asked myself, what can I do to lower that energy barrier? Starting out with a naked form is the least friendly, and auto-generating a user account is the best. So I started looking on Android and lo and behold, there is a way to get the user's gmail address. Groovy. Since the left hand side of a gmail address is very likely to be globally unique, I can then use that as the seed to create a unique user id. For the email address, it's a no brainer since we already have the email address. That just leaves the password.

When I started thinking about this, it occurred to me that I could just auto-create a good strong password for them. The app stores the password so it doesn't have to be something they need to remember. Well that's almost true: Phresheez is both an app and a web site, so they may want to know the password to see their stuff on the site as well. I fretted about this quite a bit, but ultimately I decided that a compromise was that I'd auto-generate their password, but leave it in clear text until they decided to lock it which gave them the opportunity to type their password into the web site. That and there's always password recovery. That's where things currently stand.

In doing this I realized that this method has a very interesting security property: since Phresheez generates the password for you, any compromise of Phresheez will not compromise other sites where you might otherwise use the same/similar password. Yes we all know that it is bad to use the same password on multiple sites, but it is the reality of the world that people do this. And why wouldn't they? People are required to join probably hundreds if not thousands of sites for various reasons. Are we really to expect that they create a unique and hard to guess password for each site? Of course not, that's complete idiocy and anybody who spouts such a thing  should be flayed alive.

The Linkedin fuckup got me to thinking about this again though. In my annoyance, I posted to NANOG what I thought was so completely wrong about the blog post's posturing toward st00pid lusers. Many people chimed in that anybody who isn't using a password vault thingamajig deserves what's coming to them. But that really misses the point: putting the onus on users to protect themselves is first of all a provably losing proposition, but also obscures the fact that we have been putting them in a completely untenable situation. The current username/password scheme is nearly 50 years old and it really shows. Everybody knows it sucks, so scolding users for being human is not the answer for what is really an engineering failure.

What occurred to me is that the real security advantage of the way that Phresheez does things is that it puts security in the hands of Phresheez rather than users who don't have any clue. They don't have to know to download and use some password vault thingy. Apps can already store your credentials, and all browsers have password rememberers. And even if the browser doesn't have a rememberer, you can almost certainly use html5 localStorage to remember it. As for the need for cross-device passwords that vexed me? Well, now that I consider it, the real answer is password recovery. Every site needs the ability to recover usernames and/or passwords and it is done via your supplied email address. This is just a fact, and is completely orthogonal to password generation. If password recovery has to be there anyway, why not use as feature rather than a necessary evil? Since it is a necessity we shouldn't make password recovery a semi-shameful thing that you "forgot", but the normal way of enrolling a new display head to the site. Maybe we should put a positive spin on password recovery from being something you "forgot" to being something that allows you to add a new device to see your goodies on. That it's the *normal* and expected way to see stuff on multiple display heads, not a failure of character.

In conclusion, I started down this road because auto-generating passwords was more user friendly, but it has turned out that it is seemingly a much more secure way of enrolling users as well. And it puts the onus for better security on developers rather than end users. Snicker all you like about that, but at least there's a chance that developers can be beaten to do the right thing, especially since this isn't all that hard to do.

Phresheez Has a Yard Sale

Kaboom

I wouldn't have said that Phresheez had an ironclad disaster recovery plan, but at least we had a plan. We do mysql database replication back to my server on mtcc.com and do daily backups of the entire database. We have backups of the server setup, and more importantly have a step-by-step build-the-server-from-the-disto along with config files are checked into svn. We only have one active production server, so that implicitly accepts that significant downtime is possible. On the other hand, we have been running non-stop since 2009 with exactly one glitch where a misconfigured Apache ate all its VM and wedged. That lasted about an hour or so -- hey, we were skiing at the time, so all in all not terrible for a shoestring budget.

Our downtime doesn't take into account routine maintenance, and I had been in need of doing a schema update on our largest table, the GPS point database. So I happened to wake up at 2am and decided to use the opportunity to make the change. Nothing complicated -- just take the site offline and make a duplicate table with the new schema. That's when the fun began. Each of several times I tried, the master side gave up complaining about something going wrong with the old table. I then tried to do a repair table on it, and it bombed too. Strange. After the fact -- a mistake, but it didn't matter as it turns out -- I decided to do a file system copy of the database table file. Death. Dmesg is definitely not happy either about the disk. I tried to see if the index file was ok, and same problem. I tried other large tables, and they seemed ok. A mysql utility confirmed that it was just the GPS point database, even though that was pretty bad by itself.

So... I was pretty much hosed. Something had blown a hole into the file system and torched my biggest table -- some 15 Gig big. Fsck with some prodding discovered and repaired the file system, but couldn't salvage the files themselves. So it's restore from backup time. Ugh.

There were two options at this point: do a backup of the slave or just copy over the slave's data file. I wasn't entirely coherent (it was early), and decided to give the first a try. Here's where the first gigantic hole in our strategy came in: either method required that a huge file be copied from the slave server to the master. Except the slave is a machine on a home DSL uplink getting about 100 KB/sec throughput: scp was saying about 12 hours transfer time. Oops.

The long and short is that after about 12 hours, the file was copied over, the index was regenerated and Phresheez was up again no worse for wear as far as I can tell. A very long day for me, and a bunch of unhappy Phresheez users.

Post Mortem

So here's what I learned out of all of this.

• First and foremost, the speed of recovery was completely dependent on the speed of copying a backup to the production server. This needs to be dealt with some way. First might be copying the backups to usb flash and finding somebody with a fast upstream to be able to copy stuff to the production machine. Better would be to spend more money per month and put the slave on its own server in the clould. But that costs money.
• Large tables are not so good. I've heard this over and over, and have been uncomfortable about the GPS point table's size (~300M rows), but had been thinking about it more from a performance standpoint than a disaster standpoint. I've had a plan to shard that table, but wasn't planning on doing anything until the summer low season. However, since the downtime was purely a function of the size of the damaged table, this is really worth doing.

Disaster on the Cheap

The long and short of this is that when you have single points of failure, you get single points of failure. Duh. The real question is how to finesse this on the cheap. The first thing is that getting access to copy the backup over the net quickly would have cut the downtime about an order of magnitude in this case. Sharding would have also cut the downtime significantly, and for that table really needs to be done anyway.

However, this is really just nibbling at the edges of what a "real" system should be. Had the disk been completely cratered, it would have required a complete rebuild of the server and its contents and it would have still been hours, though maybe not the 12 hours of downtime we suffered. Throwing some money at the problem could significantly reduce the downtime though. Moving the replication to another server in the cloud instead of on home DSL would help quite a bit because the net copy would take minutes at most.

A better solution would be to set up two identical systems where you can switch the slave to being a master on a moment's notice. The nominal cost is 2x-3x or more because of the cost of storing the daily backups -- disk space costs on servers. The slave could be scaled down for CPU/RAM, but that only reduces cost to a point. Another strategy could be to keep the current situation where I replicate to cheap storage at home and keep the long term backups there, but keep a second live replication on another server in the cloud. The advantage of this is that it's likely that a meltdown on the master doesn't affect the slave (as was the case above), so a quick shutdown of the cloud slave to get a backup, or switching it over to be a master would lead to much better uptime. Keeping the long term backups on mtcc.com just becomes the third part of triple redundancy and is only for complete nightmare scenarios.

Is it worth it? I'm not sure. It may be that just getting a fast way to upload backups is acceptable at this point. One thing to be said is that introducing complexity makes the system more prone to errors, and even catastrophic ones. I use replication because it would be unacceptable to have 2 hours of nightly downtime to do backups. However, mysql replication is, shall we say, sort of brittle and it still makes me nervous. Likewise, adding a bunch of automated complexity to the system increases the chances of a giant clusterfuck at the worst possible time. So I'm cautious for now -- what's the smallest and safest thing I can do get my uptime after disaster into an acceptable zone. For now, that's finding a way to get backups onto that server pronto, and I'll think about the other costs/complexities before I rush headlong into it.

So What Happened to the File System Anyway?

At some level, shit just happens. It's not whether something will fail, but when it will fail and how quickly you can recover. But this is the second time in about a month or so that I've had problems that required fsck to come to the rescue. My provider had recently moved me to a new SAN because the previous one was oversubscribed. Did something happen in the xfer? Or is their SAN gear buggy leading to corruption? I dunno. All I know is that I haven't had any reboots of any kind since I moved to the new SAN so there shouldn't have been a problem unless it went back before the SAN move. I sort of doubt that the underlying Linux file system is the cause -- there's so much mileage on that software I'd be surprised. However, after fighting with my provider about horrible performance (100kb/second transfer for days on end) with their SAN's and now this... I'm thinking very seriously about the options.

The First Two Minutes

I had an interesting email exchange with my friend J. at Twitter. Twitter had just rolled out a new version that was being hyped up as new and improved and specifically was trying to target "new" users by making Twitter more understandable to the uninitiated. I fit that bill to a Tee as I really don't "get" Twitter, so I head over to their web site to take a look expecting to be instantly gratified. It looked the same as the last time I looked though. I couldn't identify even one thing that looked different than the last time, and it was as dry and lifeless as it was the last time. I even loaded up their app. Same. So after a couple of minutes of searching for what the hype was about, I gave up and left.

Later, I sent J. some mail about this and after several exchanges he told me they don't normally care what people have to say until they've used it for about a week or so. Thud. That's the sound of my jaw dropping in disbelief. A week? You lost me in the first few minutes! There was no week for me to get used to anything because I -- like lots of people purportedly -- abandoned it. Again.

Twitter is hardly unique, of course. Every app probably has about a minute or two to engage you. If you don't, you'll be abandoned -- either outright deleted, or ignored until the user does a sweep of the deadwood cluttering up their home pages. Maybe you get one more chance, but that's probably it; some day I should try to grind those stats out for Phresheez. Since I was probably harsh on J., I decided to look at Phresheez from that perspective as well. It of course completely failed too. Phresheez is a particularly hard sell: people download it because they think it has something to do with skiing. Since Phresheez is a tracking app, unless you're already skiing at the time your gratification is not instant. From what I can tell with the data, lots of people load it either far in advance (ie in the months leading up to ski season), or load it days in advance (ie, leading up to a trip). Both require a pretty big leap of faith on the part of our users: that we'll entertain them if they use it. I don't doubt that quite a bit of our abandonment rate is nothing more than people forgetting about the app in their excitement to get on the mountain.

I don't think that explains all of it though. In fact, I suspect that the reason for abandonment of Phresheez is probably much like Twitter's. With Twitter, you are invited after joining to write something. So you do. Nothing happens. Why did I do that? This app sucks. So I never see the vast amount of content that Twitter actual has. Phresheez is sort of similar: you load the app and after you join it tells you to go outside. So you go outside and it gets a GPS fix. In pressing buttons, you'll see yourself on a map, and maybe some not terribly exciting charts and stats. Ok, maybe there's some there there, but you don't see all of the gaming, awards, cool charts, etc, etc, in full action. Maybe you see some potential and don't delete it outright, but there's a good chance you'll forget about it when the time comes.

So we're both guilty of the same sin: we failed to instantly gratify. Strangely, I have a much better idea about what Twitter should be doing than what Phresheez should be doing. Twitter has a huge amount of content, and it probably wouldn't take too much to coax some idea of the kind of content I'd like to see: Twitter is amazingly good at finding out about fires in SF in real time, for example. So I -- like lots of people -- am likely to be interested by default about what's trending in my home town. Why didn't they entertain me with that in the first few minutes? And why don't they try to figure out what might interest me either explicitly like the Netflix playlist suggestion machine, or making it obvious that searches are as important if not the most important in the first few minutes? Instead they make me feel stupid for typing in a tweet that I know that nobody will read. This is a huge advantage that Twitter has: it's not difficult to be passively entertained. Yet Twitter squanders that advantage. I don't get it.

As for Phresheez, it's still a tough problem. People do seem to respond to notification meatballs, so I added some pre-populated notifications upon join with cool days from our users. That seems to get some response. In an unreleased version, the app will also have as a top level option to view a video which shows you what the app can do. I'm not as optimistic about that because it's not all that different than the screen shots in the app store, but you never know. What I'd really like is for people who load it at home to just go out, take a bike ride, walk, etc and then check in again, but that seems to be asking for too much. Aside from the blinders of "it's a ski app!", asking for that violates the two minute rule.

One thing that really did make a big difference in the abandonment rate, however, is streamlining the join process. If you don't need or can postpone a join process, you're already ahead of the game: people hate typing, and are rightfully suspicious of giving up information. For Phresheez, we need a username, password and email address. As it happens, you can get the gmail address on Android so I use that for email, and use the local part to guess at a unique username. For the password, I just auto-generate it so that the only thing that the new user has to do is press the join button. There is no equivalent on iPhone, but I recently did a similar thing using Facebook to auto-populate these fields -- a "Join with Facebook" flow. For Android it was pretty obvious that it made a big difference, though I haven't quantified it. The "Join with Facebook" feature hasn't been released yet, but it will most likely help on iPhone.

In conclusion, my scolding of J. only managed to show my own inadequacies with Phresheez. This is a really tough problem for a Phresheez kind of app, or any sort of app that demands delayed gratification. The first step, however, is realizing how important the problem really is. So I grok that now. But still it's tough.

The Trouble with Templates

Templates have always perplexed me. They sure sound like a good idea: who likes the idea of all kinds of layout and content wrapped up in one big ball of code that plops it out? Wouldn't it be nicer to have a clean layer that separates layout, content, and code? Here's a piece of html layout, some json content blob and poof! A page pops out. Nothing can be easy though, so your majick needs to know how to bind the content to the layout.

So the first thing that comes to my mind is, well, howse about something that just binds values to particular places in some html template via some regex incantation? But that majick is weak and needs to know where to bind things. And JSON itself is just a way to marshal bits onto the wire, not a formatting transmogrifier. So it would be great if a couple of regex's were all that it really took to work, but it doesn't in the normal day to day world from what I've seen. Just take for example 1 vs. 2. Assuming your JSON blob just has a number in it, you often want to pluralize the word around the number itself. That requires either the ick factor of pluralizing the word server-side, or the embedded logic to do the same. And that's just one very small example... there are legions.

So what is a template anyway? I'd say that it's data possibly with embedded control logic. What's a normal script? Control logic possibly with embedded data. It seems to me that you pick your poison: do you want the HTML layout to drive the overall structure of the module, or do you want the programming structure to drive the layout of the page? All of the templating solutions I've seen are HTML with embedded control logic, usually with some obscure control language that somebody hacked together. With phpBB which I use for Phresheez forums, the templating is even weirder: they have built their own control language, one of whose controls is a PHP escape itself! So you have a PHP script interpreting a template which can then eval PHP. This is better than just doing an include() with embedded PHP itself? At the very least, the implicit eval in include() is going to be faster than any PHP parsing through the template (and then possibly calling eval again). So I just don't get it.

One thing seems pretty clear to me: for all of its virtues, HTML as a structured way of looking at programmatic source leaves a lot to be desired. Maybe it's me, but I find that I figure out what's going on in a module a lot faster if I look at a real source module rather than a bunch of isolated programming in a sea of markup. But then again, a bunch of html += '' are pretty heinous too. Neither method really makes a barrier if you want a separation of layout monkeys from code monkeys. They're both likely to have their fingers in the same modules, and they're certainly going to have to coordinate no matter what.

It seems possible, however, to have templates that are nothing more than markup though. If the pertinent nodes in the DOM are assigned ID's and js code can hunt them down, add the needed listeners, add the formatted data from the JSON blob, etc. How practical this is I don't know. For one, you're likely to be still generating some markup in code -- a table with a dynamic number of rows, for example. Honestly the thing that frightens me the most about that sort of mechanism is how you create, coordinate, and maintain the ID namespace between the layout monkeys and the code monkeys. That really seems like a nightmare.

I'm not really sure what the solution is, but maybe this might be a start of a conversation: don't shoehorn mock programming logic into html. Even on its best day, it's an orc-like counterfeit of a real programming language even in comparison to a the decidedly low-elf javascript. Maybe instead what would be good is to actually just *split* the HTML layout duties if that's how your shop works. That is, the layout monkey checks in a real piece of html -- without ID's as above -- and the code monkey translates that into working code that can deal with all of the subtleties that cause much consternation with an inadequately expressive templating language. Sure it takes two steps, but that's sort of a good thing in my view: each piece of code is owned by the proper stake holder. Diff is our friend: when changes happen to the layout, you just diff the html source and figure out how that affects the actual layout generating code.

So I guess the long and short of this is that I'm just not terribly convinced that solve more problems than they create. They sound good, but they always seem to keep coming up short from what I can see. Maybe it's yet another case of TANSTAAFL. In a new client centric world, maybe it's time to step back a bit and go back to first principles for a while though.