my task

"connect 80k simultaneous clients, logging in and out at an overall rate of 50Hz, and requesting pages at an overall rate of 100Hz"

lets model a client as a random walk on a state graph (a Markov chain), where the action taken by a client is drawn from a distribution conditioned on the current action. (TL;DR we will bootstrap frequency estimates from this representation using eigenvectors)



Remove redundant brackets from expressions with Falafel

I am writing a static analyser for Firebase. My approach to precedence and brackets has always been, if in doubt, whack a bracket round it. Unfortunately when writing code generation tools my output ends up with more brackets than logic :s

Anyway, I was having trouble debugging my generated code, and it was impossible to reason why it wasn’t working what with all the brackets. So I kept with maximum bracket insertion in the 1st pass (a reliable strategy), but wrote a function for removing ALL redundant brackets from a Javascript expression post 1st pass (easier to read). It took a while to get right but I was very pleased with the brevity.

So the expression: (4 + 6) * 6 needs its brackets to operated correctly, but (((4) + (7) * 8) has a number of pointless brackets in it. There are some tricky cases like 5 / (7 * 6), which requires brackets despite * and / having equal precedence. To understand why 5 / (6 * 7) needs brackets you have to understand that when operators have equal precedence, by default, they associate to the left. Anyway the code to actually do this turned out to be super compact so I share it with you!

I used node-falafel which is an awesome package that allows in place source code rewrites during a bottom up parse. Woah! The function “simplify” takes an expression as a string, and returns a functionally equivalent expression but without the pointless brackets in it. Nice!


Firesafe: A Prophylactic For Firebase

Firesafe has just been developed and uploaded to github

Firesafe is a technology to enforce data integrity, and enable complex transactions, on Firebase.

Firesafe compiles hierarchical state machines (HSM) definitions into Firebase security rules. The Firesafe HSM language is expressive, and a super set of the Firebase security language. Adding consistent, concurrent and fail safe protocols to Firebase is now a whole lot simpler (e.g. cross-tree transactions).

Firebase is already the future of databases, offering scalable low latency database-as-a-servie. Firesafe, compliments this amazing technology with an expressive syntax to get the most out of its security model. (Firesafe is not endorsed by Firebase)


In multiplayer apps/games, people cheat. It’s a loss of direct sales, AND the free loaders also diminish the fun for everyone else. Multiplayer games are webscale, which means the old solutions to data integrity and transactions don’t work (e.g. SQL).

Firebase solved one problem of the cost of providing low latency persistent data storage to mobile and desktop games. It’s the first scalable NoSQL hosted solution that didn’t suck. It also provided an unorthodox security and transactions abstraction. Turns out that abstraction has enough purchase to do some really cool things not possible in many NoSQL environments. Unfortunately, properly configuring the security layer is extremely verbose and error prone. Firesafe makes it easy.


The main trick in Machine Learning

I have been irritated that many recent introductions to machine learning/neural networks/whatever that fail to emphasise the most import trick in machine learning. Many internet resources don’t mention it, and even good textbooks often don’t drill it in to the reader the absolute criticality to success the trick is. In a machine learning context, we wish a learning system to generalise. That is, make good predictions on data it has never encounter before, based on what it learnt during from a training set. There is no easy formula to predict the ability of a learning system to generalise, but you can estimate it using held out data. That held out data is labelled but it is not used in training. It is called the validation set.


Make a game month results!


Ahhh, #makgammon, make-a-game-month, is over. I have semi-recovered from the development exhaustion. The single transferable votes have been counted. I have the results!

Before that I would like to say #makgammon was really amazing this year. We had 6 entries, all of which represented a significant contribution in different ways. The spread of games was broad, so there was something for everyone. This was represented in the votes! We asked people to order the games by personal preference (of the games they played). Those orderings were pretty random, indicating that at least someone really liked *every* game. So it just goes to show, you can’t please everyone (so you should never try)! If a game dev did not win a prize, don’t feel bad, because someone in our judging audience did like the concept.

That said, the games that did win did appear higher on average in most people’s score sheet. So congrats to those for winning the mainstream appeal awards. Our prizes were kindly donated by Scirra who make the game development software “Construct 2”. I used that software to build Universal Machine, and I was blown away by how productive I was in 4 weeks. I think its fair to say that even if Universal Machine did not “win”, it was the most complex game, indicating how useful Construct 2 is as a productivity tool. So without further “ado”, the results…

CNC Quickstart

This guide will get you running with the CNC for manual machining operations.

Turn it on!

There are three powered units:

1. computer & monitor

2. the electronics and stepper motors (powered by a computer power supply)

3. The CNC cutter, after turning it on at the wall, you also need to set the direction of the cutter, and turn the speed up. It will not turn the cutter until the speed is set to zero and then increased (so it doesn’t start spinning by accident)


Make a game month


Come join us Edinburgh for the second make-a-game-in-a-month, month. Stretch yourself creatively or technically to create something fun. Demo day is Sat. Nov 30th and open to the public, you can come to play other peoples games even if you’re not authoring. Tell your friends!


Probabalistic Scraping of Plain Text Tables

Recently I have been banging my head trying to import a ton of OCR acquired data expressed in tabular form. I think I have come up with a neat approach using probabilistic reasoning combined with mixed integer programming. The method is pretty robust to all sorts of real world issues. In particular, the method leverages Read more about Probabalistic Scraping of Plain Text Tables[…]

Robotics: Adaptive Control and Vision

Our robot arm is learning to control itself from optical feedback alone! We connected the Lagadic’s visual servoing platform (ViSP), OpenCVs robust homography estimator and University of Edinburgh’s Locally Weighted Projection Regression (LWPR) adaptive control to create a software stack for a cheap USB robot arm toy and a webcam. The hardware cost about £48, and it took us 6 weekends to connect up cutting edge open source research software. Yes! All this software is free! It’s been paid for already. I hope this article will guide people towards making use of these valuable public domain resources. We used an adaptive control scheme so at no point was robotic geometry measured, and instead the software *learnt* how to move the robot from experience alone.


To recap (here and here), our goal is to build high precision robot systems using cheap components, and now we have actually tried a control installation. The existing approach to precision machinery is to spend lots of money on precision steel components and more or less control the machine open loop (without feedback). CNCs are a good example of this where the lead screws are *really* expensive. This approach made sense when we did not have cheap methods of precision feedback, but now we have cheap cameras and cheap computation (thanks smart phones), an alternative method for obtaining precision could be to just to use dodgy mechanics and closed loop control (feedback). Visual feedback is particularly attractive because: its easy to install; it is contactless (so does not affect the motion of the thing you want to control) and it doesn’t wear. With vision you can just slap markers on a mechanical part and off you go (with the appropriate software).