But what about encrypting the image data itself, the pixels, rather than the file as a whole? That can certainly be done, but using a standard encryption algorithm would only make sense with an uncompressed format that stores raw data (like BMP), because otherwise the compression algorithm would likely perform poorly on the encrypted data, and in the case of lossy compression, it would mess it up. Also, depending on the cipher, the encrypted data (ciphertext) might be longer than the original (due to padding), and that would mess up the image or require a change in dimensions.

Since standard encryption algorithms cannot be used, alternative, graphical methods must be used instead. A quick google search for “image encryption” summons lots of results with (apparently) advanced stuff, but here I will discuss a very simple method I invented to “encrypt” images that has a few advantages. I write “encrypt”, in quotes, because this simple method is not cryptographically secure at all, but it may be used to prevent image hotlinking, or to make it hard for someone to steal (especially mass-steal) images from your server or app, etc. In fact, I came up with this technique while working on a game, to protect my artwork, because a few images from other apps of mine had been stolen, and I was a little paranoid back then.

Perhaps the best term to describe this method is scrambling rather than encryption. The idea is very simple: slice the image into small squares and shuffle them, making it look like an unsolved picture puzzle.

The shuffling has to look random, but it must be deterministic and repeatable in order to be easily reversible. This can be achieved with any PRNG (pseudo-random number generator), like the stock ones provided in most platforms (e.g Math.random() in JavaScript). The seed can be thought of as the “encryption” key: each seed number will yield a different block arrangement, and “decryption” is performed by reversing the shuffle process using the same seed.

There’s a problem, though: using stock random() functions will not work across platforms! Each language/platform may use a different algorithm –or different algorithm settings– and this may even happen within a single “platform”, as is the case with JavaScript in browsers (each one has a different Math.random() implementation).

The solution for this is simple: we just write our own PRNG! Since this method is not meant to be cryptographically secure, we can use a simple PRNG like an LCG (linear congruntial generator), which is really easy to code (it just involves a state variable, three carefully chosen constants, one multiplication, one addition, one modulus and one division). My Javascript version is:

var LCG = function(seed) {
  this.state = seed;
};
(function() {
  var m = 4294967296,
      a = 1664525,
      c = 1013904223;

  LCG.prototype.rand = function() {
    this.state = (this.state * a + c) % m;
    return this.state / m;
  }
})();

One last thing to consider is block size: how small should the squares be? It is clear that, the smaller they are, the harder it will be to reconstruct the image without the key, and that is visually intuitive: if we just slice an image into four quadrants, it’s trivial to figure out their order in one glance, as the image is still perfectly clear. If we split each quadrant into another four quadrants, and so on, it quickly becomes much harder to guess what it is you are looking at. The extreme case would be to use 1×1 blocks, effectively shuffling individual pixels into a big block of colourful noise.

As we shrink the blocks, however, two problems may arise. One is that the computational cost increases, but this may not be much of an issue, since it’ll probably be fast enough anyway. The real problem is image compression: the more the image looks like noise, the worse all algorithms will perform. This is because large solid color areas, repeating patterns or smooth transitions (which are features that different algorithms target or expect) will be destroyed.

I planned to use JPEG, and in that specific case, scrambling not only causes poor compression but may also introduce undesirable artifacts or noise in the image, particularly when low quality settings are used. Luckily, knowing a little about the internals of the JPEG algorithm allowed me to solve that problem: since JPEG slices the image in 8×8 squares (or 16×16, or 16×8 in some cases) and processes those independently, minimum interference is achieved by using 8 or 16 as the scrambler’s block size.

The following images illustrate all this. Consider this source image:

Its file size is 285.9 KB. Scrambling it with a block size of 4, we get this one:

which is pretty scrambled indeed (you can’t tell what the original image looked like). But this image has a file size of 566.8 KB. Twice as big! Here we observe the scrambling interefering with compression. On the other hand, if we use a block size of 8 to minimize interference, we get:

which is still pretty unintelligible, but has a file size of 286.2 KB. The size difference is now only just about 300 bytes!

You can try a demo, or check my JavaScript source code on GitHub. I adapted it from a previous ActionScript version I had written for my game.

What’s the difference? Simply put, when writing to an InnoDB table –as part of an insert or an update– the row being written is locked, and all other operations on that row must wait until the write operation finishes. That’s pretty reasonable, and won’t normally cause any problems. On the other hand, when writing to a MyISAM table, the whole table is locked, and all other operations on the whole table must wait until the write operation finishes.

You won’t normally want to perform lots of simultaneous writes to a single row, but a web application under heavy load may easily be performing many concurrent writes (and reads) on a given table. And if that table’s engine is MyISAM… things can get pretty slow.

As an example, a few years ago, while working on a high traffic Facebook app, we started getting complaints about some parts of the app being slow. It turned out that some update queries where taking 10+ seconds to complete! A little investigation revealed the cause: MyISAM’s table locking. By switching to InnoDB, everything started running smoothly (way below 1 second) again.

While googling for different generators and their properties, I recently stumbled upon a website that describes a new algorithm and shows some comparisons between that novel approach and many other popular ones (including LCGs, Mersenne Twister, etc) on different quality metrics (statistical quality, prediction difficulty, time and space efficiency, and more). It’s called PCG, (Permuted Congruential Generator), and, judging by the comparisons shown there, it beats the crap out of most other generators!

What makes it so good? Well, as the site states, the PCG family combines properties not previously seen together in a single algorithm, including: small code size, low memory and CPU usage, powerful features (jump ahead, distance), low predictability, excellent performance in statistical tests, and a permissive open source license (the Apache license).

How does it achieve all that? Its power apparently comes from the combination of an LCG (Linear Congruential Generator) as its state-transition function (the function that dictates how the internal state changes every time a new number is returned) and a new technique called permutation functions on tuples as its output function (the one that turns the internal state into the actual random number returned).

LCG is one of the simplest and fastest algorithms known, and has some desirable features, but used on its own it has horrible statistical performance (among other drawbacks) . It seems that by combining it with a clever output function, though, an excellent generation scheme is achieved.

The website is full of interesting information and explanations, and you can download the full technical paper and the reference C/C++ implementations.

One day, after some code changes, our memcached servers started appearing to behave in a strange way. Intermittently, in spans of several minutes, nothing would work. All caching operations — set, get, add, anything — would fail. The problem had clearly started after that last code push, but the pushed code looked innocent enough, and the strangest part was that caching operations were failing all around, not just within the new code.

Upon further inspection, I found that the problem was not on the servers’ side: all operations worked fine when talking directly to a server via telnet, and some network monitoring revealed that the webservers were not actually talking to the memcached servers at all! So it was a client problem, but a weird one: some innocent caching code in one part of the app — which, by the way, didn’t involve any configuration or setting changes — was somehow breaking cache access in the whole app.

At that point, we had always been using the Memcache extension only, and I suspected there might be a bug in it that was somehow triggered by our latest code. So I decided to try the other extension, Memcached, to avoid this hypothetical bug.

The extensions’ APIs are pretty similar, since they expose the same underlaying functionality, but not identical. That means you can’t just swap new Memcache() with new Memcached() and leave the rest of the code intact. Among the differences are:

• Each has different addServer() parameters
• Memcached has get() and getMulti(), whereas Memcache only has get(), to which you can pass either a single key or an array of keys
• Each has different parameters for set(), add(), etc. because Memcache allows setting flags and Memcached doesn’t

Since Memcache had worked perfectly fine up to that point, and I had no proof of an actual bug in it — nor any guarantees that its quasi-homonymous replacement would be any better — I wasn’t willing to ditch it altogether and rewrite all the code to use the other extension. So we wrote an adapter class to wrap the new extension and use it with our existing code. The idea was to be able to use both interchangeably, switching from one to the other to see whether the bug disappeared when switching to Memcached.

But they didn’t get on well with each other using that simple approach, and new problems came up. Sometimes Memcache would read back a chunk of binary garbage when an object was stored using Memcached, or they would fail to deserialize arrays or numbers when reading objects stored by each other, returning strings instead. I was trying to solve a problem, but was creating new problems instead…

After some struggle, I finally figured things out. The binary garbage was due to compression: by default, Memcached gzips values bigger than 100 bytes. So some objects were compressed and others were not, and that caused Memcache to return some objects as binary garbage. That was easily solved by disabling compression. The serialization issues were due to the fact that both extensions use the flags field (or part of it) to indicate data types, but they use different conventions (storing the type is needed because strings and ints are stored unmodified, but arrays and objects are serialized). The solution was to handle serialization in the adapter (serializing everything except integers) and only pass strings to the extensions’ methods, taking advantage of the fact that both extensions store strings unserialized and with a flags value of zero.

So, with all those fixes in place, we were finally able to use both extensions interchangeably… only to find that the original bug was also present when using Memcached! The same mysterious behaviour! Back to square one…

I had to look somewhere else. What about the latest code? I scrutinized that “innocent” code, the one triggering the bug in the first place, one more time. It performed increment and decrement operations, which we had never used before. That got me thinking, and led me to the memcached protocol documentation, where it said the incr and decr commands only worked on decimal representations of unsigned integers. It turned out we were using them on some values initialized with -1. Bingo! There was clearly a problem there, since those operations were guaranteed to fail on -1. But what did those specific failed operations have to do with system-wide malfunction?

Finally, I found the last piece of the puzzle: it appears that both Memcache and Memcached have checks in place to temporarily block a server that is malfunctioning (around 3 and 15 minutes, respectively). And when the server returns an error message on incr / decr failure, they both seem to misinterpret that as server malfunction, blocking the server for several minutes and causing all operations to fail.

All that trouble caused by a negative number! Well, no. Actually, the real culprit is the unfortunate error handling in Memcached clients, of course.

A few years ago, while working on a high traffic app on the Facebook platform, I ran across a caching bug. All of a sudden, our memcached servers had stopped working — kind of. Stuff could be read, but nothing could be added to the cache: all set operations failed. Actually, it was stranger than that: a few sets did go through.

I had just fixed a broken preload script (which had never actually worked but nobody had ever noticed!) shortly before caching hell broke loose, so that script was my primary suspect. I decided to try the app on a test environment with and without that script, and bingo: without that preload, memcached worked normally, whereas running that preload caused the erratic cache behaviour we were experiencing in production.

So, what’s a preload, anyway? It’s a script that fills the cache with data beforehand, in order to avoid having low performance while the cache is gradually fed objects after each miss. I hadn’t written any preloads, but back then I didn’t stop to think if they were a good or bad idea, either. And it turned out that in many cases they were a bad idea, because they are always a bad idea if done without proper consideration. Ultimately, it boils down to something that has long been known to be… the root of all evil. Yep, I’m talking about premature optimization.

So, when I fixed that preload script (a trivial edit), it… uhm, started working. And it turned out that its job was to select a whole freakin’ table — about 1.5M rows — and load it into memcached. But, hey, that would be, at most, inefficient, right? In the worst case, it might displace useful data replacing it with useless data, but things would fix themselves with usage, right? Wrong! Enter memcached slabs.

For speed and efficiency reasons, Memcached has a custom memory manager, which consists of “slabs”, each of which can be assigned any number of 1MB “pages”, which are in turn split into a number of equally sized “chunks”, each of which may hold an individual object. Slabs hold objects within a specific size range, starting at 88 bytes (I think) and growing exponentially in steps of 1.25x. So for instance there may be a 1280 byte slab, which contains any number of 1MB pages split into many chunks of 1280 bytes, each holding (unless empty) an object whose size will be under 1280 bytes (including key and flags) and above 1024 bytes (the maximum allowed for the previous slab). When you perform a set, Memcached looks at the object size and determines which slab it belongs in, and looks for a free chunk in one of the pages, assigning the slab a new page if needed (i.e. if all its pages are full, or it has no pages at all because no objects of this size were stored before). And once assigned to a slab, a page stays assigned forever and it can’t be reassigned.

That explains the problem. Our preload scripts were executed after we had to resize our cache pool, or restart servers after security updates, or restart the daemon after a mysterious crash, or migrate to a different EC2 instance type, so they acted on an empty cache (no pages assigned). And this script was storing 1.5M objects with sizes in a rather specific range, causing all or most of the pages to be assigned to a few specific slabs, leaving none or too few available for the rest of the slabs. After a short while, the result was that, unless an incoming object’s size happened to match one of the few existing slabs, it was discarded. Regardless of the amount of empty space or stale data, those objects didn’t make it.

So the fix consisted in just removing that preload. The fact that we had never noticed that this particular preload was broken hinted that it wasn’t really necessary after all. And after some investigation and testing, it turned out that in normal operation — that is, caching objects on demand — only around 35k of the 1.5M objects were stored in the cache during the first hour, and there was little to no performance impact during this ramp-up period.

My point is not that preloading itself is inherently bad. Storing those 1.5M objects upfront could have been a good idea in some situation, but it wasn’t our case. My point is: before preloading data — before optimizing anything — make sure it’s necessary. If it is, make sure you’re being selective enough to preload useful data. And keep in mind that careless preloading may not only be useless or inefficient, but possibly harmful, as there may be unforeseen side effects.

But now I finally got down to it. I decided to build a personal website and blog, a place to host my CV, showcase my work, and to share experiences, ideas, and projects (and possibly some non-technical stuff too, to include my other interests).

On the technical side, I’m planning to build a sort of portfolio, but since most of my previous professional work is not displayable (closed source and/or no public demo/live version to point to) I’ll probably take some personal projects, maybe tidy them up a bit, and open source them, hosting them on Github or some similar service. That may take some time, though.

Among the old personal projects I have lying around there is a raycasting game engine with a map editor (old-school pseudo-3D, like Doom or Duke3D), a “modern” 2D game engine which uses OpenGL, and a game audio engine, all written in Java.

Among newer ones there’s a small cloud management system written in PHP, which aims to be like a stripped-down version of Rightscale (allowing to launch servers based on templates and monitor them, etc.), and the latest one is an “interactive data plotter” written in JavaScript, which I developed to help me study bitcoin trading data (as part of an ongoing project of mine).

I also have several ideas for new personal online projects, including trivial ones (band name generator), intermediate ones (drum pattern editor/player), and involved ones (a JavaScript remake of the Mac version of the original Prince of Persia game).

On the non-technical side, I may write about maths (stuff I “discovered”, some proofs, alternative ways of understanding certain things), music (maths behind it, my favourite artists), my favourite works of literature, and philosophical topics (implications of AI, mind-body problem).

I haven’t yet decided if it’ll be best to keep the non-technical stuff in a separate blog, or have everything together. I’m also considering writing some of the non-technical stuff in Spanish. I’ll see when I get to it. If you’re reading this, welcome to my site and I hope you enjoy my posts!