Cupdev

Quantum Hardening Cryptographic Protocols

Thu, 18 Jun 2020 20:27:13 +0200

I recently did some work as a side project for company called Patero that involved creating quantum hardened prototype of one of their products. This post discusses how to secure state-of-the-art cryptographic protocols against attacks from (future) quantum computers. It starts off with an introduction to how crypto protocols in general are constructed, to serve as an introduction for those of us who haven't done much crypto work in the past.

Video auf Deutsch

In addition to my role at Adobe, I also do occasional freelance work as an software engineer focusing on rust/c++/embedded and cryptographic work. One of the clients I usually do work for is Patero, who are working on integrating quantum hardened secure communication modules in mobile devices, IoT and critical infrastructure. One of their projects is an end-to-end encrypted, hardware-based cryptography module for securing mobile calls and communications in general even in the face of a compromised operating system. To further this project I was asked to create a quantum hardened version of this chip for evaluation purposes; first identifying those parts of their cryptographic protocol¹ that would break when quantum computer based attacks become feasible and finding solutions to harden the protocol against such attacks. This blog post discusses the general approach usually taken when hardening modern crypto protocols.

To be clear here, this post is about Quantum Hardening also known as Post Quantum Cryptography; that is the study of how to secure the sort of crypto protocols used in normal machines against cryptanalysis (attacks) using quantum computers. This post is not about Quantum Computing (the study of how to use quantum computers to solve problems) and it is not about Quantum Cryptography (the study of using quantum effects to do cryptography).

The first part of this post is dedicated to revisiting some cryptography fundamentals the basic make up of modern day crypto protocols so even readers less familiar with the subject can take away something from this post. Jump down to Quantum Attacks: Grover's algorithm if you are already familiar with the inner workings of modern cryptographic transport protocols.

Summary: How to quantum harden in a hurry

The availability of practical quantum computers would render all commonly used asymmetric cryptography used today insecure. Since pretty much all modern day cryptographic protocols (like HTTPS, TLS⁵, or SSH⁶) rely on asymmetric crypto as a vital component, these protocols will also become insecure. Symmetric cryptography is also affected but doubling key sizes provides an easy fix. Updating asymmetric crypto to be secure again will require the introduction of entirely new cryptographic primitives.

This is not a large problem now, because quantum computers are currently impractical, but their technology may advance in the coming years or decades enough to make quantum computer based attacks feasible; we should prepare for this point in time soon and some data needs to remain securely encrypted over the next decades, which is why quantum hardened cryptography is starting to become an important subject.

The process to develop post quantum cryptography is currently ongoing. For key exchanges, Classic McElice is a robust choice, but its memory requirements are prohibitive for many platforms. Failing that FrodoKEM is not a terrible choice. Post quantum signature schemes are not as relevant because they are only relevant to thwart online attacks (and for those you need a quantum computer now). Still Sphincs+ is probably not a particularly bad choice, but again the memory requirements are pretty tough to meet.

Any post quantum crypto primitive is currently suspect and should not be used on it's own; there is significant risk the key exchange or signature scheme could be insecure against classical computers, meaning that you would be worse off than with state of the art protocols using such a primitive. Quantum hardened algorithms should only be used together with classical, well analyzed primitives. Employ robust combiners for this purpose. Open Quantum Safe is probably not a bad source of implementations for post quantum primitives.

Post Quantum Crypto should be limited to research and evaluation use cases at the moment except in very special circumstances. Even though redundant constructions using robust combiners are probably safe, the act of updating protocols, changing implementations may introduce implementation errors, side channels, or operational problems which can easily render your crypto system as a whole insecure. Even if extremely stringent measures are taken to publicly vet the resulting protocol and implementation, this risk persists.

What is crypto

On a basic level, the goal of cryptography is the protection of private messages sent over public channels from interception by a third party. In the slide shown above, Alice would like to send a message ("I like your cat ears!") to Berta; since this message is confidential, she would like to cryptographically protect the message.

Other use cases have even more stringent requirements: imagine sending a message requesting a wire transfer via online banking; you would not want an adversary to be able to send a transfer in your name or modify the message in somehow. This is why all crypto protocols should fulfill all three properties: Confidentiality (only recipient can read it), Authentication (only you can send), as well as Data Integrity (message cannot be changed).

Archaic Ciphers: Rotation Cipher

Modern cryptographic ciphers can be quite hard to understand, so in order to illustrate how basic encryption works, we will take a look at some historical ciphers. The first cipher I would like to introduce is the Caesar Cipher; called that because Julius Caesar was one of the users of this cipher. Before we can encrypt some text using this cipher, we first have to assign a number to each letter of our alphabet and choose a key. Then we simply add the key to each character or subtract it to decrypt our cipher text.

Using 4 as our key we get the following conversion table:

space/0 → d/4
a/1 → e/5
b/2 → f/6
...
x/24 → a/1
y/25 → b/2
z/26 → c/2

Our text "i like your cat ears" turns into "mdpmoidbsyvdgexdievw", which cannot be red without decryption.

So how hard is breaking this cipher? Not very hard as it turns out: There is one kind of attack that can always be performed; just try every one of our 27 possible keys until you find one that works. This is called a brute force attack and works well for any cipher whose key space (that is the number of possible keys) is very small; other, more advanced ciphers with a very large key space require more sophisticated sorts of attack.

Archaic Ciphers: Substitution Cipher

Substitution ciphers fit that bill; similar to how a rotation cipher works, one character maps to another random character from our alphabet. Basically instead of rotating our alphabet by a fixed offset, we shuffle it. This makes rotation ciphers a specific case of substitution ciphers.

alphabet: " abcdefghijklmnopqrstuvwxyz"
key:      "jxngzdtlciu wemkavbfsoqprhy"

plaintext:  "i like your cat ears"
ciphertext: "ijwi djhkobjgxsjdxbf"

Shuffling our alphabet yields a much larger key space; $27!$ (that is 27 factorial, $2726...21$) to be precise or roughly $10^28$ (a 10 with 28 zeroes) keys, which is far too many keys for us to try each one. Even one or many computers can not try this many keys, so attacking this cipher will require a more advanced attack: Notice how our cipher text contains the letter J a lot? We can use that and infer that J must map to a particularly common character -- space in our case. By using statistical analysis – that measuring the frequency of each character in our cipher text – we can simply look up which character is this frequent in the English language and break the cipher this way. This works because vowels for instance appear much more frequently than consonants.

Archaic Ciphers: Transposition Cipher

Finally, let's look at a slightly different sort of cipher; while substitution ciphers shuffle our alphabet, a transposition cipher shuffles the actual text turning the cipher text into an anagram of the plain text.

key:        31542
plaintext:  HELLO
ciphertext: EOHLL

Unlike with our previous cipher the problem with this cipher is immediately visible: "eohll" being an anagram of "hello" is pretty obvious. This problem gets worse when the key is used multiple times.

Modern Cipher: AES

Having looked at some archaic examples of ciphers, let's now jump into the 21st century and take a look at an example of a modern cipher. The Advanced Encryption Standard is widely used today (although there are alternatives). there is a good chance this website was transferred to you via an AES encrypted channel. It is a block cipher; meaning the data is chopped into fixed size segments which are separately encrypted. All operations used by the cipher are reversible, so to decrypt the data, all operations are applied in reverse.

Internally, AES uses a Substitution-Permutation-Network. The big round plus with a circle you can see is an XOR operation; this is similar to the rotation cipher (rotation or modular addition in binary is just xor), except that every bit of data is rotated with a single bit from the key. The boxes labeled S represent substitution steps; here four bit long words of data are replaced according to a built in lookup table. Finally the step labeled P is a transposition step; here the results of the substitution are shuffled according to a predefined pattern.

Only the xor step actually combines the data with the key; the substitution and permuatation steps are not really encryption steps per say since their "keys" are known to any attackers. Instead they basically scramble the data, ensuring that each bit in the cipher text is influenced by every bit of plain text and key. Applying these three steps multiple times effectively obscures the relationship between ciphertex, key and plaintext, to the point that statistical methods cannot be used to uncover their relationship. Every bit of the output depends on every bit of the input and every bit of the key; changing just one bit in the input or key should change roughly 50% of the output bits.

Note that we left out a lot of the steps required to really make a secure block cipher; e.g. one problem with the above description is that the same input block and key will produce the same output. This is unacceptable since this would allow attackers to detect when two blocks are the same, there are solutions to this problem but we won't go into detail here.

Now since we have a cipher that is actually secure, we can start to encrypt something, we just need a key! So, Alice does create a key and transmits it to Berta over a secure channel. Our key exchange here is really the most important part; if the key is intercepted all our efforts will be wasted. The adversary will be able to decrypt all past, present, and future messages transmitted using this key. Once the key has been exchanged however, we can start sending data, securely, over untrusted channels.

Message Authentication Codes

There is one – actually there are many problems with the protocol above, however one particularly bad one is that the protocol is not authenticated; meaning, that while an attacker could never correctly encrypt data (they would need the key), they could flip bits in the ciphertext or even send entirely random data and Berta would not automatically notice.

To remedy this problem, a message authentication code is used; the authentication tag generated by this function is a bit of redundant data, added to the message to prevent tampering. This tag can only be generated by someone who knows the symmetric key, so an attacker could only guess at the correct code.

Our protocol is still almost the same as it was before: having exchanged a shared key, Alice can now start encrypting data. She uses the shared key to encrypt our message. Before sending it, she also uses the message authentication code to generate a short authentication tag over the cipher text using the key. Both the cipher text as well as the message authentication code are sent to Berta who checks the auth tag by generating an auth tag of her own and comparing the one sent along with the message. Having successfully established that the message is authentic, she continues to decrypt the data as before.

Key derivation Functions

We now have achieved authenticated encryption; our crypto system is secure as long as the shared key is exchanged securely.

That is all well and good, however secure key transmission is actually not that easy to achieve. Sounds easy enough, but in practice it usually the key consists of around one hundred completely random characters; ciphers need high quality keys to function properly. A short key, or a key that has obvious patterns in it will not do. Hardly any messenger would be able to remember such a random figure; you could write it down, but then you may loose the piece of paper so this is not a good solution either.

Luckily, key derivation functions can at least help us lessen that problem; basically a key derivation function can take a string of random data and bring it into the format of a high quality key, provided of course that the input actually contains enough random information. The output will have the needed length, it will look random² and when used with a password, it should be slow to compute, just to thwart attacks testing lot's of commonly used passwords.

Key derivation functions really provide flexibility; you could choose a long sentence as your key (easier for your messenger to remember) or you could sent multiple keys and combine them using the KDF; as long as one key was not intercepted, the generated key will still be secure.

To integrate this key derivation function in our protocol, Alice chooses a password instead of the key itself and transmits it to Berta. Both parties run the key derivation function to generate the actual key and can now transmit data as they did in the previous two protocols.

Asymmetric Cryptography

Using a key derivation function drastically simplifies key transmission, without actually solving our core problem: We still need to transmit our keys via a secure channel. Creating a secure channel is really hard; the entire point of this exercise is to create a secure channel, so wouldn't it be nice to be able to sidestep this problem? Luckily, there is a solution and that solution is called asymmetric cryptography.

While in symmetric crypto there is just one key that is shared between both our parties, in asymmetric crypto there is two keys; one that can be absolutely public³ and one that is to be kept absolutely private; not even shared with your communication partner. The public key can be used by anyone to encrypt data they would like to transmit to you, while only you could decrypt the data again. There also is a corresponding signature scheme that can be used to proof that you are really the sender of some data; only you can sign a message using the private signature key, but anyone can validate the signature against your public key (notice to how this is somewhat similar in purpose to an authentication tag).

In practice, when sending a message, you would encrypt using the recipients public key and sign using your own secret key and since public key crypto is actually a bit inefficient, you would generate a random shared key, encrypt and authenticate the data using that and finally encrypt and sign the symmetric key using asymmetric cryptography.

Combining asymmetric and symmetric crypto in such a way to gain both – the performance of symmetric cryptography and the flexibility of public key cryptography – is called Hybrid Cryptography. When asymmetric encryption is used to encrypt a shared key for further encryption, this is called a Key Encapsulation Method.

Diffie Hellmann Key Exchange

Most cryptographic connections today are online – meaning it is possible to quickly send messages back and forth. Key encapsulation can be used to exchange the keys for such a connection, however there is a better way: The Diffie Hellmann Key Exchange; this is a function that produces a shared key directly using my own private key and your public key. To execute the key exchange, both parties generate a random key pair and transmit their public key before executing the Diffie Hellman function. That's it. Just two packets to transmit and the secret never actually has to be transmitted over the wire.

Putting it together: Modern day Crypto protocols

Now, using both symmetric and asymmetric cryptography we can build a cryptographic protocol like this:

Alice and Berta generate random Diffie Hellmann key pairs, and exchange their public keys and generate a secret using those keys.
They establish each others authenticity; often this involves using a digital signature scheme, although different ways to achieve this exist.
They use a key derivation function to generate a good key using the secret they just exchanged.
They use the key to encrypt data.
And a message authentication code to make sure the data is not tampered with.

That's it. This is pretty much how state-of-the-art cryptographic protocols work; there is a good chance your browser is used something like this to download this blog post.

Quantum Attacks: Grover's algorithm

There are two ways that quantum computers can attack modern cryptography; the first and less important of the two is grovers algorithm. As we discussed before, there is one attack that works on every cryptographic method: A "brute force" attack which is just a fancy word for trying every possible key until you find one that works. In computer science this is known as a "linear search" – the most basic search algorithm. Given a million possible keys it would take five hundred thousand tries on average to find the correct one; the formula is just $^N/_2$ where $N$ is the number of keys.

Grover's algorithm is the generic quantum search; this takes (don't ask me how) just a thousand tries for a million possible keys: While the generic classical search takes $^N/_2$ tries, quantum search just takes $sqrt(N)$ attempts on average, so it is much faster, or one thousand tries given a million possible keys. Luckily, we can just double our key size from six decimal places to twelve (or from 128 bits to 256 bits for actual cryptographic algorithms) to achieve the same security level as before.

So while Grover's algorithm can be used against any cryptographic algorithm, there is a relatively easy fix.

Quantum Attacks: Shor's algorithm

Shor's algorithm presents a bigger problem; pretty much all asymmetric cryptography⁴ used today is either based on the Discrete Logarithm Problem or the difficulty of factorizing integers. Both problems can be solved very efficiently using Shor's algorithm.

This means: Our handshake is broken; it is entirely insecure against attacks from quantum computers. This is why it is commonly said that quantum computers break all modern crypto; the symmetric part is fine, but if the handshake is insecure the symmetric key will be known so the protocol is insecure as a whole

Quantum Attacks: Are they feasible?

Now, this sounds really bad until you remember that practical quantum computers simply do not exist at the moment. They are really expensive, need to be cooled using cryogenics and just have on the order of 50 to 60 quantum bits; you would need thousands of quantum bits to even attempt breaking modern crypto. Attacking modern crypto with ENIAC (a very early computer) seems more feasible than using quantum computer at this time.

However, there is a good chance practical quantum computers will arrive in the next decades, so it is good to be prepared. Besides, in addition to all the cat videos, there is some data stored and transmitted now that we would like to remain secure in thirty to forty years; think financial or medical records. An attacker could simply commit all data they can get their hands on to cold storage and wait until quantum computers are advanced enough to decrypt it; a concept that is called "store now break later". To thwart this, we should really start to use post quantum crypto as soon as possible.

Quantum Hardening: Currently in early alpha

To this end, the National Institute of Standards and Technology – the same organization that standardized SHA, AES, and RSA – has organized a competition to find suitable post quantum crypto algorithms. The competition started 2017 and is expected to take half a decade more. In the beginning around 90 candidates where submitted, but a lot of them where discovered to be insecure in the first couple of hours after submission. Some similar ones have also been merged, but 26 are left and are now candidates in round two of the competition; you can follow the progress here.

The new primitives include signature schemes as well as key encapsulation methods and a single example SIKE of a replacement for Diffie Hellmann style key exchanges. They are based on problems like Learning With Errors, Elliptic Curve Isogeny, or the difficulty of decoding Error Correction Codes (as if I knew what the first two meant).

Experimental implementations of NewHope, NTRU, and SIKE where created for TLS⁵ and used in the Chrome browser and some servers with good results. BSI (the German equivalent of the American NIST) recommends using FrodoKEM or ClassicMcElice if you so choose to use quantum hardened key exchanges. Post Quantum Signature Schemes haven't received as much attention because they are not quite as relevant for protection from "save now decrypt later" style attacks. They protect from man in the middle attacks which have to be performed live – just at the time the connection is being created.

Robust Combiners

Post quantum crypto primitives are new and shiny; with crypto as with databases this is a bad thing. It means we can not really trust these algorithms' security as there is still a good chance some of these will turn out to be insecure in future cryptanalytic works. If we were to rely on them alone, our cryptographic protocol may not only be susceptible to quantum attacks but to classical attacks as well. This would be a total failure.

Optimally we would find some way to combine state-of-the-art asymmetric cryptographic primitives with post quantum ones, similar to how we combined symmetric and asymmetric cryptography to get the best from both worlds. There is a way and the study of how to do this securely is called "Robust Combiners"; enter the term in the search engine of your choice and you will find that there are a lot of constructions that are unsafe – proven to be bad. One of the few things that has been show to work as a robust combiner is the concatenation combiner: Simply perform both key exchanges; and concatenate the secrets they produce. You will end up with a secret twice the size, which is not really a problem because we just use it as the input for our key derivation function anyways. Combining signature schemes is trivial because we can just use both; test each signature and if one of the tests fails, consider the signature to be invalid.

Putting it together: The make up of a quantum hardened crypto protocol

Now, finally, we can actually come up with a quantum hardened cryptographic protocol: Perform two key exchanges – one post quantum key exchange and one legacy key exchange; concat their secrets. Perform two authentication steps; again, one quantum hardened and one legacy authentication. Use your key derivation function to format the combined secrets and start encrypting data. That's it.

Choosing primitives and implementations

Should you ever be given the challenge to quantum harden a protocol, you will find yourself trying to decide which cipher to use. Unfortunately, there is no straight answer to this question yet; the jury is still out on which post quantum primitives will turn out to be secure and which ones have vital flaws. If you had to choose, Classic McEleice is based on a crypto system invented in 1979 that has been analyzed pretty extensively. It is probably secure, but its key sizes are huge, too huge to fit into the memory of the chip I was programming. If memory is not a giant constraint for your application, use Classic McEleice. FrodoKEM is not as well analyzed, but probably not a terrible choice either and it's keys are still large, but not prohibitively so at least for the chip I was using. Sphincs+ is probably not a terribly choice for a post quantum signature algorithm, but it's signatures are around 40Kb large, so probably too large for a lot of platforms especially when trying to establish a chain of trust.

As for implementations, libsodium is what you should use in most applications for pre quantum algorithms and symmetric crypto. It provides a very small, secure set of cryptographic functions and follows usable security API design practices which make it comparatively hard to create unsafe constructions. There are alternatives implementing a wider array of cryptographic primitives, but often these include unsafe primitives or ones which are only secure when used in very specific ways. A limited set of choices can be a good thing.

Finally, there are a couple of libraries that implement post quantum crypto primitives; these implementations are of course not as tried and true as libsodium, but they do follow similar API design guides. There is Open Quantum Safe which is a good default choice as a source of post quantum crypto algorithms. PQClean provides a lot of the implementations used in OQS; I ended up using this one just because I had an easier time compiling it for my platform. mupq implements variants of post quantum primitives optimized for Arm M4; I could have used this, I ended up not using this simply because speed wasn't an issue at all for my platform, memory was the primary constraint.

[1]: https://en.wikipedia.org/wiki/Cryptographic_protocol; this is the sort of technology used to encrypt live connections; TLS – the protocoll your browser is using to fetch this blog post – is an example of such a protocol.

[2]: On average – there must be a 50% probability for each bit to be one or zero.

[3]: Download my public key from my web server https://cupdev.net/keys/7EA37614E956818EBFEA76526739B1556DBB745A.asc or from the key servers: gpg --search-key 7EA37614E956818EBFEA76526739B1556DBB745A

[4]: This includes RSA, DSA, Diffie-Hellman and all Elliptic Curve Cryptography

[5]: Transport Layer Security, your browser is using this to encrypt the connection with the blog's HTTP server.

[6]: Secure Shell; the most widely used protocol for administrating servers on the internet.

How to Analyze Assembly Code to Guide Optimization Strategies

Fri, 29 May 2020 16:09:40 +0200

Bin2c is a tool for embedding files in C89 compliant source code. The advantages and disadvantages of bin2c over existing methods of embedding files are discussed at length in the below project’s ReadMe. In short: bin2c’s goal is to quickly generate C code that is fast to compile and very compact. As such, using it is more portable but slower than directly embedding the file using ld for example. Compared to xxd -i, its output is much more compact, quicker to generate, and quicker to compile.

How bin2c achieves this level of efficiency is discussed in this post!

Choosing the output

Before we are able to implement the converter, we need to decide which format we are going to produce. xxd -i encodes binary data as an array literal containing hexadecimal integers:

$ echo -n 'Hello World my big floof' | xxd -i 0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x57, 0x6f, 0x72, 0x6c, 0x64, 0x20, 0x6d, 0x79, 0x20, 0x62, 0x69, 0x67, 0x20, 0x66, 0x6c, 0x6f, 0x6f, 0x66

This format is quite readable, but it also has a very high overhead needing 6.167 characters to encode one byte of input data. By getting rid of the extra spaces and using decimal numbers instead of hex (getting rid of 0x) we can reduce space requirement down to 3.58 chars/byte. The resulting format is less readable than before:

72,101,108,108,111,32,87,111,114,108,100,32, 109,121,32,98,105,103,32,102,108,111,111,102

We could also consider storing the data as a hex encoded array of eight byte words (uint64_t):

0x48656c6c6f20576f,0x726c64206d792062,0x696720666c6f6f66

This format is even more compact (at 2.4 chars/byte), however it is also really fragile: The memory area would have to be cast to a byte array and the encoding of each hex number depends on the endianess of the target platform.

Bin2c follows an entirely different approach: Instead of encoding the data as a byte array, we encode it as a string! This way, text files can be included almost verbatim; just a few characters like " or \n have to be escaped. Unicode characters and binary can be encoded using escape sequences.

$ echo -n 'Hello World my big floof' | build/bin2c
hHello World my big floof
$ head -c 20 </dev/urandom | build/bin2c
\241dT\000\314\022\241\351\355\377#\352\244\216\334\336\004\002\350\31

This encoding is extremely compact for text (requiring 1.2 chars to encode one byte) and still decently small binary (3.1 chars/byte). Turns out compilers are also 20x-30x faster at parsing this format.

Our first implementation

As usual, when approaching optimization problems, we should focus on writing an obvious, easy-to-read implementation of our solution. This way we will have a baseline to compare our code against. Here it is:

#include <stdio.h>
#include <ctype.h>

void bin2c() {
  for (int inc = getchar(); inc != EOF; inc = getchar()) {
    // Detect characters that need a special escape sequence
    switch ((char)inc) {
      case '\a': fputs("\\a", stdout);     continue;
      case '\b': fputs("\\b", stdout);     continue;
      case '\t': fputs("\\t", stdout);     continue;
      case '\n': fputs("\\n\\\n", stdout); continue;
      case '\v': fputs("\\v", stdout);     continue;
      case '\f': fputs("\\f", stdout);     continue;
      case '\r': fputs("\\r", stdout);     continue;
      case '\\': fputs("\\\\", stdout);    continue;
      case '"':  fputs("\\\"", stdout);    continue;
      default: {} // pass
    }

// Handle printable characters (excluding $, @ and ?)
    if (isprint(inc) && inc != '$' && inc != '@' && inc != '?') {
      putchar(inc);

// Fall back to octal encoding
    } else {
      printf("\\%.3o", inc);
    }
  }
}

This very first version already outperforms XXD, albeit by a relatively small factor of 1.6; I am not entirely sure why this is the case. The hot loop for encoding C source files in XXD seems to call fprintf in a loop while using some complicated logic to choose the format string and supply the delimiter string. This is probably one of the major reasons for xxds slowness; clever code does not tend to perform well. Coders have a hard time reading — and thus optimizing — clever code and sometimes, so do compilers.

It probably also helps that bin2c can skip a call to fprintf in many cases and just use puts or putc directly.

Inlining

Let us now take a look at the assembly produced when compiling the code above with optimization. Not many surprises there: getchar() turns into a call _IO_getc, our puts() calls in the switch statement become fwrite(), putchar turns into _IO_putc, isprint() turns into__ctype_b_loc , and the printf() call stays the same in the assembly.

That is a lot of call instructions — two or three per character encoded and the functions are pretty complex. The stream writing functions need to do error handling, isprint takes the locale into account, printf needs to parse the format string. Optimizing out the bin2c function is going to involve creating alternate implementations of the called functions that are both simpler and can be inlined.

We will have to use manual buffering coupled with fread and fwrite and use setvbuf to disable the built in stream buffering. This way our hot loop just has to write from one buffer and into another, deferring all the complexities of writing to C streams to whenever the buffers are flushed.

isprint() can be replaced with chr >= ' ' && chr <= '~' , if we handle any special cases in the switch statement and octal encoding is really easy to implement because we can get the ascii digit for a numerical value by adding '0' or 48.

#include <ctype.h>
#include <stdint.h>
#include <stdlib.h>
#include <assert.h>
#include <string.h>

// Not using the normal strcpy here because the `call` instruction emitted would be expensive…
inline size_t b2c_strcpy(char *from, char *to) {
    size_t r = strlen(from);
    strcpy(to, from);
    return r;
}

inline size_t bin2c_single(uint8_t chr, char *out) {
  switch ((char)chr) {
    case '\a': return b2c_strcpy("\\a", out);
    case '\b': return b2c_strcpy("\\b", out);
    case '\t': return b2c_strcpy("\\t", out);
    case '\n': return b2c_strcpy("\\n\\\n", out);
    case '\v': return b2c_strcpy("\\v", out);
    case '\f': return b2c_strcpy("\\f", out);
    case '\r': return b2c_strcpy("\\r", out);
    case '\\': return b2c_strcpy("\\\\", out);
    case '"':  return b2c_strcpy("\\\"", out);
    case '$':
    case '@':
    case '?':
      goto octal;
  }

// isprint, but inline
  if (chr >= ' ' && chr <= '~') {
    out[0] = chr;
    return 1;
  }

octal:
  // Octal digits are three bit long, so we can just a bit mask 0b111
  // and bit shifts to extract each digit
  out[0] = '\\'; // octal encode
  out[1] = (chr >> 6 & 7) + '0';
  out[2] = (chr >> 3 & 7) + '0';
  out[3] = (chr >> 0 & 7) + '0';
  return 4;
}

void bin2c(uint8_t **in, uint8_t *in_end, char **out, char *out_end) {
  // We just work around the 4 byte buffer limitation in the enclosing
  // read/process/write loop…not an issue for our specific application
  assert(out_end-*out >= 4);
  // (hot loop) While data in inbuff & outbuf has 4 free slots
  // (bin2c needs four free slots)
  for (; *in < in_end && out_end-*out >= 4; (*in)++)
    *out += bin2c_single(**in, *out);
}

These changes just gave us an 8x improvement in performance. Take a look at the assembly again; looks less complex, not a single call instruction.

Lookup Table

Having eliminated call instructions is great, as doing so improves cache coherence and means less work to store parameters (fulfilling the calling convention).

In fact, all sorts of jumps and branch instructions are somewhat expensive. Again, this is because the cache coherence will be slightly worse, but also because of the impact on CPU pipelining. Optimally we should not use any conditional execution (if statements, switch statements or ternary operators) in our hot loop for optimum throughput.

Luckily, there is a way to achieve that in our case: Our encoder is just a mapping from one specific input byte to between one and four output bytes. This is not a lot of data so we can simply use a lookup table! Four bytes per value, zero padded — we need 1KB of lookup table; that is small enough to easily fit into most CPU caches!

That should be pretty easy: Use a lookup table with 256 32bit words (uint32_t), lookup the value for the input char, and copy the output value to the output buffer. Unfortunately, because our lookup table uses variable sizes, we still need an efficient way to determine the length of our output sequence. We could, for instance, use strnlen to determine the length (which results in a call instruction being emitted). Failing that we could use a similar pattern as before and implement our own, inlinable strnlen, but that results in a few conditional jumps.

After playing around with various implementations, even writing my own branch free version of strnlen (based on the Bit Scan Reverse instruction), I decided to simply store the string length inside some bits of the lookup table, which happened to be unused.

Then I just made sure that clearing those bits and copying the data to the output was done in four byte blocks (uint32), instead of byte for byte. Even if the output sequence is short, we still copy all four bytes from the lookup table. It might seem like copying fewer bytes would be less work, but that would require us to use conditional execution which is simply not worth it if the payoff is copying just one byte instead of three or four.

#include <stdint.h>
#include <stdlib.h>
#include <assert.h>
#include <string.h>

static inline size_t bin2c_single(uint8_t chr, char *out) {
  extern const char bin2c_lookup_table_[];

  // NOTE: The length is in the two most significant bits of
  // the last character of the output. The length in there is
  // 0..3, our actual value is 1..4 so we need to add 1.
  // In order to store this mask in an endianess independent way
  // we store it a byte array; in order to apply it efficiently
  // we cast to an uint32_t so we can perform a one 4 byte instruction
  // instead of multiple single byte instructions when copying into
  // the output buffer.

  // Lookup table char -> escaped for C.
  // Each code is between one and 4 bytes and padded by zeroes.
  const uint8_t *slot8 = ((uint8_t*)bin2c_lookup_table_) + chr*4;

  // The bit mask for erasing the length information; stored as a byte
  // array instead of a uint32_t literal to avoid endianess issues
  uint8_t mask8[] = { 0xff, 0xff, 0xff, 0x3f };

  // Copy the data from the lookup table and erase length info
  // This really is just `out = slot & mask;`.
  // Both casting to a packed struct and using memcpy are ways to express
  // unaligned 32 bit pointer access; this is faster than doing the copying
  // byte by byte on x86_64
#ifdef __GNUC__ // GCC/Clang optimized version
  struct __attribute__((__packed__)) uint32_noalign {
    uint32_t v;
  };
  ((struct uint32_noalign*)out)->v = ((struct uint32_noalign*)slot8)->v & ((struct uint32_noalign*)mask8)->v;
#else 
  uint32_t mask32, slot32;
  memcpy(&mask32, mask8, 4);
  memcpy(&slot32, slot8, 4);
  slot32 &= mask32;
  memcpy(out, &slot32, 4);
#endif

// Extract length info from lookup
  return ((slot8[3] & 0xc0) >> 6) + 1;
}

void bin2c(const uint8_t **in, const uint8_t *in_end, char **out, const char *out_end) {
  // (hot loop) While data in inbuff & outbuf has 4 free slots
  // (bin2c needs four free slots)
  for (; *in < in_end && out_end-*out >= 4; (*in)++)
    *out += bin2c_single(**in, *out);

}

Using the lookup table strategy yielded another 2.3x speedup. In the assembly we can see why: Just two jump statements left in the hot loop, the code is also much more compact than before and all code in the hot loop is executed every time, so there is no opportunity for the branch predictor to generate incorrect predictions.

The original implementation is still included, by the way, since we need a way to generate the lookup table in the first place.

Conclusions

The optimization strategies discussed in this post demonstrate how reading assembly can give vital clues when trying to improve your code’s efficiency and that even basic operations like calls and jump instructions can slow down your code when used inside the hot portion of your code. We also discussed a common optimization strategy: Using a lookup table.

The greatest gains though can often be reaped from taking a step back and reexamining your approach; changing the format from hex numbers to escaped string not only improved output size, sped up compilation, and laid the foundation for later performance improvements.

Benchmarks

These benchmarks show the optimization progress; taken on a i7–9700K CPU @ 3.60GHz. This contains some more development steps, like fixing unaligned access to uint32_t (which is undefined behavior; fixing this resulted in a performance hit).

xxd                             13.9282  Mb/s
bin2c-0001_initial              24.5181  Mb/s
bin2c-0003_inline_octal         129.244  Mb/s
bin2c-0002_manual_buffer        129.292  Mb/s
bin2c-0005_lookup               165.952  Mb/s
bin2c-0004_inline_isprint       180.759  Mb/s
bin2c-0006_branchless_length    218.387  Mb/s
bin2c-0007_lookup_length        435.427  Mb/s
bin2c-0008_32bit_lookup_access  593.669  Mb/s
bin2c-0009_fix_unaligned        437.677  Mb/s
bin2c-000a_packed_unaligned     553.805  Mb/s
bin2c-000b_discard_b2c_memcpy   555.226  Mb/s

Filtering Duplicates on the Command Line: 30x Faster than sort|uniq

Tue, 25 Feb 2020 00:00:00 +0100

sort | uniq sorts data given to it via stdin and then removes any duplicates. Personally, I use this quite often to create a ranking of something. For example, looking below, you could create a ranking of the words I use most often in this blog:

$ curl "[https://cupdev.net/blog/](https://cupdev.net/blog/)" | html2text2 | sed 's@[^a-zA-Z_0-9]@\n@g' | grep -v '^\s*$' | sort | uniq -c | sort -n
     34 s
     39 com
     39 tag
     40 it
     40 that
     48 you
     49 is
     52 in
     53 search
     82 of
     88 I
     94 and
    112 a
    114 to
    180 the

Unsurprisingly, I use ‘the,’ ‘to,’ and ‘a,’ quite a lot.

The first two commands download the blog, the third removes all special characters and splits lines into words, grep gets rid of any empty lines left over, sort|uniq -c counts duplicates, and the final sort -n ranks them.

You might notice, that for the purpose above, sorting is not really required. We just need to count duplicates. A more speedy way to get this job done might be to use a hash table.

That’s why I created huniq in order to count/remove duplicates by using a hash table. huniq implements two modes; it removes duplicates and counts them. In this blog post, we’ll look at how to implement and optimize the first mode.

Writing idiomatic Rust code

My initial implementation was very simple; I used clap to parse command line arguments, anyhow to handle errors (actually I used failure, but that was later replaced), and wrote the simplest implementation I could think of for uniq (source).

fn uniq_cmd(delim: u8) -> Result<()> {
    let mut out = BufWriter::new(stdout());
    let mut set = HashSet::<Vec<u8>>::new();

    for line in BufReader::new(stdin()).split(delim) {
        let line = line?;
        if set.insert(line.clone()) {
            out.write(&line)?;
            out.write(slice::from_ref(&delim))?;
        }
    }

    Ok(())
}

Each line is read using BufReader/BufWriter. You then insert it into the hash set. If it succeeds (because the value wasn’t in the hash set before), the line is also printed to stdout.

This design already includes one optimization, albeit the reason for including it is that it makes the code actually easier. By using BufReader/BufWriter, instead of the plain stdin() and stdout(), the number of system calls can be reduced. For more on why this improves performance, keep reading.

Allocation, system calls, and context switches

Two inefficiencies are immediately visible: BufReader::split allocates a new Vec<8> for each element read and clone creates another copy, which also needs to be allocated.

Memory allocations can be quite slow; they need a bit of time to find an available memory location and, if there is none, the allocator will fall back to using a system call to request more memory from the system.

A system call requires at least two context switches: one into the kernel and one back.

Context switch on the other hand means switching to another process, thread, or into the kernel (the core of the operating system). All the data currently in the cpu registers need to be written back into RAM and the CPU caches need to be cleared. The page table (hardware accelerated supported memory management, see Memory Management Unit) needs to be cleared. New data is loaded into the CPU from the new thread of execution.

To sum it up: Accessing RAM is really slow, and in order to do a system call/context switch/some memory allocations, a lot of new data needs to be loaded from the RAM. The CPU caches are used avoid needing to access RAM directly, so clearing them is also bad for performance.

To get rid of at least one of those allocations, a trick can be used: you can store the hash of the line only instead of the full value. This way, we can avoid calling clone() and thus avoid the extra allocation andsaving some memory. We now just need 8 bytes for each string, while Vec would require at least 16 (one for the size and one for the location of the allocated string in memory, in addition to the actual data; rusts implementation actually needs 24 as it also stores the capacity).

Here is the code: We now just call set.insert(hash(&line)) to insert the hash; this also let’s us get rid of the clone() call since hash() just needs a reference instead of moving the whole string.

for line in BufReader::new(stdin()).split(delim) {
    let line = line?;
    if set.insert(hash(&line)) {
        out.write(&line)?;
        out.write(slice::from_ref(&delim))?;
    }
}

Due to the birthday paradoxon, doing this is really only safe for about 232 elements when using a 64-bit hash. We're not too worried about this, because 232 is a large number, but if you expect more than a billion unique elements, you should probably use a 128-bit has function, which is good for 2**64 elements.

Benchmarking

At this point in the process, I decided huniq was ready for a first release. I tested the code (turns out it worked) with a simple shell script and created some benchmarks.

Benchmarking is tough. For huniq, I decided I didn’t really need something sophisticated and simply used /usr/bin/time -v on the command line to measure execution time and memory usage.

$ /usr/bin/time -v echo

Command being timed: "echo"
        User time (seconds): 0.00
        System time (seconds): 0.00
        Percent of CPU this job got: 100%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1512
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 7
        Minor (reclaiming a frame) page faults: 73
        Voluntary context switches: 2
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 56
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

This already gives a lot of insight; most interesting is the “Elapsed (wall clock) time” (how long the execution took), and the “Maximum resident set size” (how much memory it used). There are other interesting values here, too— for instance “System time” in relation to “Elapsed time” can help us understand how much time of our program spends in system calls.

I created a little script (see here) for benchmarking so I wouldn’t always have to run the benchmarks manually. Later I would keep the binaries from different optimization steps around and add them to the benchmark script, so I could see whether my optimization attempts are working. At this point, the benchmarks looked like this:

implemetation  seconds  memory/kb
rust             10.26      29548
cpp              18.87      26128
shell           151.40      10060

Pretty good, right? This simple rust implementation is already beating sort|uniq and my previous c++ implementation!

Using idiomatic rust code already beats the old C++ implementation. The reason for this likely is the choice of hash table (although I did not verify this). It just goes to show you how well the rust standard library is crafted!

Stdin::lock()/Stdout::lock() As suggested in these responses (thanks awesome Rust Reditters), I decided to use lock() on stdout and stdin. This method lets you avoid the overhead of thread synchronization when using these functions, which is acceptable in our case, since we’re accessing the method from just one thread. Check out the code. fn uniq_cmd(delim: u8) -> Result<()> { let out = stdout(); let mut out = BufWriter::new(out.lock()); let mut set = HashSet::::new(); for line in BufReader::new(stdin().lock()).split(delim) { let line = line?; if set.insert(hash(&line)) { out.write(&line)?; out.write(slice::from_ref(&delim))?; } }

    Ok(())
}

Using lock() did not have a measurable effect on performance, but I still decided to keep it because it made sense conceptually and was a very simple change.

Getting rid of redundant work: Extra allocation, extra hash

Just storing the hash, instead of the full value, was a good idea, but that also meant we were now calculating two hashes. First, we manually hashed the line, but then the hash table would execute run hash function again! We solved that problem by creating a custom hasher that just returns the data it received. The drawback here is that we needed to create custom implementations of Hasher (my source) and of BuildHasher (my source); this was worth it because it resulted in a speedup of 10-20 percent.

struct IdentityHasher {
    off: u8,
    buf: [u8; 8],
}

impl Hasher for IdentityHasher {
    fn write(&mut self, bytes: &[u8]) {
        self.off += (&mut self.buf[self.off as usize..])
            .write(bytes).unwrap_or(0) as u8;
    }

    fn finish(&self) -> u64 {
        u64::from_ne_bytes(self.buf)
    }
}

#[derive(Default)]
struct BuildIdentityHasher {}

impl BuildHasher for BuildIdentityHasher {
    type Hasher = IdentityHasher;

    fn build_hasher(&self) -> Self::Hasher {
        IdentityHasher { off: 0, buf: [0; 8] }
    }
}

fn hash<T: std::hash::Hash>(v: &T) -> u64 {
    let mut s = DefaultHasher::new();
    v.hash(&mut s);
    s.finish()
}

In addition, our split() invocation was still allocating on the heap. To solve this, you can use BufRead::read_until to remedy that which can reuse a Vec. Again, our code was becoming more and more complicated; using read_until() requires you to declare the Vec and handle an edge case when there is no delimiter before eof; but then again, we're another 10- 20 percent faster.

Now our code looks like this:

fn uniq_cmd(delim: u8) -> Result<()> {
    let out = stdout();
    let inp = stdin();
    let mut out = BufWriter::new(out.lock());
    let mut inp = BufReader::new(inp.lock());
    let mut set = HashSet::<u64, BuildIdentityHasher>::default();
    let mut line = Vec::<u8>::new();
    while inp.read_until(delim, &mut line)? > 0 {

        if *line.last().unwrap() == delim {
            line.pop();
        }

        if set.insert(hash(&line)) {
            out.write(&line)?;
            out.write(slice::from_ref(&delim))?;
        }

        line.clear();
    }

    Ok(())
}

Going zero-copy

Keen eyes will spot that our code is still doing unnecessary work — copying the current line from BufRead into the line buffer. I haven’t found anything in the standard library that lets me avoid the copy, so I had to resort to writing my own split function.

The split_read_zerocopy (source, bit too long to post inline but with lots of comments) function takes a delimiter (u8), a Read stream (anything that implements Read), and calls a callback function whenever a new token is found. Now instead of copying the data found into a user provided buffer, the function emits slices of the internal buffer, which can be resized when we need to fit in very long lines.

This zero-copy optimization was really high effort; for many projects, you should consider whether it is even worth it. Huniq is a small project with a very specific use case, and I do think a boost of 15-25 percent was worth it.

Choosing a hash function: ahash, fxhash, or xxh3

The default hash function in rust hash tables is currently siphash, which is a good choice against hash collision attacks. Such an attack could allow an attacker to slow down huniq massively and even hide elements from the input, since it doesn’t involve storing the original value.

Huniq isn’t really advertised as particularly safe for untrusted inputs, but keeping basic mitigations in place against collision attacks is probably a good idea. At the very least, I wanted to use a secret key/seed to hash the data, which meant we had to find a hash function that has some basic support for a secret keys and is also faster than siphash.

I tried various ways of hashing including ahash, which was 20 percent faster than the default hasher, fxhash, which sped up the process by another 20 percent. Fxhash isn’t really optimized though; it is not optimized for long values and — more importantly — is not very collision resistant.

After cleaning up the code using ahash and trying two ways of manually applying a random seed to fxhash, I still wasn’t really satisfied. And honestly, I didn’t really trust my own way of seeding fxhash (getting cryptographic problems right is hard).

Enter: XXH3. This is a relatively new, very quick hash function. It is based on xxhash (which is very fast, especially for long inputs) but includes additional optimizations for short inputs. It provides the XXH3_64bits_withSecret function, which pretty much exactly covered our use case and should be relatively safe against collision attacks. The fact that this is a single function call instead of byte oriented hashing (that is a combination of Hasher::write() and Hasher::finish()) also probably helps.

In the end using XXH3 yielded another 15 percent speedup over using fxhash, while being optimized for long values too and providing some level of protection against collision attacks. This was the optimization with the most effort yet; there is no crate exposing the XXH3 hash function, so huniq includes additional code to compile the xxhash repository and create bindings for XXH3 on the fly.

Link Time Optimization, alternative Hash Tables

In addition to the optimizations outlined above, I also tried to use different hash tables, activating Link Time Optimization.

Both attempts ended up decreasing the efficiency of huniq. For the hash table, it seems the standard library is already using one of the most efficient hash tables out there. This seems to be an advantage of rust’s faster paced approach over the design-by-committee approach used in C++. std::unordered_map is famously slow, because the standard imposes very specific guarantees in regards to pointer stability and memory layout, which makes it very hard to write a fast hash table.

As to why Link Time Optimization slowed down the code, I don’t know; I even tried setting codegen-units=1 as suggested here, but to no avail.

Conclusion

By porting huniq to rust and applying a number of standard optimization techniques I was able to improve the performance of filtering duplicates on the command line by a factor of 30x, compared to using the standard sort|uniq technique. With this, I was able to achieve an improvement of 4x-5x compared to huniq1, which was written in C++.

In this benchmark below you can see the improvements achieved in each step (full benchmark).

repetitions  implemetation  seconds  memory/kb
50             shell         374.30      11180
50             huniq1         54.48      26136
50             datamash       95.37      10896
50             awk            62.73     322052
50            0original       29.91      29484
50            1anyhow         29.65      29564
50            2iolock         30.76      29664
50            3noalloc        26.10      29544
50            4singlehash     23.80      29632
50            5ahash          21.35      29612
50            6fxhash         17.96      29556
50            7copyelision    15.03      29548
50            8cleanup        20.13      29668
50            9seeded-fxhas   17.17      29664
50           10cache-seed     14.42      29608
50           11xxh3           12.46      29548

No single step had the largest effect; most iterations improved the efficiency by 10-20 percent, but a lot of steps involved improving the hash function, so using XXH3 is probably the most important optimization that was applied to huniq. The least impactful optimization was using Stdin/Stdout::lock(), yielding no measurable performance benefit.

The most unusual kind of optimization used in huniq is probably the zero copy Stdin splitter, which yielded a moderate performance improvement.

One thing still irritates me though; using LTO should not necessarily yield great performance improvements, but it should not slow down the code as it did with huniq. I am still unsure how that happened; possibly a regression in rustc? Maybe a reader can shed light on this mystery…

Why implementing equals() in javascript is hard and how use Symbols to do it correctly!

Sun, 19 Jan 2020 20:02:19 +0100

A while back, we released ferrum.js, a relatively small javascript library that “brings features from rust to JavaScript” in a way that is supposed to feel native to js developers. When we first started to write the library we began by trying to answer two key questions; the first of which is a bit provocative I admit: Why is there no good library for working with es6 iterators that integrates well with javascript syntax?

We decided to tackle the problem head on: by writing just such a library!

The second question we had to address, turned out to be more complex; “How can we implement a function like equals() or hash() correctly in javascript?" Here’s how we tackled that one, too.

How to (badly) implement equals

It seems like there are a lot of implementations of these kinds of functions; lodash has one for instance. For hash() there is object-hash, which I contributed to myself a while ago! In principle, implementing a function like equals is relatively easy:

const assert = require('assert');

const equals = (a, b) => {
  if (a.constructor !== b.constructor) {
    return false;  // Implement for a variety of simple types

  } else if (a.constructor === Date) {
    return a.toString() === b.toString();  // Implement for data structures using recursion

  } else if (a.constructor === Array) {
    if (a.length !== b.length)
      return false;    for (let idx = 0; idx < a.length; idx++)
      if (!equals(a[idx], b[idx]))
        return false

    return true;
  } else if (a.constructor === Object) {
    if (Object.keys(a).length !== Object.keys(b).length)
      return false;    for (const [key, val] of Object.entries(a))

    if (!equals(val, b[key]))
      return false;

    return true;
  } else {
    // Provide a fallback for any other types
    return a === b;
  }
};

// Test our equals implementation!
const d = new Date();
const e = new Date(d.toString())

assert(equals(
  {
    foo: 42,
    baz: d,
    bar: [1, "foo", 3],
  },
  {
    foo: 42,
    bar: [1, "foo", 3],
    baz: e,
  }));

All these implementations have a drawback though: They can’t support any types they don’t know about; like this one for instance:

class Rational {
  constructor(p, q) {
    this.p = p;
    this.q = q;
  }...
};

assert(equals(new Rational(2, 2), Rational(1, 1)));

Even if the equals() implementation you are using has some support for custom types (automatically comparing each field), this example would still fail, even though 2/2 clearly equals 1/1.

Using ferrum.js to get it right

So, in order to implement equals() correctly, we need to support all the types your users might want to create; the function needs to be extensible!

Here is where ferrum.js comes in; it provides a helper class called Trait (one of the features borrowed from rust — see Rust Traits) to define extension points for functions like equals(). Ferrum already has an Equals trait and an eq() function, so we can just reuse it:

const { Equals, eq } = require('ferrum');

class Rational {
  constructor(p, q) {
    this.p = p;
    this.q = q;
  }

  // ...

  normalize() {
    // ... Normalize the rational so that 2/2 becomes 1/1
  }

  [Equals.sym](other) {
    // Tell ferrum in here how to compare your custom type
    const a = this.normalize();
    const b = other.normalize();
    return a.p == b.p && a.q == b.q;
  }
}assert(equals(new Rational(2, 2), Rational(1, 1)));

Ferrum already provides implementations of eq() for all standard types (Array, Map, Number, Date, etc.), so you just have to implement an equality function for your new type.

Internally, the library uses ES6 Symbols to find the implementation; this is the recommended way to implement generic interfaces in JavaScript. All ferrum does is wrap this in order to provide a more convenient interface to handle a lot of edge cases.

The Iterator Protocol uses Symbols in this way to implement ES6 iterators. You can even wrap existing protocols; for instance, the Sequence Trait is just a wrapper around the iterator protocol, created to make use of the advanced edge case handling of ferrum. One example: The Sequence Trait can support plain Objects, while the Iterator Protocol cannot.

Ferrum is also designed to be null/undefined safe; many functions explicitly handle null/undefined as a special edge case. Traits can even be implemented for null and/or undefined; our equals() implementation above on the other hand would simply crash. Ferrum even provides the typesafe module to safely deal with null/undefined values.

This was one of our main motivations when creating ferrum; while Rust has been designed to be safe and avoid a lot of those edge cases, JavaScript has historically had a lot of them. Ferrum is designed to take as much of the edge case load of the developer…anything that fits that description should probably be part of the Ferrum ecosystem — make JavaScript a bit safer.

What’s next?

Ferrum is currently under active development. One upcoming big feature (again, borrowed from rust) is documentation testing. Ever found that the examples in your documentation were full of bugs? This allows you test your documentation!

Other features expected to be released this year as a part of ferrum:

A Hash trait, and Hash tables supporting arbitrary keys.
A Ord trait and ordered maps supporting arbitrary keys.
Support for rxjs Observables and Asynchronous Iterators; all using the familiar Ferrum Sequence api!

This was originally posted on the adobe tech blog.

All the fibonacci sequence algorithms you will ever need

Mon, 19 Jun 2017 11:45:02 +0200

This blog post shall given an overview over different methods to compute the fibonacci sequence or parts of it. Since this is a common interview question, you may print out this blog article and show it to your potential employers, not only to demonstrate how specialized your abilities are in the broad area of fibonacci computation but also to demonstrate your humor and free spirit.

TL;DR: Type "fast fibonacci algorithm" into your mobile browser before the eyes of your soon-to-be boss and download the resulting algorithm.

The silly algorithm

O(wtf this is calling itself twice recursively must be something truly horrific)

def fib(n):
    if n < 2:
        return n
    else:
        return fib(n-1) + fib(n-2)

The naive algorithm

O(n)

Simply generate the entire fibonacci sequence, discarding unneeded values.

# Utilities

def seek(n, seq):
    it = iter(seq)
    for _ in range(n):
        next(seq)
    return it

def at(n, seq):
    seek(n, seq)
    return next(seq)

# Implementation

def fib_sequence():
    a = 0
    b = 1
    while True:
        yield a
        yield b
        a += b
        b += a

def fib(n):
    return at(n, fib_sequence())

The sensible

O(log n)

Search the internet for "fast fibonacci algorithm", find this page https://www.nayuki.io/page/fast-fibonacci-algorithms, and download the source code for fast doubling from that page. This is the only fibonacci implementation you will ever need. Use this and see how amazed (or potentially angry or scared) recruiters will be at how fast you can compute any fibonacci number.

Bonus: Recursive naive

O(1)

This is like the naive algorithm generating the entire sequence except that it's recursive, much more pretty and it will fail after a couple hundred iterations because the python joksters have out of pure trolling prowess not yet implemented proper tail call optimization

def fib_sequence(a=0, b=1):
    yield a
    yield from fib_sequence(b, a+b)

...

Why neither the command line nor the GUI is usable

Sun, 20 Mar 2016 18:33:50 +0100

Note: I wrote this as a comment http://gandre.ws/blog/blog/2015/04/07/why-the-command-line-is-not-usable/

TL;DR: As you said, the shell is lacking is a way to hint users towards common use cases, while in the GUI, it is almost impossible to combine applications.

I think we need to use a new, combined approach that lets users start with user interfaces as concise as today’s excellent GUIs, but providing tools to empower users to learn the power of programming in a rewarding learning curve. Without the leap of faith that is jumping from GUI to CLI or any programming language today.

—

Thanks for your post; the shell is indeed entirely unsuitable for use by non-experts: if a user opens a shell to solve a particular task (let’s say they want to display the current time), they will probably understand that they have to text something. Maybe they think of typing ‘time’ or ‘help’, but both of these commands will be entirely useless for what they want to achieve. Maybe they will take some time to research, and after a considerable time they might successfully be able to type date. Next, if they want to list the items on their desktop, they won’t have any idea where to begin with: after even more research they will learn about cd and ls, but at this point we’re just in day two of learning a programming language.

In the GUI, by contrast, you are likely to find out in seconds how to do these things; you will see the current time anyways on any decent system and even from only knowing how a mouse works you can probably learn how to navigate the file system from randomly clicking on the screen within 30 minutes or so.

Now let’s take a much more complex task; let’s delete all the images from a folder. This can be accomplished easily if you are an experienced shell users, yet if you are an experienced GUI user, it will be extremely difficult. I just tried to research how to “mac delete all mp3 from folder”, and the first result I got was a forum showing me an appropriate command line.

We create GUIs often by analyzing what usecases an application will have and based on that we will create hints for maybe 90% (arbitrary guess) of them. We can optimize that by using sane defaults (values for preferences, or just showing all the tracks in the playlist without any need for user intervention). We can group actions/views so users can make an educated guess about where to find a certain action. Ultimately though, there is a limit to the space of actions that can be performed using a GUI; we can only display so many options and hints until we simply can’t fit more information onto a screen: a list of 10000 preferences is about as useful as displaying none – just like we do in a shell.

GUIs also put a considerable load on the programming team: I am a programmer and I’d usually estimate two times the effort for creating a terrible GUI I design myself and more than ten times the effort for creating an excellent GUI with professional designers and usability studies. This is because good user interface design is very hard, but also because with a GUI you have to explicitly handle every single use case: if a want a feature that lets the user select a file to write to, or read from a website, I will have to built that in manually. Moreover, for the URL feature I should probably even display the website I am reading from within my application. In the shell, I usually just assume that the user will use curl and redirection to accomplish both those features, so I can support those features without actually programming anything.

GUIs show what can be achieved, by limiting what can be achieved. Having the computer hint users at what the system needs them to do quickly reaches it’s limits: ultimately the user needs to tell the computer what they want it to do.

I think it is possible to combine those approaches, because in a way GUIs already do this: App stores let you expand your system and customize it to your needs. At the beginning you see a very limited set of possibilities with those applications you are most likely to use. When you then expand your set of applications and add an application, you will have actively selected it and learn what it does.

I also think, we can take that even further and create a learning curve for normal users that empowers them to expand the way they use their system and even get to a point where they can perform moderately complex programming tasks (such as removing all mp3s from a folder).

It is hard to predict what such a system could look like and you’d need a big research team to get any meaningful idea, but my best guess would be having a normal GUI on top, backed by a graphical gui editor and a flow based, graphical programming language as a replacement for the shell and it’s pipes. Simple things like moving a button, or connecting the value of one field to some data receiver could be done using drag and drop (drag a text field to the notification icon to display notifications when it changes; drag it to the browsers URL bar to automatically follow the url). You could show users what happens in the flow interface when you drag a value somewhere, to teach them about the flow interface. If the GUI can easily be edited, this could also be useful to admins, optimizing an interface for their specific organization. The Flow interface could be stored as a full programming language (Rust or GO or something), so you can use that for algorithms and tasks that exceed the flow interface and you could learn about the language by using the flow interface and looking at the resulting code. You could even package sets of hints how to program as GUIs: how about a window that just displays the date in various configurations, so when you want your date in some weird format, use the GUI to configure how the date format should look like and then drag the result to the menu bar (on os x).

Note: I didn't mention in the comment, because I don't like plugging things, but this was the idea behind https://shocto.de/ , which at the moment purely exists in the form of an idea written on a website.

Blackmailed with Patreon account data

Sat, 21 Nov 2015 14:20:09 +0100

Note: if you receive a similar mail, don't pay. Your data is already public and their claims to have sensitive data is very likely a bluff.

So, some time ago I created an account with Patreon, mostly to support the Hoaxilla Podcast (check it out. They're really great). Unfortunately they have been cracked end of September and lots of data (torrent/magnet) has been leaked.

So, today I received the following mail:

Subject: Noah recommends - I will leak your identity
From: sharingservices@aol.com
Reply-To: abc8537458@163.com
To: patreonjo@jayceland.com, patreonjohari@gmail.com, patreonkaro@cupdev.net

Unfortunately your data was leaked in the recent hacking of the Patreon web site and I now have your information. I have your tax id, tax forms, SSN, DOB, Name, Address, Credit card details and more sensitive data. Now, I can go ahead and leak your details online which would damage your credit score like hell and would create a lot of problems for you.

If you would like to prevent me from doing this then you need to send 1 bitcoin to the following BTC address.

Bitcoin Address: 1QAQTyhCzAfvp8uLpneBNamWTNRR1hx9Cp

You can buy bitcoins using online exchanges easily. The bitcoin address is unique to you. Sending bitcoin takes take, so you better get started right now, you have 48 hours in total. abc8537458@163.com has shared an article with you

(Other people also received this)

First blackmail of my life. Yay. Fortunately it's a also bluff. The data was leaked publicly; It seems to contain postal addresses, email addresses and well encrypted passwords.

This means, that their claims to having tax ids, tax forms, SSN, DOB and credit card details are wrong. Their claims to be able to damage your credit score are also wrong. Since the data is already public, they can not do any more harm by leaking it again.

Besides, I am pretty sure I didn't even give them anything but a password, my legal name and a paypal account. So there is no way they could ever have data like a Tax ID. And what the hell is a SSN and a DOB. The worst thing they could possibly do with the data I gave patreon in the first place is to send me money, and I wouldn't object to that.

In the end this just adds to the miserable experience micropayment services provide. Bot my patreon and my flatter payments started failing after a few weeks; I tried to restore flattr payments and failed. I didn't even try with patreon.

Patreon was probably hacked because the run the same half assed approach most projects have towards security. I mean, I am running around finding random occurrences of possible buffer overvloas and private keys in random code I find. The most common cause of breaches is that someone calls and says "Hi, this is tech support. Could I please have your password".

How the hell are micropayments supposed to work with such sloppy tech. You are dealing with money, hire someone to do code reviews. Asshole.

(The full source of the mail for anyone interested):

Return-Path: <sharingservices@aol.com>
X-Original-To: patreonkaro@cupdev.net
Delivered-To: mapc@cupdev.net
Received: from taclomr-a001e.mx.aol.com (taclomr-a001e.mx.aol.com [204.29.187.81])
    (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
    (No client certificate requested)
    by cupdev.net (Postfix) with ESMTPS id BCB4E203C1E
    for <patreonkaro@cupdev.net>; Sat, 21 Nov 2015 12:30:12 +0100 (CET)
Received: from vm-149-174-150-116.asset.aol.com (vm-149-174-150-116.asset.aol.com [149.174.150.116])
    by taclomr-a001e.mx.aol.com (Outbound Mail Relay) with ESMTP id 1104F3800214;
    Sat, 21 Nov 2015 06:30:10 -0500 (EST)
From: sharingservices@aol.com
Reply-To: abc8537458@163.com
To: patreonjo@jayceland.com, patreonjohari@gmail.com, patreonkaro@cupdev.net
Message-ID: <534381049.34352343.1448105409977.JavaMail.dpadmin@vm-149-174-150-116.asset.aol.com>
Subject: Noah recommends  - I will leak your identity
MIME-Version: 1.0
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit
From-IP-Address: 172.29.108.145

Unfortunately your data was leaked in the recent hacking of the Patreon web site and I now have your information. I have your tax id, tax forms, SSN, DOB, Name, Address, Credit card details and more sensitive data. Now, I can go ahead and leak your details online which would damage your credit score like hell and would create a lot of problems for you.
<br><br>
If you would like to prevent me from doing this then you need to send 1 bitcoin to the following BTC address.
<br><br>
Bitcoin Address:
<br>
1QAQTyhCzAfvp8uLpneBNamWTNRR1hx9Cp
<br><br>
You can buy bitcoins using online exchanges easily. The bitcoin address is unique to you. Sending bitcoin takes take, so you better get started right now, you have 48 hours in total.
<title>abc8537458@163.com has shared an article with you</title>

Refactoring Sauerbraten – When two STLs break

Mon, 09 Nov 2015 11:33:36 +0100

Yesterday night I found this commit in one of the branches of Inexor:

commit ba8dd46289e39a51a67b40fe32d3e19607b63f2e
Author: a_teammate <madoe3@web.de>
Date:   Wed Nov 4 12:29:11 2015 +0100

    replace std::min max with selfwritten implementations to fix lightmapping

    vec::min/max were broken with the std versions

diff --git a/inexor/shared/tools.h b/inexor/shared/tools.h
index 6a32520..51f9d20 100644
--- a/inexor/shared/tools.h
+++ b/inexor/shared/tools.h
@@ -40,8 +40,13 @@ typedef unsigned long long int ullong;
 #endif

 using std::swap;
-using std::min;
-using std::max;
+/// return the minimal value of the two given.
+/// we do not use the std:: version here because it crashes inside the vec-implementation.
+template<class A, class B> inline A(min)(A val1, B val2) { return val1 > val2 ? val2 : val1; }
+/// return the bigger value of the two given.
+/// we do not use the std:: version here because it crashes inside the vec-implementation.
+template<class A, class B> inline A(max)(A val1, B val2) { return val2 > val1 ? val2 : val1; }
+
 using boost::algorithm::clamp;

Sauerbraten has a lot of self written implementations for a lot of standard algorithms and containers. That includes things like vector, hashmaps (unordered_map), sorting algorithms and also min, max and clamp implementations and even it's own math and crypto libraries. It has been argued that the predecessor of Sauerbraten – Cube – is so old, that all the libraries did not exist back then, but I looked it up and both OpenSSL and the first standardization of C++98 date back do 1998 while Cube started development in 2001.

In the course of Inexor development I was trying to replace at least a tiny little part of the sauerbraten special STL with the standard stuff, because the sauerbraten implementations do have some nasty stuff.

Here is an except from the sauerbraten vector implementation:

  /// get the last index and decrement the vector's size
  T &pop() { return buf[--ulen]; }

This is a pop method. It should remove the last element and pass it to the caller. Maybe by using return value optimization r possibly by taking a writable lvalue reference from the outside and moving/swapping into that. The STL vector's pop() just deletes the last element because both of the above implementations of pop are not quite optimal. The sauer implementation does neither and just returns a reference to the last element in the vector's buffer and marks that element as non-existent.
This is extremely dangerous; first of all any successive push/appends will overwrite the old element; secondly the destruction of the element is now in the hands of the caller who would need to manually call the destructor so the code behaves correctly (of course this would have to be done before another element is appended). Unfortunately large parts of the Sauerbraten code rely on such misbehaviour and thus I was unable to remove them easily.

I was however able to replace the implementations of min, max, clamp and swap and quite recently I was also able to replace sauerbratens custom random number generator. Though even that created some problems.

Now one of the other Inexor developers reintroduced the old implementations of min/max because STL implementation of std::min was crashing. This kind of problem already smelled like a deeper underlying problem so I decided to investigate:

First of all, I tried to reproduce the problem and I failed on recent clang and gcc versions, so it must be some compiler related problem.

Then I looked at the signature of both versions of min/max; maybe some code relied on the quirks of sauerbratens min/max, just like with vector::pop(). And indeed found a difference – the STL version of min max takes a single template argument and operates entirely on lvalue references while the sauerbraten version takes two template parameters (one for each parameter), uses the first parameter's type as return value and operates entirely by value rather than by reference.
I looked at the code that crashed but I found nothing to indicate such a problem.

Finally I got access to some from the crash itself:

STacktrace:

>    inexor.exe!_VCrtDbgReportW(int nRptType, void * returnAddress, const wchar_t * szFile, int nLine, const wchar_t * szModule, const wchar_t * szFormat, char * arglist) Line 481    C++
     inexor.exe!_CrtDbgReportW(int report_type, const wchar_t * file_name, int line_number, const wchar_t * module_name, const wchar_t * format, ...) Line 273    C++
     [External Code]    
     inexor.exe!vec::min(const vec & o) Line 156    C++
     inexor.exe!genclipplanes(const cube & c, const ivec & co, int size, clipplanes & p, bool collide) Line 1220    C++
     inexor.exe!getclipplanes(const cube & c, const ivec & o, int size, bool collide, int offset) Line 22    C++

from _CrtDbgReportW on were somewhere inside error reporting code probably. however checking the values of the variables inside that code reveals an error inside algorithm:4178 which is:

definition where we get stuck inside std::min:

        // TEMPLATE FUNCTION min
template<class _Ty> inline

    _Post_equal_to_(_Right < _Left ? _Right : _Left)

    _CONST_FUN const _Ty& (min)(const _Ty& _Left, const _Ty& _Right)
    {    // return smaller of _Left and _Right
    return (_DEBUG_LT(_Right, _Left) ? _Right : _Left); /// <- THIS IS the line which crashes
    }

Here you can see that it really crashes inside the min implementation. A look at the stack trace reveals that _CrtDbgReportW and _VCrtDbgReportW are being called; at first I though those where outputting debug info about the crash, but apparently there wasn't any, so we tried compiling in Release mode and the crash was gone.

Seems we found a bug in the Visual C++ STL Debug version; serves to remind that you should always trace bugs to the root cause before patching them.

We had recently updated to Visual Studio 2015 and I guess that introduced the problem.

If you want to reproduce this, I suggest you try with Visual Studio 2015 (minor update 1) and the current master at the time of this writing.

Hetzner Supportanfrage zu Zensur

Sat, 09 Aug 2014 11:03:28 +0200

Ich habe gerade einen Heise Artikel und einen Artikel auf SPON gelesen in denen beschrieben wird, dass Hetzner offenbar den Zensurwüschen der Russischen regierung nachkommt. Das passt mir nicht, weil mein eigener Server ja auch bei Hetzner läuft, also habe ich beschlossen mal ne Supportanfrage zu senden (und so vielleicht etwas Druck aufzubauen):

Sehr geehrte Damen und Herren, ich habe soeben folgenden Spiegel Artikel gelesen:

Webhoster Hetzner: Deutsche Firma hilft russischer Medienaufsicht bei Zensur

http://www.spiegel.de/netzwelt/netzpolitik/webhoster-hetzner-fordert-zensur-von-ukrainischer-website-glavcom-a-985147.html#js-article-comments-box-pager

Darin wird beschrieben, dass sie eine Mitteilung der russischen Medienaufsicht erhalten haben, die die Aufforderung enthielt teile der Ukrainischen Seite Glavcom zu Zensieren. Offenbar haben sie darauf reagiert indem sie die Website aufgefordert haben, die Inhalte vom Netz zu nehmen.

Das irritiert mich sehr, denn sie sind einen Deutsche Firma und ich hätte von ihnen erwartet, dass sie die Interessen ihrer Kunden wahren.

Bitte erläutern sie, wie rechtfertigen sie diese Vorgehensweise? Inwiefern widersprechen die Inhalte auf der Seite Glavcom Punkt 6.2 ihrer AGB? Im lichte auf meine eigenen Webseiten, möchte ich außerdem wissen unter welchen Voraussetzungen sie weitere Zensurmaßnahmen durchführen werden? Ist generell damit zu rechnen, dass den Zensurwünschen anderer Regierungen nachkommen?

Mit freundlichen Grüßen, Karolin Varner

Die Anfrage lizensiere ich natürlich unter CC-0 und ich lade alle anderen Herzlich ein dieses Schreiben als Vorlage für ihre eigenen Anfragen zu benutzen!

Update

(Heute ist der 17.3.2015)

Hetzner hat mir in der Tat geantwortet und das auch recht schnell – also zumindest viel, viel schneller als ich das hier veröffentliche.

Guten Tag,

Wir möchten uns zu dem von Spiegel Online veröffentlichten Artikel wie folgt äußern.

Wir als Webhosting-Unternehmen konzentrieren uns auf unsere Kernkompetenz, unseren Kunden auf aller Welt Leistungen rund um das Hosten von Webseiten bereit zu stellen, und halten uns politisch neutral. Wir pflegen keine politischen Beziehungen im In- und Ausland.

Grundsätzlich werden bei uns alle Beschwerden gleich behandelt. Sobald eine Beschwerde über eine auf unseren Servern gehostete Webseite eingeht, wird die Beschwerde formell geprüft und der betroffene Kunde per E-Mail an die bei uns hinterlegte E-Mail-Adresse informiert und gebeten, binnen 24 Stunden eine Stellungnahme dazu abzugeben.

Bei der formellen Prüfung prüfen wir, ob die beanstandete Webseite tatsächlich bei Hetzner Online gehostet wird. Darüber hinaus erfolgt eine erste Blickprüfung der Webseite, eine inhaltliche Prüfung findet in dieser Stufe noch nicht statt.

Erfolgt nach Fristablauf keine Reaktion/Stellungnahme des Kunden, so wird eine Erinnerung an den Kunden gesendet, mit Angabe einer evtl. Sperrung bei weiterer Nichtreaktion.

Bei weiterer Nichtreaktion wird diese Erinnerung nach weiteren 24 Stunden in der Regel nochmals versendet. Reagiert der Kunde weiterhin nicht, wird der Inhalt der beanstandeten Webseite geprüft und ggf. die Sperrung durchgeführt, um Schaden für den Kunden und Hetzner Online abzuwenden.

In dem von Spiegel Online veröffentlichten Fall wurde von unserem Kunden auf unsere Anfragen nicht fristgerecht eingegangen. Eine Serversperrung ist trotz der mangelnden Kommunikation des Kunden nicht erfolgt.

Bezüglich des Inhalts der angemahnten Website (sogut das über die Sprachbarriere hinweg möglich ist) sehen wir unsererseits keinen Grund, das Abuse-Ticket weiter zu verfolgen. Der Kunde wurde darüber informiert, dass wir das Abuse-Ticket schliessen, und unsererseits keine weiteren Schritte notwendig sind. Was uns in künftigen ähnlichen Fällen sehr helfen würde, wäre eine Stellungnahme des Kunden. Wenn man die Argumente beider Seiten kennt, fällt eine vernünftige Entscheidung im Abuse-Vorgang wohlmöglich leichter, als wenn nur die Beschwerde eines Beschwerdeführers vorliegt, welche vom Kunden ignoriert wird.

Wir bedauern den Vorfall sehr und hoffen, dass die kritischen Stimmen unsere Vorgehensweise nachvollziehen können.

Mit freundlichen Grüßen / Best Regards [NAME ENTFERNT]

Non breaking multi line strings in Ruby

Fri, 08 Aug 2014 12:44:49 +0200

Personally I like my code to be no longer than 60 characters (that's because so I can comfortably split my Vim screen and view two files side by side).
When I am coding rails it often happens that I need to create some error message or user feedback and these strings get far longer than 60 characters normally. Until now I've split my lines using + in these cases.

print "This is a nice, long string giving you some " +
    "feedback. Unfortunately it does not fit on the " +
    "screen so I have to split it."

I dislike that syntax because it is a lot of hassle to get the spaces and the quotes right. There seem to be a few better syntaxes (according to this StackOverflow Question), but they do not substantially improve the situation I think:

print "This is a nice, long string giving you some "\
      "feedback. Unfortunately it does not fit on the "\
      "screen so I have to split it."

The solution I came up with was a little helper function:

# Helper for multi line strings:
# Normalizes the string, so that every sequence of spaces is
# replaced by a single space.
# This also strips the string.
def NOCR(s)
  s.gsub!(/\s\s*/, ' ')
  s.strip!
  s
end

print NOCR "This is a nice, long string giving you some
    feedback. Unfortunately it does not fit on the screen
    screen so I have to split it."

This is still not a perfect solution, but it at least saves me the hassle of manually formatting the spaces and adding lots of quotes.

Of course, when using this, support for inserting newlines or tabs is completely gone. It would be possible to use some kind of escape syntax (%%, %n, %t, %s) to introduce it again, but I think in these cases it is better to fall back to manual formatting.

Git stats – LOC per author in the current head

Fri, 01 Aug 2014 04:08:47 +0200

So suddenly I am fluent in awk. Well, this is unexpected!

I've been working on one of my projects for quite a while and today I got interested in how much code I actually produced, so I wrote a little script to generate that statistic:

#! /bin/bash
# List the amount of code per author in the current HEAD

git ls-files "$@" | xargs -l1 git blame -s -- | awk '
  function get_author(commit) {
    author = commit_map[commit]
    if (author == "") {
      cmd = "git show -s --pretty=format:%ae " commit 
      cmd | getline author
      commit_map[commit] = author
      close(cmd)
    }
    return author
  }

  {
    author = get_author($1)
    counter[author] = counter[author] + 1
    if (counter[author] % 500 == 0)
      print($1 " -> " author " -> " counter[author]) | "cat >&2"
  }

  END {
    for (author in counter) {
      print counter[author] " " author
    }
  }
' | sort -n | column -t

Github Gist

This lists the files in the current head, git-blames each of those and runs the result through AWK. Awk does the author lookup (caches the result in an associative array) and then amount of lines for each author in another associative array. Finally the result is being printed, sorted and formatted.

Shuffling movies with mplayer and bash

Thu, 08 Aug 2013 22:20:05 +0200

Suppose you have a bunch of movies in a directory and want to play them randomly; switch into that directory and run:

$ find -print0 | sort -zR | xargs -0 mplayer

Generate a list of files in this directory and it's subdirectories; the list is zero terminated because I have some special characters in my filenames
Shuffle the list with sort -R
Pass each line as an argument to mplayer

The zero-termination of the list is accomplished with find -print0, sort -z and xargs -0.

You can of course alter the find command: For instance use find -maxdepth 1 to skip subdirectories.

Batch rename in Bash

Sun, 12 May 2013 23:25:53 +0200

There are a few tools that provide batch renaming for the shell, but most of them are quite huge and need installing. It is not necessary to use these utilities, because unix already all tools necessary:

$ ls -d *.txt | sed 'p;s/foo/bar/' | xargs -l2 mv

The first part should be quit clear: Print all files ending with txt. The \*.txt makes use of the shell's globbing features and filters all the .txt files. The -d switch tells ls not to print the contents of directories.

The sed expression consists of two parts: p prints the current line and s/foo/bar/ is the actual transformation (in this case: a replacement). If I run this on my home directory I get this:

$ ls
down  duh  files  fuu  fuubar pr0j  tmp  usr

$ ls | sed 'p;s/fuu/bar/'
down
down
duh
duh
files
files
fuu
bar
fuubar
barbar
pr0j
pr0j
tmp
tmp
usr
usr

Notice that most files have just been printed twice, but fuu and fuubar where changed to bar and barbar.

Now comes the tricky bit: xargs takes each two lines and applies them to mv as arguments, so when I run xargs in debug mode I get this:

$ ls | sed 'p;s/fuu/bar/' | xargs -l2 echo mv
mv down down
mv duh duh
mv files files
mv fuu bar
mv fuubar barbar
mv pr0j pr0j
mv tmp tmp
mv usr usr

Notice that I did not use the '-d' flag this time, because I print the content of the directory '.' this time, not a list of given files. This would happen if I did use '-d'.

$ ls -d
.

$ ls | sed 'p;s/fuu/bar/' | xargs -l2 echo mv
mv . .

Getting complicated

In the above example I did not do any filtering, because the files foo and foobar (which existed before I began to write this article) do not have an extension, but I could use a filter to select only those files I actually want to rename:

$ ls -d *fuu* | sed 'p;s/fuu/bar/' | xargs -l2 echo mv
mv fuu bar
mv fuubar barbar

I can get as elaborate as I want with my filter if I use grep; here is the same as above using grep:

$ ls | grep 'fuu' | sed 'p;s/fuu/bar/' | xargs -l2 echo mv
mv fuu bar
mv fuubar barbar

Instad of using xargs you can also use a while loop; I use that variant to save me the trouble of dealing with escaping in xargs:

$ ls | grep 'fuu' | sed 'p;s/fuu/bar/' | while read a && read b; do echo mv "$a" "$b"; done
mv fuu bar
mv fuubar barbar

One last example, where we replace files recursively in the home directory: I use find instead of ls which lists a directory recursively (I am not actually running this and neither should you):

$ find | sed 'p;s/fuu/bar/' | xargs -l2 mv
...

Hello World

Mon, 06 May 2013 23:30:36 +0200

Soo. If you can see this it means I managed to setup my blog. Yaaay

A wonderful world awaits you on this blog…just not yet. Move along. Nothing to see here.