DWARF backtracing can be quick and safe

There is a widespread perception that DWARF-based unwinders are slow and crashy. For example, a famous rant by Linus: https://lkml.org/lkml/2012/2/10/356. That rant applies to one specific Linux sub-project. User space has its own challenges with DWARF-based unwinding. The kernel uses an alternative called ORC. At least two efforts now aim to replace or amend DWARF unwinding in user space. And of course, there is a somewhat popular movement to build everything with frame pointers. But frame pointers don’t make all backtraces perfect. Not for CPU profiling, where profiling “ticks” in epilogues, prologues, or PLT are common. And imperfect backtraces can be confusing and painful to deal with.

We can do much better than that.

So for the last few months I was building aw-backtrace. aw-backtrace is the DWARF-based unwinder that won't crash and is very quick. It is a good-sized step towards perfect backtraces.

While I cannot “mathematically prove” that aw-backtrace cannot crash, I did put a serious effort into performing bounds checking for all the accesses, both for reading “DWARF stuff” and for reading register values from the stack.

Here is a quick introduction to DWARF unwind info. Every compiled function has FDE (slight simplification, but good enough for an intro). Each compile unit (.o file) has a CIE that describes language and architectural defaults (e.g., x86 functions start with the return address pushed on the stack). Another notion in the unwind info is the CFA, which is basically the stack pointer at the call site. Unwind info typically looks like: “to establish CFA take RSP register and add 16, then to take return address read machine word at CFA - 8, value of the RBP register is saved at CFA - 16”. This is represented as a sequence of CFI instructions that define such unwind info for each instruction in the function. Tools like readelf (with the -wf flag) can dump this and show you what it looks like. Alternatively, checking the compiler's assembly output is useful too, e.g., on godbolt: https://godbolt.org/z/dErnbcnT1

What makes DWARF unwinding a slightly harder target is that it was originally produced for debug info and was intentionally very powerful and general. But that means nearly unbounded complexity. Stuff like an arbitrarily deep stack of .cfi_remember_state. In practice, as we know from ORC and related efforts, compilers produce a fairly limited “shape” of unwind info. There are occasional non-trivial DWARF unwind info. For example, some hero has put into glibc for Linux/x86-64 (but I think only there) a very elaborate unwind info for signal trampolines. Because describing the restoration of every register from ucontext’s gregs fields involves indirections, it is full of DWARF expressions. Another very smart example is unwind info for the PLT section produced by the GNU linker (but for some reason the LLVM linker doesn’t produce any unwind info for PLTs).

For aw-backtrace, the explicit choice is to keep things safe. So it supports only a practical subset of unwind info, roughly on par with SFrame V3. aw-backtrace special-cases and explicitly handles known unusual cases like signal handler trampolines or the PLT. There are currently few functions that I see in libc (e.g., longjmp)  and OpenSSL where unwind info is too advanced for aw-backtrace. I plan to address this eventually, but what it has now comfortably covers the vast majority of code.

I believe what compromised DWARF backtracers before is that backtracing was so far always just a side effect of more general code for propagating exceptions. If we’re in the middle of unwinding a stack for an exception and face a function with broken or excessively complex unwind info, there is nowhere to go anyway. With backtracing, you can always stop unwinding. Yes, truncated backtraces are imperfect. But crashing or hanging is much worse.

How do I know that uncommon unwind info is rare? It is straightforward to dump unwind info and grep for DWARF expressions. But also because I single-stepped through a bunch of code while comparing the actual backtrace with what aw-backtrace produces.

It is actually a somewhat remarkable piece. Grown out of my earlier effort to “torture” atfork handling in gperftools. I have code that sets the x86 TF flag, which makes the CPU “single step”. Stepping delivers an interrupt to the kernel, which passes it to user space as SIGTRAP. From that SIGTRAP handler, we run the comparer. To speed this up, I built a “mini interpreter” for the common subset of x86 instructions, so we don’t go through the interrupt/signal boundary every time. There are, of course, a bunch of nasty details, like fighting with other code for altstack or dealing with SIGTRAP masking. So it is by no means perfect. Some details are in the README, and the code is hopefully not too large and somewhat readable. I am actively replacing it with a portable solution that single-steps via a QEMU plugin.

Another part of the DWARF inspection sub-effort is the unwind-check tool I released together with aw-backtrace. It disassembles the code, compares its effects with unwind info, and complains if there is a mismatch. It already found 2 bugs in LLVM (but thankfully the mismatches are relatively short and don’t cross call-sites). In the future, I hope and expect compilers will “never again” produce bad unwind info. Because verifying unwind info is relatively straightforward.

And finally I have things to say about backtracing speed. We will likely never fully match the speed of trivial frame-pointer-based backtracers, but it is relatively easy to come very close. The key is caching. Workloads that capture many backtraces tend to step through the same call sites. Implementing a lockless, async-signal-safe cache was a mild challenge, but ultimately it is just a matter of engineering. The README has some details, and the code should be very readable too (for now, the focus was to keep things mega-simple; I undid the more advanced caching and eviction approach so the code is hopefully easy to verify and understand).

Beyond caching, yes, DWARF unwind info is by no means perfect for performance. Those inconvenient augmentation details. Or having to decode padding NOPs in CIE. But the basic stuff (redundant instructions in CIEs, basic prologue/epilogue bits) is basically very bounded in size and thus in decoding cost. The bigger inefficiency is having to advance sequentially from the start of the function to where we need to unwind.

In most cases, even larger functions are fairly straightforward. Most of such functions are covered by one or two large advances. But sometimes unwind info gets large when smaller advances occur between CFI updates. This happens more easily for functions that pass arguments on the stack, since pushing or popping anything updates frame size; in the common case of using stack pointer-based frame layout, it requires updating unwind info.

aw-backtrace comes with a recursion-test benchmark that artificially produces such a case. On those larger functions, one step of unwinding could approach 100 nanoseconds even in the cache-hot micro-benchmark. Thankfully such cases are not very common. I perf-recorded a kernel build, so most profiling ticks are in the guts of gcc, and the average number of CFI instructions per frame I see in this profile is around 19. Not great, but not too bad either. SQLite’s benchmark comes in at around 26. So still not a disaster.

In short, we could make unwinding much cheaper, and likely encode much tighter as well. But what we have today is already close enough to be useful in practice and not orders of magnitude worse. We can work with that. Especially given that caching more or less obviates the need for quick DWARF decoding.

Of course, the biggest performance bonus of aw-backtrace is that it unlocks wins of -fomit-frame-pointer.​

Visualizing perf profiles using pprof

Perf is a pretty powerful profiling tool (in addition to it's other features). But it is not as good as pprof at visualizing profiles.
Now you can capture profiles using perf (including profiling running processes), but deal with them using pprof. With help of perf2pprof tool. It is also available via rubygems. Simply do gem install perf2pprof and you're ready to rock.

Better productivity writing golang code with help of supermegadoc

At work I'm writing more and more golang code recently. It is nice little language. But as a language noob I have yet to internalize which features are available in standard library and how to use them. So I'm spending a lot of time staring at pages under golang.org/pkg looking for types or functions or methods I need.
For my Erlang work I'm quite regular user of supermegadoc erlang integration (video demo with my nice/odd accent is here). And I was seriously lacking something similar for go. Today I finally spent few hours and I have something that looks like what I need.

Observe:

I can quickly find things that are visible to godoc. Just like with other supermegadoc integrations, I can see function signatures and constant and variable values. And I can see if type is struct, "typedef" or interface. From experience working with Erlang's supermegadoc, I know that this means that I often don't even need to open corresponding doc entry. It is often enough to see that it's there and (in case of functions or methods) what it's signature is.

I expect my golang productivity to increase.

Have a nice day folks!

Massive power of (liblzma based) XZ archiver

I've recently restarted gathering of bitcoin market data. I'm grabbing samples of market depth every 3 seconds and I'm collecting trade events.

Market depth samples can be quite large. Every mtgox sample appears to be about 90 kilobytes big. So 1 hour of samples is about 100 megs of data. And month is about 3 gigs. Which is a bit too much.

gzip is able to compress that about 5x. But that's still a bit too large.

I've found xz to really shine on that kind of data. More than 1 gig of data gets squeezed down to less than a meg! And what's extra cool is xz is very quick to decompress. For static data like btc market archive that's very useful.

So quality compression does matter. And I just wanted to express my ultimate respect to authors of that extremely useful software.

Have a nice day and happy hacking!

Playing with Intel TSX

I've recently got access to a box that has Intel Haswell CPU inside. And I was quite looking forward playing with one of it's most interesting features: hardware transactional memory. My particular interest is to see how cheap it is.

My use case is per-processor data structures (e.g. malloc caches). And without explicit binding of threads to processors, there's only optimistic way of doing it. Which requires some synchronization to defend against pessimistic case of rescheduling of thread to different cpu. That would look like taking cpu id, locking it's corresponding lock which in most cases would be in cache and uncontended and thus reasonably quick, and then doing something with per-cpu data. So in this approach we always pay some performance price even if majority of actual runs will hit fast-path. Lack of really cheap optimistic locking makes that price significant which makes it less attractive.

So lets return to Intel's implementation of transactional memory (aka TSX). Wikipedia article describes that thing pretty well. My understanding is that it's expected to be most useful for somewhat coarse locks where multiple threads would normally contend for the lock yet they touch different memory locations. E.g. imagine different threads touching different buckets of hash table or different branches of binary search tree. It can also be used as compare-and-exchange operation that allows you to process multiple memory locations at once. There's already glibc support for it that optimises pthread mutex operations described in usually nice lwn.net article.

My hope was that this feature ends up being even faster than atomic operations in fastest path (everything is in L1 cache) given it's optimistic nature. And that it might be useful for quick optimistic locking I'd like to have.

You can see my test case here. It simulates fastpath of "lock" that guards a counter. There is no locking itself, just check that "lock" is free. Which is what glibc lock elision code is doing. And you can see how TSX allows to avoid actual locking. "On the other side of the ring" is code that changes counter via traditional compare-exchange atomic operation (no locking either, to give me purer numbers).

On the box I have access to (with Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz processor) I'm getting about 67 cycles per loop iteration for TSX case. And about 27 cycles for atomic CAS code (and same for more traditional locking instructions "xchg mem, register"). Note that it's very likely that larger transactions will have bigger overhead. Also note that usual synchronised region is two atomic ops (unlocking atomic operation being potentially significantly cheaper than locking operation), so in this limited case TSX appears to be somewhat competitive with traditional locking, but not faster.

So TSX is not faster than single atomic operation. Which breaks my hope of using it for quick optimistic lock It is somewhat sad that on today's hardware there's seemingly no way to have fast-path of locks to be lightning fast without playing crazy tricks (e.g. slow path stopping lock owner thread via signal or ptrace like jvm "biased locking" appears to be doing).

Anyways, being slightly more than 2x slower than simple atomic operation is pretty good news IMHO for use cases for which TSX is designed. And it's "multi-word cas" application appears to be very interesting and useful too. So I'm looking forward using it somewhere.

And finally I have to note that, especially at the beginning, debugging transactional memory code can be quite tricky and very weird. That's because transaction is fully isolated while it runs, so there's no way to printf something to see why it fails. Or set breakpoint inside it and inspect things. This hit me initially because my simplistic code wasn't at all prepared to handle transaction failures. I.e. my code is only supposed to test fast-path without any real-world synchronization contention. After few minutes of struggling with it I realized, that even otherwise conflict-less code will abort from time to time. For example, any interrupt (e.g. timer tick) will abort in-flight transaction, as well as in fact any user- to kernel-space transition will.

So lesson number one is that debugging hardware transactional memory code should be done very carefully. Especially if code path is significantly different between successful and abort-ful cases. I.e. imagine some real transaction that might span several layers of code and consider that debugger/printf will never be able to see or expose "guts" of aborted transactions. And lesson number two is that aborts have to be handled always, even in toy code.

Have a nice day and happy hacking.

gpicker 2.2 is out!

Hello there! I've just made long due release of gpicker 2.2. Some notable changes are:

• new project type -- script, that I'm using to handle multi-repository project (i.e. couchbase)
• implemented poor man's isearch on steroid's -- gpicker-isearch
• big improvements for gpicker-imenu
• more optimization

Savannah project page has link download area with source .tar.{gz,bz2,xz} archives and binary .deb packages (built on lenny) for i386 and amd64. If you haven't heard of gpicker before also check out supermegadoc which is very convenient gpicker-using tool.

Me and Gnome3

Hi. Quite a bit of time passed since my last past. That was busy time with continued hard work on (still forthcoming) Couchbase Server 2.0 release and, most importantly, I've found beautiful girl and got married!

Anyway, I just got remind that I should not forget about writing something from time to time. And today's "hot" topic is Gnome 3.

About a month ago (or was it 2 ? Time flies so weirdly with so much happening around me now) Debian Sid got Gnome 3. Even earlier it got some components of Gnome 3. Most noticeable was upgrade of gnome-terminal to Gnome 3 version. And that was almost immediately reverted back to gnome-terminal 2 from last Debian stable. The reason is very simple. Default theme of gtk3 (which is, naturally, used by all gnome 3 apps) is ugly. Like very very ugly. And, surprisingly, there's only one non-default theme engine for gtk3. The one that's heavily using CSS3. I don't like it's look either, but the most worrisome aspect of it is quite noticeable slowness. There are ways to adjust look with CSS3 hackery after all. I've found that some porting work of old gtk engines was initiated. But quick and minimalistic Mist engine I'm used to is not yet ported.

That's basically my whole Gnome 3 story. I cannot tolerate Gnome 3 not because of it's experimental UI, but because I need usable gtk3 theme first. I cannot even say what I'm thinking about gnome's UI, because I haven't even tried using it on daily basis.

Whoever makes Mist work on gtk3 will become my hero. Meanwhile, I was forced to find refuge in XFCE land, that's missing few things I had on my gnome 2 desktop.

Unbreaking LXC on latest Debian unstable

With recent switch to /run directory in Debian I was getting error from lxc when it was trying to mount /dev/shm in container and failed because /dev/shm is now symlink to inside /run. The simplest fix I found is replacing symlink with bind mount. Here's what I've added to /etc/rc.local

if [ -L /dev/shm ]

then

  mv /dev/shm /dev/shm~

  mkdir /dev/shm

  mount --bind "`readlink -f /dev/shm~`" /dev/shm

fi

Setting up Distel and erlang remote shell for membase

I decided to do myself a small present. I've just wrote some Emacs lisp code that via REST API grabs erlang otp node and cookie and connects my emacs with that node. I've made integration with Distel and erlang remote shell. The later was most problematic because of quite weird TTY handling in Erlang shell. But in the end it seems to work great! Grab code here: https://gist.github.com/952958 And the usage is M-x alk-membase-shell and M-x alk-membase-setup-distel.

Should we avoid exceptions in JS ? (Or the problem with jsPerf micro-benchmarks)

In general my take on benchmarking is: it's too easy to screw up something. The biggest problem is that with normal programming/engineering it's easy to see if you're doing it right. With benchmarking you have some (plausibly looking or otherwise) numbers. I actually think that most of benchmarks we're seeing are flawed in some way. So it's very easy to jump to conclusions, only to be found guilty of screwing up some important aspect. So first of all I'd like to make the following statement:

I understand that benchmarks suck. Please be careful and don't blindly believe any benchmarking numbers. Yes. Even mine.

Now back to main story. Today my pull request for underscore was rejected. I've proposed nice call/cc like helper to do non-local return from JS functions. It should be especially helpful for breaking out of forEach/map loops. More ideally ES5 designers could have added some ability to terminate loops early. Like via returning some special value. But apparently they're too conservative. Maybe I'll post some rants about that some other day.

Anyway, my patch was rejected because underscore author does not want to use exceptions for control flow. His point is that exceptions are very slow.

But is that true? I'm not sure. If we want all performance we can get, then probably we shouldn't use forEach & friends in first place. And then ES3's break statement can actually break out of several loop nestings (via break to label). But note, that standard forbids breaking across function calls, otherwise it would be nice (and potentially faster) alternative to exceptions.

Part of the problem is that, apparently, V8 forces heap allocation of frames of functions with try/catch block.

The fact is, any code that creates closures has potential to force heap allocation of stack frames (or parts of), which if done often enough will trigger (relatively) expensive GC pauses. That's because if closure is 'leaked' out of it's dynamic scope, it effectively out-lives function that created it. So any variables that are needed by closure cannot live on normal stack.

Lot's of tricks in modern JS interpreters allow them to avoid heap allocation in many important cases. But sometimes you still have to pay that price.

My point is: there is no other way to do non-local return in JS other than by throwing exception. And it doesn't seem to be too slow. I also expect JS-engine vendors to gradually optimize that case.

But even though I understand micro-benchmarking drawbacks. Here's my (potentially flawed) take on for-loop-with-break vs. forEach with throw problem.

First, here's jsPerf page on plain for vs. forEach performance. We see that on FF3.6 forEach is much slower than loop. But on V8 it's about same.

Second, here's jsPerf page on for with break vs. forEach with throw performance. We see that throwing out of loop is not too bad. It actually seems to be faster on Firefox and only 1.5-to-2 times slower on Chrome's V8.

Also note, that somehow manual inlining of add function produces much larger scores, which might indicate  major flow in this microbenchmark. In general of course we'd like to compile/optimize benchmark code once and then run it multiple times. But lack of automatic inlining in this very trivial case hints that we might be causing JIT to optimize it each time it's run. So feel free to play with that benchmarks more and find some flaw!

Note that there's some plausible evidence that throwing non-Errors is much faster. I think that's because when you throw Error instance, it causes runtime to collect backtrace and fill in 'stack' attribute, which seems to be much costlier then just throwing. In fact you can google for Java exception throwing performance and you'll discover that optimal throwing performance is reached by throwing 'prepared' exception instance. Exactly for same reason.

But returning to original question, I still feel that in some cases exceptions are perfectly good way to do some non-local returns. There are so many creative ways to use all language features. And blindly rejecting any of them, just because it seems to be slow is not smart, IMHO. Sure, exceptions have their cost, but sometimes avoiding them is costlier. And all trade-offs should be carefully analyzed and understood.

Caching autotools outputs for fun and profit

GNU make parallell building feature is cool. And ccache which speeds up rebuilds of same files is cool too. But even with this great tools rebuilding some project is still not as fast as it can be. Why ? Because preparing GNU autotools files (./configure & friends) takes quite long time. And there is no way to parallelize it.

But we can cache autotools products just as ccache caches compiler products. And now there is tool that's capable of doing that! You can grab it here. I've adapted fabricate.py for this job.

Just put fabricate.py into PATH and prepend it to your command. Like:

fabricate.py ./bootstrap

or

fabricate.py sh -c './bootstrap && ./configure'

And it'll copy command outputs from cache if all dependencies are same. Nearly instantaneous!

Welcome to 21th century, Aliaksey!

I'm used to slow, expensive & unreliable Internet. Because of that I tended to keep as much as possible on my local hard drive. I had tons of specs, articles, e-books. And I even kept local mirror of entire Debian unstable repository! That's because at home and until recently at Altoros office I had quite limited and expensive Internet access. I tried (successfully) to reduce my dependency on Internet as mush as possible.

Around 2 weeks ago I accidentally deleted most of my 'stuff'. And now I'm forced to use online resources. And quite surprisingly that's not bad experience at all! Internet here, at Bay Area, is fast enough. And googling for information and reading it online sometimes seems to be even faster than locating and opening it from local disk.

This is unusual and new experience. When I, for example, need some information on how something is done in python I just ask Google. And within mere seconds I have precisely what I need! I've even started reading PDF articles online via Chrome's plugin without downloading them first.

I'm slowly getting used to 21th century Internet. I'm still not quite 100% comfortable with depending on Internet availability. And some lookups are definitely much faster against local data. For example, I don't think anything can be faster than local supermegadoc documentation lookup of some function. But I'm going to stop downloading interesting stuff and will instead bookmark it. Let's see how it'll go. I'm especially curious how it will work when I'll return home. Hopefully, we'll have speedy ADSL connection by then :)

How to use two monitors in IntelliJ IDEA

I'm working on some educational Scala project. And IDEA is natural choice for that. Normally I use Emacs for my work, but for anything Java related it's not as good as IDEA. And I cannot express how powerful and polished this nearly free software IDE is.

However, I was missing one very useful feature of Emacs, which is ability to open several frames to effectively use two monitors. Until few minutes ago.

When you want to open same "buffer" on two monitors you can first split it horizontally (C-x 2 if you use Emacs key bindings) and then you can simply drag one of the windows out of it's frame to second monitor! If you want different files you can just drag tabs out of main frame.

How to testament... childs' death

Often Unix processes spawn childs. Contrary to the popular belief childs DO NOT automatically die when their parent exits.

Unix job control has a concept of sessions and session leaders. Session leader is the process that 'owns' terminal, and when this guy dies, each process of session gets SIGHUP. This usually leads to death. Every time you open new tab in your terminal emulator, you're creating new session. And bash (or other shell) is leader of this session. So your childs will likely die when terminal emulator tab will be closed, but they will easily outlive your process. And they will live forever when your main process is spawned from cron or your favorite continuous integration tool.

Of course the whole topic of Unix job control is slightly more complex. You can read more by reading GNU libc manual (or aptitude install glibc-doc && info libc). Posix man pages (aptitude install manpages-posix{,-dev}) contain lots of details, much more than usual Linux man pages.

So one of the ways to ensure your childs die is by using sessions. This is usually implemented by creating pseudo terminal pair (PTY) and giving slave side of that pair to child. Session leader (i.e. child) gets SIGHUP when master side of PTY pair is closed. Master side is closed when your main process exits (if you don't pass it's fd to child by mistake). When child dies, this causes delivery of SIGHUP to whole session. {spawn,fork}pty-like functions are used for that.

And there's another even less widely known way. There is a concept of orphaned process group that can be used to ensure SIGHUP delivery without relying on PTYs. I'll refer to posix man page for details. But basically you fork and deliver SIGSTOP to child. When your process group becomes orphaned (e.g. when main process dies), each member will be sent SIGCONT & SIGHUP. Added benefit of this approach is that if parent of your main process dies, your process and it's child will die too. This relies on main process or it's parent being process group leader, but that usually holds.

I'm using this to control a bunch of erlang VMs when running Membase in development mode. You can grab python code for doing that here. Ruby code for doing same is removed, but you can dig it's grave.

UPDATE: unfortunately this trick doesn't always work in OSX/Darwin due to different definition of orphaned process group in OSX: http://developer.apple.com/library/mac/#documentation/darwin/reference/manpages/man2/intro.2.html

Two envelopes sophism puzzle

I'm currently on vacation. And it allows me to return to some small topics that I postponed earlier. One of them is Two envelopes problems. This is quite interesting and difficult sophism to solve and 'get'. This morning I'm not in 100% work-ready mood so I decided to return to my list of 'Later'. Surprisingly, this time it took quite small amount of time to solve this puzzle. Actually, I'm a bit puzzled why I postponed it earlier. Maybe I should take vacations more seriously ?

I'm copying definition from wikipedia:

The basic setup: You are presented with two indistinguishable envelopes containing some money. You are further informed that one of the envelopes contains twice as much money as the other. You may select any one of the envelopes and you will receive the money in the selected envelope. When you have selected one of the envelopes at random but not yet opened it, you get the opportunity to take the other envelope instead. Should you switch to the other envelope?

The switching argument: One line of reasoning proceeds as follows:

1. I denote by A the amount in my selected envelope.
2. The probability that A is the smaller amount is 1/2, and that it is the larger amount is also 1/2.
3. The other envelope may contain either 2A or A/2.
4. If A is the smaller amount the other envelope contains 2A.
5. If A is the larger amount the other envelope contains A/2.
6. Thus the other envelope contains 2A with probability 1/2 and A/2 with probability 1/2.
7. So the expected value of the money in the other envelope is
   
   
   
8. This is greater than A, so I gain on average by switching.
9. After the switch, I can denote that content by B and reason in exactly the same manner as above.
10. I will conclude that the most rational thing to do is to swap back again.
11. To be rational, I will thus end up swapping envelopes indefinitely.
12. As it seems more rational to open just any envelope than to swap indefinitely, we have a contradiction.

The puzzle: The puzzle is to find the flaw in the very compelling line of reasoning above.

IMO the most easily understandable flaw in that reasoning is that it adds values of A from two possible realities. And those values obviously differ. If you analyze this problem correctly then it's obvious, that switching envelopes does not make sense. So the lesson is: be careful with your variables.

Reflections on Ruby vs. Python meetup

This is cross-posted from Altoros' developers blog. I want to thank Altoros PR department for editing this text.

I never understood developers and customers that willingly choose Python over Ruby. Ruby does feel like a native language to me, while everything else doesn’t. No, this doesn’t mean that I don’t like working with other technologies. Over the last year and a half, I was really inspired by my positive experience of using Erlang and JavaScript, while working on the Membase project (where Python is widely utilized). I really enjoyed doing my work, so it’s obviously possible to have fun with languages that are less well designed than Ruby.

However, I always wanted to understand why this happens. Why do people choose other languages than Ruby so often? That is why I was excited to accept the invitation to the “Ruby vs. Python” meetup that was sponsored by our company. I chose the topic “Why Ruby Is a Brilliantly Designed Language” and wanted to summarize what makes Ruby different.

The Holy War

The event was very dynamic and intriguing. The speakers and attendees were split into two groups: one defended Python, while another fought for Ruby. Each group prepared presentations describing history, community, main concepts, development features, and perspectives of their language of choice.

I was a part of the group that presented Ruby. It was my first public speech in many years. My previous presentation—the defense of the diploma—was a big failure. I’m glad that this time I’ve learned some lessons and wasn’t so bad. 

So, why is Ruby a brilliantly designed language?

With hindsight, it is obvious that I picked up a very hard topic. Squeezing my experience into a 15-minute speech was really tricky. While preparing for the session, I reinforced my belief that Ruby is the best designed programming language I’ve ever used. However, when it was just one day left before the meetup, I realized that I could bring in only the most essential arguments to prove that.

1. My first pro-Ruby argument was that, from the moment the language was created, there were no missing features to add and no design bugs to fix. With over 10 years of its history, Ruby remained—mostly—stable with only minor syntactic features added/tweaked.
2. Second, Ruby represents all powerful object-oriented programming (OOP) features, which it adopted from Smalltalk. It is a “pure” OOP language, where everything is an object. The pureness, open classes, method_missing, and other OOP features provide for a huge semantic power. Surprisingly, nothing game-changing was introduced to the programming language design over the last 30 years.
3. After that, I talked about what powerful features Ruby adds on top of the traditional form of OOP. For instance, mix-ins, a powerful alternative to multiple inheritance is a great invention. During the presentation, I demonstrated how it makes functional programming even more natural than in LISP. Another remarkable Ruby feature worth mentioning was the ability to extend individual objects. It really makes defining “static” methods natural and can be useful in some other cases.
4. I concluded with touching upon Ruby’s concise syntax. As a part of this argument, I demonstrated Ruby’s syntactical killer-feature – blocks.

Revealed pros and cons

There were some other nice discussions during and in between the speeches. For example, the Python group argued that its language is more widely used. However, I noted that one could resort to such argument only because of the lack of other worthy pro-language arguments.

Another controversial point was that Python is good for education and is adopted by Massachusetts Institute of Technology as the primary programming language for students. I objected that Python is not the best language to start with. I think it is wiser to begin developing your programming skills with a pure OOP language, such as Ruby, because it can build a better solid ground for object-oriented design and way of thinking.

The Ruby group was also presented by Aliaksandr Rahalevich, Altoros alumni, who spoke about how Ruby allows him to design great APIs, and Hleb Stiblo, who presented an “Introduction to Ruby.” At the end, my university classmate Pavel Gabriel called upon pro-Python and pro-Ruby folks to summarize their arguments and language trivia.

The discussion ended up with the confessions from both sides about what sucks in their language of choice. Python 2.6/3 incompatibility was the biggest issue that Pythonistas admitted. Ruby developers noted the lack of speed, the lack of support for real concurrency, and cumbersome multiple encodings support in Ruby 1.9. The first two negative points were well understood by Pythonistas, since they suffer from them, as well. Ruby’s incompatibility of 1.9 vs. 1.8 was also mentioned, although nobody said that the difference between these versions is far less than the difference between Python 2.6 and Python 3.

I really enjoyed attending this event and making a presentation there. However, it seems like everyone—including me—forgot to mention that it is hard work of human beings and their attention to details which mostly contributes to an end-result. I think that as long as the technology doesn’t actively interfere with the programmer’s job, it affects the end-result very slightly. The language is merely the tool, while the engineer is the master.

How to use time travel for testing javascript libraries

The project that I'm working on right now is Membase. It is and was awesome experience in many ways. One cool part of that project is that I'm allowed to do quite advanced stuff for membase UI. Part of that is Cells library that is growing as part of Membase UI effort. But this post is not about cells. I promise to write about cells later. This post is about mocking JS clock for testing cells.

The code is here. You're interested in files named: mockTheClock.js and wrapped-date.js. It is inspired by equivalent code in JSUnit, but it's more full featured and more efficient. In particular I'm using binary heap to order pending timers.

The code is Apache-licensed, so feel free to grab and re-use it for your own projects.

News from the past — gpicker 2 is out!

I've released gpicker 2.0 more than half year ago. And I planned to write about it since then. Lack of time and lazyness delayed that more and more. But some people outside of my immediate 'circle of influence' are now using it for some very creative things. And I cannot wait anymore. It's time to realize my plan at last! :)

So gpicker 2.0 is out and it has some nice improvements. All new features are detailed in README file. But I'd like to describe the most important things here.

When I announced gpicker, the project didn't even had a home. It's official home now is here: http://savannah.nongnu.org/projects/gpicker.

gpicker became even faster. Loading and filtration were moved into separate threads, so that UI is always ultra-responsive. With speed improvements and separated filtration (which nicely aborts/restarts when user changes filter) it's now very responsive even on my whole filesystem which has more than 1E6 files.

supermegadoc and erlang docs

Perhaps the most significant improvement is ability to select from arbitrary list. gpicker is very good at selecting files, but you can feed it list of arbitrary strings via standard input. And I have quite interesting use of that feature – supermegadoc. This thing indexes man pages, ri, devhelp & erlang documentation and allows you to quickly choose and open documentation you need. But it's also quite useful for exploration of documentation. You can try typing various words and see if there's anything like that in documentation. I found it especially helpful for getting up to speed with Erlang standard library. I also did some very basic imenu & tags completer for Emacs which is based on this feature too.

Another area of improvement is gpicker integration. VIM integration (contributed by Sergey Avseyev) is included in gpicker distribution. And we also have plug-ins for netbeans (by Sergey Avseyev) and for gedit  (by Yury Tolstik).

We also have some support for building/running on OSX. But unfortunately it's not very easy (with limited Internet & OSX access I have) to get proper installation of Gtk+ on OSX. The canonical way seems to be building from source via jhbuild. But gpicker-on-OSX work was done against ancient distribution of Gtk+ which is long withdrawn from web. So any success/failure reports with proper Gtk+ installation as well as contributions in this area are very much welcomed. I expect, that only minor configure.ac fixes should be required at most. Contributing pre-built osx binaries is very much appreciated too. This can be done by Bundler.

Yet another improvement is ability to build gpicker without Gtk+ and glib. This flavor of gpicker doesn't have any GUI, obviously. But it's able to filter standard input against filter specified via command line, which should be enough for some useful things. Some people, for example, would like to have gpicker-like functionality with text-mode Emacs. gpicker-simple can help here. In fact I did some very preliminary (and buggy) integration of gpicker-simple filtration and IDO user-interface. If anyone is interested in making this work, please contribute. You can find it in gpicker.el from gpicker distribution.

The creative work by other people I mentioned before is a very interesting idea of switching windows by auto-completing their names (and other info) in gpicker. See Switch Window Using Fuzz Matching :: DoIT for more info.

I'm very glad that I can say thank you to people that helped with code (Sergey & Yura) and/or bug reports/suggestions (folks from Ruby department at Altoros). I'm happy that I have some users outside of Altoros. In particular, Marcelo De Moraes Serpa has been very helpful with suggestions and spreading the word. Thank you very much guys!

I thought that gpicker would take over the world, it's awesome after all, but my (not evil at all) plan utterly failed. This is a bit unfortunate :) But I'm very glad that gpicker is slowly getting some adoption. The future is bright!

My life with Debian

Tribute

I'm a long time Debian user. And I'm a happy Debian user. My current Debian installation is very (and I mean very, very) old. I don't remember exactly how old it is, but it's more than 6 years old. And it's probably as much as 9 years old. Since early months it was unstable branch of Debian. It is updated regularly, as often as my (quite poor at times) network connectivity allows.

When I'm getting new machine I simply copy my existing installation on new machine. And my particular OS has already been reincarnated from one machine to another for countless times.

I'm not trying to set any records here. I'm just lazy enough to avoid re-installing my OS and setting it up. But I seriously doubt than any other OS or distro can handle it. It's unique combination of Debian's approach to distro development, package upgrade-ability policies and attention to software quality that makes it possible.

So I owe a big thank you to Debian guys for their outstanding job during all this years. Thanks a lot!

Of course sometimes I had issues during package upgrades. It is inevitable when you run unstable and when you do, probably, as much as thousands of package upgrades per year. But I don't remember having anything really major. I was able to resolve all issues that I had.

Tips

During this years of running and constantly upgrading my system I've learned a couple of tricks to keep it in best shape. And I'd like to share them, though it's nothing really new.

Sometimes due to various reasons old versions of some libraries stay installed on your system. It's useful to run deborphan (http://www.debian-administration.org/articles/134) periodically. With new (well a number of years already) features of apt & aptitude it should happen less often, but some cruft can still accumulate.

Another trick that I discovered quite recently is package database de-fragmentation. It's described here (http://ubuntuforums.org/showthread.php?t=1004376).

Probably, the widest known tip is to do purge instead of just uninstall when removing packages. After normal uninstallation, Debian keeps config files. Purging package removes those too. Synaptic package management tool can be quite useful for purging any packages that where uninstalled in default mode.

I found another cleanup opportunity with package database. It turned out that dpkg kept information about uninstalled packages in is /var/lib/dpkg/available file. And during all this years this file has grown quite substantially (15 megs). I wrote simple script that cleans this up. And I've used it a couple of month ago. So far I don't see any bad effects of that. So I can recommend it for anyone with long lived and often upgraded Debian-based distro. Here it is.

amd64 versus i386

Another piece of knowledge, that I can share is that Debian is probably the only distro that more or less supports running 64 bit kernel with 32 bit user-space. I think, that's best combination. You get all 4 gigs of address space for 32 bit apps, you can run 64 bit apps, if you want, and you don't waste memory on twice as large pointers.

It would be great to have advantages of amd64. That's first of all larger register file and modern instruction set (i386 Debian still targets i386). But in my opinion, for typical desktop & developer machine larger memory consumption of 64 bit programs outweigh the benefits. In particular, most java programs really consume very close to twice as much of memory on 64 bit. And don't forget, that AFAIK there's still no lightweight 'client' JIT for amd64. Larger memory consumption causes less cache hits and more memory bandwitch, so amd64 is often slower. I also did some benchmarks with ruby & rails and found, that i386 version is faster.

So I've decided to stick with 64-bit kernel and 32-bit user-space. (And avoid re-install yet again). I'm running this combo for around half of year. One thing that was a bit annoying after switch, is that uname reported amd64, which caused issues with most (all?) configure scripts. My simple trick, which relies on debian's default init is to put the following file in /etc/initscript


It makes sure that everything that's spawned by init has i386 personality. This fixed issues with configure scripts.

For some rare programs, that require kernel component and don't support mixed user- & kernel-space (virtualbox), I have amd64 version installed in chroot. This also helps with development.

Thats all. Keep your systems clean and efficient.

How to cheaply turn single machine into a cluster

For development of membase which is a distributed storage system, I often need to run it on cluster of machines. Luckily I have two machines at home and at work so with trivial use of rsync & ssh running cluster of two nodes was easy. But I sometimes need to run more than two nodes so I decided to find something cheap that allows me to run multiple true nodes even when I have single machine. This is more important now, because I'll be on business trip for next month. So I'll have only single machine at my disposal.

One approach is to use virtualization. I tried it around a year ago for some project. Starting complete OS just to run single application is a bit too slow, but that can be alleviated by use of snapshots. In practice this is too painful. Even with snapshots it's slow. And free software virtualization products either have it wrong (virtualbox) or buggy (kvm & qemu). Bridge networking is relatively slow to come up. And I remember having some networking issues when restoring from snapshot.

Yesterday I finally tamed 'virtualization' that's cheap and works. My approach is to use LXC which is Linux's built-in containers implementation. It supports network virtualization as it's core feature and it doesn't require separate root for it's instances. So with my current solution I can create large number of 'servers' all having shared filesystem, but different hostnames and network stacks. It's reliable, starts quickly and it's easy to kill.

One of the problems was that I needed to create virtual host-only network that connects host and all containers. The problem is that current implementation of macvlan link type doesn't support networking with host. I worked that around by creating virtual ethernet pair and linking macvlans to one side of it, while using other side as host's end. So far it works beautifully!

The main script is at the following gist: http://gist.github.com/459693. It takes care of allocation of ip address & hostname and simply runs provided command inside container. It also takes care to kill everything inside container when main command exits. This is useful for killing any daemons (e.g. erlang port mapper) that might still be running inside.

Here's what I added to project's makefile to launch multiple instances of membase:

'make lxc-run' starts one instance of membase inside container. 'make lxc-cluster' starts three instances in tabs of new terminal window.