Handling security bugs is not the Open-Source community’s strong point

A funny story

Here is a funny story. Earlier this century, I moved into an apartment with electrical heating, so that during Winter, electricity is free(*) while I am home. Over the course of several years, I helped simulate proteins folding onto themselves or look for the optimal Golomb ruler of order 27. Last year, I decided to teach myself SIMD assembly by hacking an scrypt cryptocurrency miner. This is something you cannot do with protein-folding or Golomb-ruler-enumerating software: the results of these are centralized and the central authority, benevolent as it may be, does not trust you not to mess up the software if you got access to the source code. With cryptocurrencies, it’s every man, woman or fish for themselves, so if you can write a better miner, good for you. And if you introduce a bug and send bad results, you can trust your pool to promptly ban you (the protocol between pool and miner is designed to allow the pool to check the validity of proof-of-work submissions by miners, if only because rewards are distributed pro rata of the work done). Earlier this year, I downloaded the source code of a client and started tightening things a bit.

 Earlier this week, while my eyes rested on a bit of code that was not part of the computational core, what looked like a serious security bug stared back. A malicious pool could send extra-long strings to a miner, who would, at the time of preparing a reply, incorporate these strings in its answer. The answer was prepared in a buffer of a fixed size, on the stack, and it did not seem like lengths were checked at any point. The buffer was large enough to contain any substrings of reasonable lengths, but the client did not check that the server only sent reasonable substrings to incorporate.

 (*) as long as it is 100% converted to heat, up to the point where the electric radiator’s thermostat remains off all the time

A bug report

Long story short, I reported the bug today. I only knew the author of the source code from their GitHub account, and GitHub only seems to have public bug reports, so my report was public, but who looks at bug reports for network-connected software on GitHub? Not the NSA I hope!

 The bug was fixed in 40 minutes, faster than the nearest Chinese restaurant delivers takeouts on Saturday nights. This is an incredible response time, so I am going to say it again: on a Saturday, 40 minutes after reporting the bug, a fix was available for anyone to use. The fix looks sound, although Miod Vallat would like to point out that the result of malloc should be tested.

 The problem now, of course, is the propagation of this bugfix. Cryptocurrency miners are Open-Source at its frictionless best. “Bazaar” does not even begin to describe the joyous mess of contributions. Anyone with an idea can fork an existing client, and many have. The code I started from was already a fork of a codebase known as “cpuminer”, and the version I used as starting point has also been used as the basis of video-card-targeting miners, which change the computing paradigm but reuse the network code. The ancestry goes all the way back to a Bitcoin-only client.

 My initial idea was that I would wait for 90 days, and then look where the bugfix had propagated and make a blog post about that. I started to worry that despite reporting the bug to the place where I had found the software, I had handled the situation carelessly, so I started to investigate the means at my disposal to have the issue registered in a CVE.

The punchline

This is when I found that I didn’t have anything to do: the vulnerability I had stumbled upon had already been made public in 2014. On the plus side, this meant the study I wanted to make three months from now about the propagation of security fixes in decentralized Open-Source software could be done immediately.

 So what is the current state of mining software, since a serious vulnerability was published three months ago? Until this afternoon, you could compile pooler-cpuminer from sources, using the absolute latest GitHub snapshot, and get software with a vulnerability that had been public for three months. If, last month or today, you google for a binary version of a miner for an Nvidia video card, you are likely to get an archive called something like CudaMinerOSX-2014-02-28, which predates the publication of the bug and appears to include it according to a quick disassembly:

$ otool -V -t ~/CudaMinerOSX-2014-02-28/CUDA-5.5/cudaminer.10.9
…
_submit_upstream_work:
00000001000031c0 pushq  %r14
00000001000031c2 pushq  %r13
00000001000031c4 movq   %rsi, %r13
00000001000031c7 pushq  %r12
00000001000031c9 pushq  %rbp
00000001000031ca movq   %rdi, %rbp
00000001000031cd pushq  %rbx
00000001000031ce subq   $0x190, %rsp
…
00000001000032ff leaq   0x3811a(%rip), %rcx     ## literal pool for: "{\"method\": \"mining.submit\", \"params\": [\"%s\", \"%s\", \"%s\", \"%s\", \"%s\"], \"id\":4}"
0000000100003306 movq   %rax, (%rsp)
000000010000330a xorl   %eax, %eax
000000010000330c leaq   0x20(%rsp), %rdi  ; First argument of sprintf
0000000100003311 movq   %r12, 0x10(%rsp)
0000000100003316 movq   %rbx, 0x8(%rsp)
000000010000331b callq  0x100039816             ## symbol stub for: ___sprintf_chk
…

The above ready-made binary is what I have been using to convert additional Watts to warmth. I downloaded a binary version because XCode no longer includes GCC and Clang lacks some critical component to compile CUDA code and computers are horrible. But would I have fared better if I had compiled from source? Apparently not, unless the lengths of work->job_id and xnonce2str are tested elsewhere.

Conclusion

Recent years have shown that the “many eyes” quip did not work for security bugs. One can see why it would work for functional bugs: with enough motivated users of the same software, you stand a good chance that the bug that could have annoyed you has already annoyed someone else who was able to fix it. This reasoning does not apply to security bugs. The one described here does not prevent any of the many clients to work with a pool that does not send 500-byte-long job ids.

 The anecdote in this post shows another bleak trend: not only Linus’ law does not apply to security bugs, but at its most disorganized, the Open-Source community does not seem to be very good at propagating, for known security bugs even, either fixes from source to binary distributions nor the information that a vulnerability exists from fork to fork.

 Ack: in addition to pooler fixing the discussed bug crazy fast and Miod Vallat pointing out flaws in pooler’s patch even faster, John Regehr once provided a link to David Andersen’s story that I cannot find again now. David Maison and Laureline Patoz helped with the general tone of the post.