My nightly Emacs builds stopped about a month and a half ago. A couple days after I noticed it was failing I tried to debug the issue and found that building openssl was hanging—I found that Jenkins was timing out after an hour or so. I should mention that it’s dying on a Mac OS X 10.10 (Yosemite) VM, which is currently the oldest macOS version I build Emacs for. I tried building manually in a terminal—the next day it was still sitting there, not finished. I decided it was going to be annoying and so I avoided looking deeper into it for another month and a half (sorry!). Today I tracked it down and “fixed” it—here is my tale…
I (ab)use homebrew to build openssl. Brew configures openssl then runs make and then make test. make test was hanging. Looking at the process list, I could see 01-test_abort.t was the hanging test. It was also literally the first test. Weird. I checked out the code:
#include <openssl/crypto.h>
int main(int argc, char **argv)
{
OPENSSL_die("Voluntary abort", __FILE__, __LINE__);
return 0;
}
Well, that seems straightforward enough. Why would it hang? I tryed to kill off the test process to see if it would continue. There was a lib wrapper, a test harness and the actual binary from the source shown above—they all died nicely except for the actual aborttest executable. I couldn’t even kill -9 that one—that usually means there’s some sort of kernel issue going on—everything should be kill -9able.
Next I ran it by hand (./util/shlib_wrap.sh test/aborttest) and confirmed that the test just hung and couldn’t be killed. I built it on a different machine and it worked just fine there. So I dug into the openssl code. What does OPENSSL_die() do, anyway?
// Win32 #ifdefs removed for readability:
void OPENSSL_die(const char *message, const char *file, int line)
{
OPENSSL_showfatal("%s:%d: OpenSSL internal error: %s\n",
file, line, message);
abort();
}
Ok, that’s nothing. What about OPENSSL_showfatal()? Also not much:
{
#ifndef OPENSSL_NO_STDIO
va_list ap;
va_start(ap, fmta);
vfprintf(stderr, fmta, ap);
va_end(ap);
#endif
}
That’s just a print, nothing exciting. Hmmm. So I wrote a test program:
#include <stdlib.h>
int main() {
abort();
}
I compiled it up and… it hung, too! What?? Ok. I tried it as root (hung). Tried it with dtruss:
...lots of dtruss nonsense snipped...
37772/0xcaf1: sigprocmask(0x3, 0x7FFF5DD71C74, 0x0) = 0x0 0
37772/0xcaf1: __pthread_sigmask(0x3, 0x7FFF5DD71C80, 0x0) = 0 0
37772/0xcaf1: __pthread_kill(0x603, 0x6, 0x0) = 0 0
So it got to the kernel with pthread_kill() and hung after that. So I tried another sanity check: In one terminal I ran sleep 100. In another I found the process id and did kill -ABRT $pid. The kill returned, but the sleep was now hung and not able to be killed by kill -9, like everything else. Now I was very confused. This can’t be a real bug, everyone would be seeing this! Maybe it’s a VM emulation issue caused by my version of VMWare? I can’t upgrade my VMWare because the next version after mine requires Mac OS 10.14 but this Mac Mini of mine only supports 10.13. Sigh. Also, the Emacs builds were working just fine and then they suddenly stopped and I hadn’t updated the OS or the host OS or VMWare. Nothing was adding up!
As I sanity check, I decided to reinstall the OS on the VM (right over top of the existing one, nothing clean or anything). There was a two hour long sidetrack here with deleting VM snapshots, resizing the VM disk (which required booting into recovery mode), downloading the OS installer and finally letting the install run. But that’s not important. The important part is that I opened up terminal immediately after the OS installed and ran my abort() test:
$ ./test
Abort trap: 6
$
It worked! How about OpenSSL?
$ ./util/shlib_wrap.sh test/aborttest
test/aborttest.c:14: OpenSSL internal error: Voluntary abort
Abort trap: 6
$
Yay! But why?? I don’t actually know. Was it a corrupt kernel? A bad driver that got installed? (What drivers would get installed on this Jenkins builder?) I don’t feel very satisfied here. I’m quite skeptical, in fact! But it’s working. Emacs builds should start coming out again. And I can ignore everything again until the next fire starts! 🙂