{"id":514,"date":"2015-03-05T14:46:31","date_gmt":"2015-03-05T22:46:31","guid":{"rendered":"http:\/\/porkrind.org\/missives\/?p=514"},"modified":"2015-03-05T14:48:42","modified_gmt":"2015-03-05T22:48:42","slug":"perl-module-xs-configuration-is-hard","status":"publish","type":"post","link":"https:\/\/porkrind.org\/missives\/perl-module-xs-configuration-is-hard\/","title":{"rendered":"Perl Module XS configuration is hard"},"content":{"rendered":"

I wrote a simple little Perl Module<\/a> recently, and it reminded me how frustrating it is to get it all working.<\/p>\n

I’m not even talking about the .xs<\/code> preprocessor (xsubpp)\u2014that’s weird, but it’s fairly straightforward. Most contingencies are accounted for and you can make it do whatever you want, and in my case, the results were pretty minimal and beautiful in their own way<\/a>. No, I’m talking about actually building it and making it compile in a cross platform way.<\/p>\n

If you’re used to standard unix C source releases, you know you can just run .\/configure<\/code> and then make<\/code> and it’ll (probably) just work. Behind the scenes configure<\/code> is madness, but the principle it works on (feature detection by actually compiling things) is sound. My module is an interface to a small third party library<\/a>. I can’t count on the library being installed on the machine already, so I want to statically link against it. It includes a configure script for unix machines and some windows source that isn’t covered by .\/configure<\/code>.<\/p>\n

In Perl, there are two different ways to build your module: ExtUtils::MakeMaker<\/code> and Module::Build<\/code>. For pure Perl modules, I will only use Module::Build<\/code>, as ExtUtils::MakeMaker<\/code> seems too hairy. But for an XS module, I’m not sure. ExtUtils::MakeMaker<\/code> looked fairly configurable so I try that first<\/a>. It works very well for compiling XS Unix stuff, but because it builds a Makefile and lets you just drop your own rules in, it encouraged me to just call my library’s .\/configure<\/code> and then make<\/code> in its directory:<\/p>\n

use ExtUtils::MakeMaker;\n\nWriteMakefile(\n    # ... boilerplate stuff stripped for brevity...\n\n    INC               => '-I.\/monotonic_clock\/include',\n    MYEXTLIB          => 'monotonic_clock\/.libs\/libmonotonic_clock.a'\n);\n\nsub MY::postamble {\n'\n$(MYEXTLIB): monotonic_clock\/configure\n    cd monotonic_clock && CFLAGS=\"$(CCFLAGS)\" .\/configure && $(MAKE) all\nmonotonic_clock\/configure: monotonic_clock\/configure.ac\n    cd monotonic_clock && .\/bootstrap.sh\n'\n}\n<\/code><\/pre>\n
This of course worked just fine on my Mac.<\/p>\n
But it utterly failed everywhere else<\/a>. Ugh, there’s hardly any green there! Ok, so I focused on the Unix failures first\u2014I know this, I should be able to get stuff working. Turns out Linux wants -fPIC<\/code> on the library’s code because I’m statically linking against it and it will end up in my modules shared library “.so” file. Ok, so I can just unilaterally add -fPIC<\/code><\/a>:<\/p>\n
sub MY::postamble {\n'\n$(MYEXTLIB): monotonic_clock\/configure Makefile.PL\n    cd monotonic_clock && CFLAGS=\"$(CCFLAGS) -fPIC\" .\/configure && $(MAKE) all\nmonotonic_clock\/configure: monotonic_clock\/configure.ac\n    cd monotonic_clock && .\/bootstrap.sh\n'\n}\n<\/code><\/pre>\n
That works on clang on the Mac and gcc on Linux. I actually tested on my Debian machine. Everything should be great now!<\/p>\n
Nope<\/a>. Half the Linuxes are still failing, some of the Macs too, and I haven’t even addresses Windows yet. I discover that -fPIC<\/code> happens to be defined by ExtUtils::MakeMaker<\/code> on the appropriate platforms in the CCCDLFLAGS<\/code> make variable, and switch to it<\/a> thinking this should solve the remaining Unix problems.<\/p>\n
Then I start thinking about Windows. I boot up my Windows 8 VM where I have Strawberry Perl installed and try out my module. It fails utterly. I forgot\u2014you can’t just call .\/configure<\/code> in windows! Plus my library doesn’t even handle Windows in its configure script. I start thinking about writing my own Makefile rules to build it so I can drop the configure script completely. But I don’t like it. I’m going to need to detect which back-end the library should be using. I basically have to recreate what configure does, but in a Makefile. And I don’t know how much shell I can use in my rules and still work on Windows, meaning the detection is going to be a huge issue.<\/p>\n
So I decide to drop ExtUtils::MakeMaker<\/code> and use Module::Build<\/code> instead<\/a>. It’s actually pretty straightforward:<\/p>\n
use strict;\nuse warnings FATAL => 'all';\nuse Module::Build;\n\nmy $builder = Module::Build->new(\n    # ... boilerplate stuff stripped for brevity...\n\n    extra_compiler_flags => '-DHAVE_GETTIMEOFDAY', # We're going to assume everyone is at least that modern\n    include_dirs => 'monotonic_clock\/include',\n    c_source     => ['monotonic_clock\/src\/monotonic_common.c'],\n);\n\n# Add the appropriate platform-specific backend.\n#\n# This isn't as good as the configure script that comes with\n# monotonic_clock, since it actually tests for the feature instead of\n# assuming that non-darwin unixes support POSIX clock_gettime. On the other\n# hand, this handles windows.\npush(@{$builder->c_source},\n     $^O                 eq 'darwin'  ? 'monotonic_clock\/src\/monotonic_mach.c' :\n     $builder->os_type() eq 'Windows' ? 'monotonic_clock\/src\/monotonic_win32.c' :\n     $builder->os_type() eq 'Unix'    ? 'monotonic_clock\/src\/monotonic_clock.c' :\n                                        'monotonic_clock\/src\/monotonic_generic.c');\n\n$builder->create_build_script();\n<\/code><\/pre>\n
I figure out that I can abuse the c_source<\/code> configuration option. It’s supposed to be a directory where the source code lives and they search it recursively for “*.c” files. But it turns out if I pass a C file to it instead of a directory, the recursive search finds that one C file! So now I can add specific C files for the particular platforms. I now have something that compiles on my Mac, my Debian machine, and my Windows VM. Hooray! That covers everything. Finally!<\/p>\n
Sigh<\/a>. That’s worse than my last Makefile.PL based version! What’s going on? Ok, my c_source<\/code> hack is biting me. I noticed that it was helpfully adding -I<\/code> options for the sources directory to the compiler, but since I was passing in actual files to c_source<\/code> I was getting compiler lines like -Imonotonic_clock\/src\/monotonic_clock.c<\/code>. This was a warning on my Debian gcc and on Windows gcc, but my Mac’s clang just ignored it with no message at all. So I blew it off. Well, it turns out other compilers are more strict.<\/p>\n
So I start poking around the Module::Build<\/code> source code. I discover that the c_source<\/code> is adding to an internal include_dirs<\/code> list. So I override the compile function to strip .c files out of the include_dirs<\/code> list.<\/a>:<\/p>\n
# This hacks around the fact that we are using c_source to store files, when Module::Build expects directories.\nmy $custom = Module::Build->subclass(\n    class => 'My::Builder',\n    code  => <<'CUSTOM_CODE');\nsub compile_c {\n  my ($self, $file, %args) = @_;\n  # Adding to c_source adds to include_dirs, too. Since we're adding files, remove them.\n  @{$self->include_dirs} = grep { !\/.c$\/ } @{$self->include_dirs};\n  $self->SUPER::compile_c($file, %args);\n}\nCUSTOM_CODE\n<\/code><\/pre>\n
It seems really hacky (I hate having to write Perl code in a string) and a bit fragile (I sure hope they don’t change the API in a new version) but it actually works! I publish that and wait to see my wall of green.<\/p>\n
I don’t get to see it yet<\/a>. This is baffling to me. The worst part is the test reports themselves<\/a>. They aren’t showing me the build process, and they don’t try to identify the Linux distro, leaving me to intuit it from the versions of various things. It looks like one of the failing distros is Debian Wheezy (aka the current “stable”). So I as a last ditch effort use “debootstrap” to build a minimal install that I can chroot into. I try compiling and I can actually reproduce the error. This is phenomenal because I don’t have to guess any more.<\/p>\n
After poking around for a while I discover that in older glibcs, the clock_gettime<\/code> function that my library uses requires librt<\/code> and therefore a -lrt<\/code> option to the linker. Ok. But I don’t want to add that unilaterally\u2014I got bit by that earlier in this process. I’m also frustrated that my backend detection code is just hardwired to Module::Build<\/code>s os_type()<\/code> function. Do I really know that old FreeBSDs support clock_gettime<\/code>? No, I don’t, and I don’t want to figure it out. I think back wistfully to .\/configure<\/code>\u2014it just detects stuff by compiling it and seeing if it works. That’s really what I want\u2014it’s the only way to know for sure without researching and testing on every. single. platform.<\/p>\n
And then it hits me, I guess I<\/em> could do that. Module::Build<\/code> uses ExtUtils::CBuilder<\/code> internally, and so I can too. It turns out to actually not be that bad<\/a>. The longest part of the code is redirecting stdout<\/code> and stderr<\/code> so you don’t see a bunch of compilation errors while it’s figuring things out:<\/p>\n
# autoconf style feature tester. Can't believe someone hasn't written this yet...\nuse ExtUtils::CBuilder;\nmy $cb = ExtUtils::CBuilder->new(quiet=>1);\n\nsub test_function_lib {\n    my ($function, $lib) = @_;\n    my $source = 'conf_test.c';\n    open my $conf_test, '>', $source or return;\n    print $conf_test <<\"C_CODE\";\nint main() {\n    int $function();\n    return $function();\n}\nC_CODE\n    close $conf_test;\n\n    my $conf_log='conf_test.log';\n    my @saved_fhs = eval {\n        open(my $oldout, \">&\", *STDOUT) or return;\n        open(my $olderr, \">&\", *STDERR) or return;\n        open(STDOUT, '>>', $conf_log) or return;\n        open(STDERR, \">>\", $conf_log) or return;\n        ($oldout, $olderr)\n    };\n\n    my $worked = eval {\n        my $obj = $cb->compile(source=>$source);\n        my @junk = $cb->link_executable(objects => $obj, extra_linker_flags=>$lib);\n        unlink $_ for (@junk, $obj, $source, $conf_log);\n        return 1;\n    };\n\n    if (@saved_fhs) {\n        open(STDOUT, \">&\", $saved_fhs[0]) or return;\n        open(STDERR, \">&\", $saved_fhs[1]) or return;\n        close($_) for (@saved_fhs);\n    }\n\n    $worked\n}\n<\/code><\/pre>\n
Using it is pretty easy:<\/p>\n
my $have_gettimeofday = test_function_lib(\"gettimeofday\", \"\");\nmy $need_librt = test_function_lib(\"clock_gettime\", \"-lrt\");\n<\/code><\/pre>\n
The one annoyance is that the feature detection doesn’t completely work on Windows. The technique I used (stolen unabashedly from autoconf) just declares the function with the wrong arguments and return value and tries to link with it. This works in general because C doesn’t encode the arguments or return values into the symbol name. Except Windows apparently does with its API functions: my feature detector detects gettimeofday()<\/code> just fine, but not QueryPerformanceCounter()<\/code>. If I declare QueryPerformanceCounter()<\/code> properly then Windows links with it, otherwise I get an undefined symbol error. I don’t care enough to properly research Windows’s ABI, so I decided to just check $^O<\/code>, assuming that all Windows will work. A quick check of the Windows documentation<\/a> reveals that it’s has been supported since Windows 2000 and that’s good enough for me.<\/p>\n
I upload to CPAN and finally<\/em> see that green I’ve been looking for<\/a>.<\/p>\n
But it makes me wonder, why did I<\/em> have to write this? It’s 2015, has nobody ever needed to build their Perl XS module differently for different platforms? I know I can’t be the first, but I had a really hard time finding any documentation or discussion about it. I didn’t see anything on CPAN that looked like it might solve my problems. Should this be a feature in Module::Build<\/code>? I’ve only written two XS modules in my life, but both times I needed different sources on different platforms. Does anyone want to point me to something that does this well, or failing that, build off this idea and make something neat? I would love it, certainly.<\/p>\n","protected":false},"excerpt":{"rendered":"
I wrote a simple little Perl Module recently, and it reminded me how frustrating it is to get it all working. I’m not even talking about the .xs preprocessor (xsubpp)\u2014that’s weird, but it’s fairly straightforward. Most contingencies are accounted for and you can make it do whatever you want, and in my case, the results … Continue reading Perl Module XS configuration is hard<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-514","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/posts\/514","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/comments?post=514"}],"version-history":[{"count":13,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/posts\/514\/revisions"}],"predecessor-version":[{"id":527,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/posts\/514\/revisions\/527"}],"wp:attachment":[{"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/media?parent=514"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/categories?post=514"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/porkrind.org\/missives\/wp-json\/wp\/v2\/tags?post=514"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}