smashing.org

I had a fantastic time in Chicago at YAPC::NA this past week. Kudos to the whole Chicago crew, including Josh, Pete, and of course Heather, for putting this together and putting up with all of us. The dorms were, well, dorms, but the beer more than made up for it.

My talk went very well, and ended right on time. I was hoping for a better turnout (I think about 40 people showed up), but I believe those that did were impressed with what's happening in the mod_parrot world. In addition to explaining the goals and architecture, I was able to demonstrate actual working PHP scripts and Perl 6 handlers.

My lightning talk was even better. Jerry went first and created his "lightning" language in 5 minutes. Then I got up and made a registry-style Apache module out of it, complete with live demo, in 5 minutes. I couldn't hear anything from the stage, but apparently there were oohs and aahs, and even a "did that just f*cking happen?", and that's a good thing for the mod_parrot hype machine. We'll hopefully do this again at OSCON, with a comedic twist. But not like Ingy's. ;-)

The speakers dinner was fantastic, and the view of Chicago and the lake from the 32nd floor was amazing. It was good to see a nicer part of the city, since IIT isn't in the greatest location. The fireworks were a nice touch too.

On a non-technical note, Allison, chromatic, Jerry, Coke and I kicked off the Parrot Foundation with a long trip to the bank, a quick board meeting and a public Q & A session. I'm both honored and excited to be a part of this, as I believe the future of Parrot is a bright one.

Next up, OSCON. And a hotel room with a bathroom all my own!

One of the more common uses of mod_perl is as a CGI accelerator. The dirty work is done by ModPerl::Registry (or Apache::Registry in mod_perl 1.0), which emulates a CGI environment, loading and executing CGI scripts while caching them for future requests. You can also use Apache::RequestRec and friends in your scripts to access more data about the request, set headers, and do many other interesting things.

Up to this point, mod_perl6 has been lacking support for registry-style scripts, and all my examples have been handlers for various Apache phases. My goal was to remedy that in time for YAPC, and this week I succeeded. Rakudo needed the following things to make it work:

• basic eval support
• nested subroutines
• interpolated namespaces
• the ability to capture stdout (via ties, redirection, or something else)

Eval support was easy -- it magically appeared in r26963 in Parrot. It doesn't yet support passing the lexical scope of the caller, but we don't need that. Nested subroutines also worked out of the box.

Interpolated namespaces required some work. They give us the ability to call a subroutine in a namespace defined at runtime, such as ::($mynamespace)::mysub(). Fortunately I did most of this work to support pure-perl mod_perl6, and had to tweak it for recent changes in Rakudo's calling conventions. Since it only supports subroutine calls right now, I am not committing this change to Rakudo just yet, as I'd like to investigate a more generic implementation first.

Capturing stdout required some deep thought, and that can be good or bad, depending on how much coffee I've had. Fortunately the coffee was plentiful the day I was working on this. mod_perl 2.x captures CGI output using filehandle ties and other magic. We don't have that yet in Parrot. I proposed writing an IO layer to do what I needed, but at some point both chromatic and Allison noted that IO layers were going away or being completely reworked, so I threw that idea out. I decided to use what we had already, the string IO layer. Yes, it's going away eventually, but I can very easily plug whatever replaces it down the road. The string IO layer works by capturing output to a file descriptor in a string. The upside is that the implementation is simple. The downside is that, at least in mod_perl6, you can't dump the output until the script is complete, meaning you have to store the entire output of the script in memory, possibly twice. But hey, this is a first pass and I really don't care, so I added some methods to capture stdout to mod_parrot, which any language can use.

With all the prerequisites satisfied, I was able to produce a very simple pure-perl ModPerl6::Registry, which loads each script into its own namespace and caches it for future requests. Here's a snippet of the juciest part of the module, the mod_perl6 handler:

module ModPerl6::Registry;

our %registry;

...

sub handler($r)
{
    my $script = $r.filename();
    unless (%registry{$script}) {
        my $data = load_script($script);
        my $mod = gen_module_name($script);
        my $code = "module $mod; sub _handler { $data }";
        eval $code;
        %registry{$script} = $mod;
        %registry{$mod} = $script;
    }

    my $interp = ModParrot::Interpreter.new();
    $interp.capture_stdout(1);
    my $mod = %registry{$script};
    ::($mod)::_handler();
    my $buf = $interp.dump_stdout();
    $interp.capture_stdout(0);

    $r.puts($buf);

    0;
}

The Apache configuration should look familiar:

Alias /perl6-bin/ /usr/local/apache2/perl6-bin/
<Directory /usr/local/apache2/perl6-bin>
    Options +ExecCGI
    SetHandler perl6-script
    ParrotHandler ModPerl6::Registry
</Directory>

ModPerl6::Registry is available in mod_parrot as of r345, but I have yet to commit the required patches for Rakudo, so they won't work for anyone yet. Expect the patches next week, and hopefully native support in Parrot sometime real soon now!

I'm finally all set for air and hotel for both YAPC::NA and OSCON. What a pain, with airfares as high as they are. Regardless, I'm looking forward to speaking, hacking, drinking a beer or six, and seeing everyone again! And yeah, maybe I should actually prepare a talk. ;-)

The OSCON schedule is finally out, and while I'm happy with my timeslot (Wed. 7/23 at 2:35), I'm disappointed to see it's at the same time as the Rakudo talk. First, it's likely that the mod_parrot and Rakudo audiences will have interest in both talks, and with the current schedule they will have to choose one over the other, or try to poke their heads into both. Second, I want to see the Rakudo talk too! Perhaps there's a chance we can be split up, though 2:35 is a nice time. I'll start asking around...

Followup: My talk is now scheduled at 10:45 AM on Thursday July 24th. Thank you Patrick and Allison!

My talk, mod_parrot: One Apache Module for Many Languages, was accepted for OSCON this summer. I know there were several other Parrot and Perl 6 talks submitted, and it would be great to show off all the progress we've made in the past year. I've also submitted the talk for YAPC in Chicago, so this could be a busy conference season!

Having worked for three startup companies in my career, of my favorite shows this winter has been Startup Junkies on MOJO. It documents the daily trials and tribulations of running a startup company, in this case Earth Class Mail. Their service is both useful and innovative, the staff and management seem capable, and the show is entertaining to watch, at least for me.

However, something in the latest episode really irked me. Earth Class Mail was planning to announce their partnership with Microsoft at a conference, and was scrambling to port their software to .NET in time for the conference. Okay, that's fine. I have no beef with .NET or Microsoft, and I usually don't question anybody's choice to use those platforms. However I found the following statement from Ron Wiener, CEO of Earth Class Mail, very disturbing:

...we launched our platform on open source code, which was free, and we knew that later on we're going to need to be able to scale -- we're going to need to move to the Microsoft .NET plaform so that we could handle millions of simultaneous users.

Could this be a more obvious shill for Microsoft? What's worse though, is that this statement promotes more of the usual FUD about open source, implying that it can't scale. Somebody forgot to tell Yahoo! and eBay and Google. And all the other companies we work for that use open source. Shame on you, Ron. I'll still watch the show, but I'm keeping my eye on you!

I'm back home from Minneapolis, and I'm thawing out from Frozen Perl, with the temperature a balmy 64 degrees in Philly today. I'm sure we'll be snapped back to reality soon enough, but I'm enjoying it while I can. Frozen Perl was a very well run workshop, especially for an inaugural event. Many thanks to Dave Rolsky and all of the other organizers. And to Chris Prather (perigrin), for his Parrot/Moose patch and a leisurely ride to the airport. :)

My talk went very well, as did Allison's, and I think we garnered a lot of renewed interest in both Perl 6 and Parrot. When all of your live demos work, you know it's a good day!

Thankfully, it's a holiday today. It's been a long weekend.

Yesterday I committed a new version of Parrot's embedding docs, now available at http://www.parrotcode.org/docs/embed.html. Most of the document is based on my work with mod_parrot, and reflects the current state of the embedding API rather than any formal spec. Much more to come as I start to verify code coverage, add missing docs, fix missing prototypes, and tweak whatever else creeps up.

I'm pleased to announce that after 2 years of hiatus and 6 months of coding, mod_parrot 0.4 has been released. It's been worth the wait, as we now have working mod_perl6 and PHP proof-of-concepts, per-server interpreter pools, the framework for a more comprehensive test suite, and even 64-bit support. You can download the full distribution at http://www.smashing.org/mod_parrot/dist/mod_parrot-0.4.tar.gz. The comprehensive list of changes is below. Enjoy!

0.4    January 20, 2008
       * Support for Parrot 0.5.1 (Hatachi)
       * Many fixes for 64-bit platforms
       * Begin transition to server-side tests
       * Keep testing infrastructure on make clean (versus distclean)
       * Update PIR syntax and types that have changed since mod_parrot 0.3
       * Convert classes to new object model
       * Convert to nested namespaces
       * Remove apxs from compile and link phases to add flexibility
       * Search Parrot runtime path for mod_parrot init code
       * Remove underscore prefix from HLL handler subs
       * Map Apache MIME types to HLLs
       * Map Apache handler names to HLLs
       * Initialize interpreter at the earliest possible Apache phase
       * Per-server context pools for virtual hosts (default is per-process)
       * Properly merge server configs
       * Can pass a PMC note via pmc_notes()
       * Implement mod_perl-like cleanup handlers
       * Use proto-objects make mod_parrot classes instantiable from an HLL
       * New ModParrot;Context class for accessing mod_parrot data structures
       * New ModParrot;Interpreter class for interpreter introspection
       * New TODO list replaces the outdated ROADMAP
       * Update documentation and examples
       * New example HLL handlers:
         - Perl 6
         - PHP/Plumhead
         - NQP
         - Perl 1/Punie
       * New example handlers in eg
       * New Apache directives:
         - ParrotIncludePath
         - ParrotAddType
         - ParrotAddHandler
         - ParrotOptions
           o Parent
           o Enable

smashing.org will soon be hosted on an 64-bit Opteron platform. As I'm planning to serve up a number of live mod_parrot examples on the site, I thought it prudent to make sure mod_parrot worked on x86_64. To my surprise, only one test failed, but boy was it a nasty little bugger.

mod_parrot uses NCI to call Apache's API functions, and you have to provide each function's signature so NCI knows how to pass the various types of arguments and return values between Parrot and Apache. Most of the time there is a one-to-one mapping between datatypes and all is well. If not, you need to write an NCI-friendly wrapper function to get mod_parrot and Apache talking.

The prototype for ap_log_rerror is:

AP_DECLARE(void) ap_log_rerror(const char *file, int line, int level,
                               apr_status_t status, const request_rec *r,
                               const char *fmt, ...)

I planned on letting the HLL layer format the message so mod_parrot would only have to pass a single string as the log message. So in a perfect world the last two arguments would always be "%s" and one string in the vararg (...). A constant number of arguments, and all data types supported under NCI. Perfect! Here's how I coded the NCI signature:

dlfunc func, nul, "ap_log_rerror", "vtiiiptt"

That says the function ap_log_error returns nothing (void), and accepts a string (t), 3 integers (iii), a PMC pointer (p) and two strings (tt).

Flash forward to today, and I'm faced with a test that fails only on x86_64. Not only that, but it's failing randomly and smashing the stack along the way so I can't even generate a reliable backtrace!

Then I got to thinking...what touches the stack? va_start(), va_end(), and all their little varargs friends! I immediately remembered reading somewhere that NCI didn't support passing args to variadic functions (though chromatic was threatening to work on it). I assumed that this was still the case and the fact that it worked for me on 32-bit x86 was pure dumb luck.

I recoded things so NCI calls a wrapper function, which then calls ap_log_rerror from C, where varargs are handled appropriately. I'm really glad this wrapper infrastructure was already in place!

The NCI declaration now reads:

dlfunc func, nul, "mpnci_ap_log_rerror", "vJtiiipt"

The C wrapper looks like this:

void mpnci_ap_log_rerror(Parrot_Interp interp, char *file, int line,
    int level, apr_status_t status, request_rec *r, char *msg)
{
    ap_log_rerror(file, line, level, status, r, "%s", msg);
}

And it all works! Let it be known that mod_parrot is now 64-bit friendly. Full credit goes to the x86_64 platform, which exposes everyone's stupid programming mistakes. Especially mine.