//log//

For quite a while now we’ve been using Jython, a (re-)implementation of the Python programming language that runs on top of the Java VM and along the lines of JSR-223 (Scripting for the Java Platform), in our environment to deal with some use cases which addressed way easier using a scripting language than, in example, using Java code inside a webapp. However, so far our integration left a lot to be desired, so it was about time to get this things improved somehow.

Reasons to use scripting

In our environment, there are few general yet important use cases for dealing with (P|J)ython in our application:

• Solution prototyping: In some situations, lacking reliable and detailed information about the actual problem to be solved, we start out building small prototype implementations of features more or less from scratch, but in most of these situations, these solutions still need to be part of our application platform so they need to, in example, work within the context of a Java EE web tier application. From this point of view, as these features usually need to be accessible by (external) customers pretty fast, integrationg Jython and using Jython scripts in prototyping stage has proven to be rather effective as it works well without having to (re-)deploy or otherwise mess with the Java web application running inside the Glassfish 3.1 application server. It’s simply a pretty straightforward way of getting closer to a reliable conception of something to by then be implemented in Java (beans, servlets, …).
• Dynamic extensibility of some features: In other situations, we use Jython scripts in a DSL-like way to allow for customizing small, dedicated parts of the overall system behaviour in an “almost declarative” way. This also has proven to be rather effective as, in most of these situations, these customizations also are subject to frequent changes and, then again, aren’t generic or widely-used enough to justify a more robust implementation. In most cases it’s a modification specific to a certain customer project which ain’t of any use anymore as soon as the project has ended.
• Introducing “quick hacks”: Though somewhat similar to the prototyping approach, the “quick hacks” use case yet is in some way different as this exclusively deals with tools, features, … used by internal developers (or better DevOps) to address a special need in a certain situation. Integrating part of our internal domain-specific logic with nagios2 for monitoring purposes comes to mind here. Linking these things to nagios generally seems somewhere in between hacking nagios2 plugin code and linking this to the internals of the business application in a meaningful way. Being able to do so at server runtime without having to stop/redeploy the running Java webapp itself is beneficial without a doubt.

Of course, this does come at a price, and there are quite a bunch of fairly good reasons to think twice before mixing languages and technologies in such a manner. Always being in need of developers familiar with “both” languages and language environments immediately seems to be a drawback here. Or knowing that the Java EE module lifecycle (including deployment / redeployment) is there for good reasons and not just to make developers life more difficult. Or maybe also knowing that dynamically (re-)loading code at runtime generally is a good way to put the stability or availability of the system as a whole at risk. It’s something that, like any other technology at hand, needs some discipline or regulation, and be that by drastically limiting the amount of people who may use it. But if done so, it’s a rather powerful tool.

Annoyances

By now we have been using a bunch of Jython scripts already, and it works out rather well, but introduced few annoyances, too:

• At first, we use to deploy the jython.jar all along with the (maven2 built) webapp to have these things put together in a slightly reproducible way. Using maven, the build process needs / wants to find the Jython module in some maven repository. Unfortunately, the latest version of Jython to be available in public maven repositories so far seems 2.5, whereas JSR-223 functionality was just introduced with 2.5.1. So, it requires a manual interaction with the (local) maven repository in order to make things out.
• Our first integrations worked well using basic Python stuff and accessing Java classes but failed to integrate with most of the Python (core) libs installed on the machine, starting with Pythons very own os module. This required some fixing, too.
• Finally, though we were easily able to import Java classes in Python scripts, so far we failed to make new Python libraries accessible / importable (in example simplejson which we wanted to use for one special purpose). This also ended up to be a bit annoying as, after all, if running [P|J]ython, you also would want to have most of its libraries at hand, won’t you? 😉

Mavenizing Jython

The first of our problems, including a recent version of Jython with our maven2 build process, got resolved rather easily and in a pretty straightforward way. Though not really nice, an antrun plugin definition does the trick rather easily, assuming there is a maven wrapper artifact (of type jar) that contains jython.jar in src/main/resources/:

    
        
            
                maven-antrun-plugin
                
                    
                        package
                        
                            run
                        
                        
                            
                                
                            
                        
                    
                
            
        
    

We do so in quite some situations to deal with binary code not available in any public maven repository. Though, to state it again, this is thoroughly clumsy (not to say ugly), it so far has proven to be a rather good way to seamlessly include binary jars into a larger set of maven projects easy for everyone to see where artifacts come from. Only real drawback this solution brings: Having such a wrapper artifact opened all along with artifacts depending upon it in an IDE like Eclipse is likely to cause trouble and make the IDE complain about missing classes simply because, even though there’s binary stuff, the project doesn’t actually contain any classes visible to the IDE. Still gotta resolve this, but that eventually is another story.

Fixing os and decimal

This issue, after spending a while reading and looking at the Jython installation base, also didn’t take too long to fix: We learnt that, in our initial installation, we just used the jython.jar which is installed after downloading and installing the Jython distribution, which obviously doesn’t contain the $JYTHON_HOME/Lib folder and, thus, does miss quite a bunch of essential Python core classes and libs. Reading through the Jython installation guide once again, we got to know the “standalone” installer which, in the end, created a larger jython.jar – including the whole Lib/ folder as well. Seems a typical case of “it will immediately work as soon as you do it right”. Throwing this .jar into the maven2 process made all of our issues related to exceptions in import os Python statements successfully go away and allowed us to use additional third-party Python libraries inside our scripts…

Adding new Python libs

… as soon as we finished the last part, the configuration of the library search path for the Jython deployed to the webapp. Usually, Python makes use of the PYTHONPATH environment variable which resembles the classpath known to the Java programmer and is a set of folder specifications where the Python interpreter will look for modules / libraries to import. Jython, for these purposes, has a python.path property in its configuration which works just the same. python.path usually is configured in the Jython registry which it tries to find in various places before proceeding.

Having jython.jar deployed with the webapp however, there wasn’t an obvious place where to put the registry file and, subsequently, where to store down the python.path configuration we wanted in order to make our application work well with external libraries. In the end, we decided to follow a rather pragmatic approach and set python.path programmatically at runtime. In our case, this is easily possible because most of the scripting logic is wrapped in one or two support / tooling classes to not have the (Jython-dependent) scripting code spread across the application too much. We then introduced a configuration member extPythonLibraryPath to these classes and made them add this one to the python.path property like this:

        PySystemState.initialize(PySystemState.getBaseProperties(), null, null);
        PySystemState sys = new PySystemState();
        if (!sys.path.contains(Py.newString(extPythonLibaryPath))) {
            sys.path.append(Py.newString(extPythonLibaryPath));
        }

        Py.setSystemState(sys);

Summary

What’s the overall outcome of this? Well, we managed to include [P|J]ython more smoothly with our current project environment. The setup itself is stable enough to easily add third-party Python libraries without too much ado as well as to easily make Python scripts modular by implementing parts of the scripting code in reusable functions or classes. It’ll now be all about making sane use of this in order to straighten up scripting code a little, in order to make things more extensible without making them less maintaineable. More to come.

Filed under: