Out there in the wild, when it comes to talking about Pentaho, the first impression people have: Its large, its big, it's heavyweight, it's THE SERVER. Its a beast that eats CPUs at night and during the next day administrators make barbecue on the remaining CPUs.

You can really stun people by revealing the well-hidden secret, that Pentaho the Platform is not heavyweight at all. Quite the contrary, if used correctly.

My journey on that started a couple of years ago, when I had to debug the reporting integration in the platform. This was the first time in my life, I actually contemplated about retiring as a farmer somewhere in the north-american desert (not many plants, true, but one or two interesting exist). With every compile, redeploy and then restart of the heavy weight JBoss server, I wondered: Have I sinned that much to deserve that hell? I'm sure Dante Alighieri had a 10th circle in his Divine Comedy, which involved lots and lots of J2EE and JBoss-debugging. But obviously he considered that to cruel to be believable, and therefore cut it out. Even Satan would not be so low ...

Obviously, as I'm sitting here and writing this article, I found salvation beyond Peyote or eternal torture. And salvation is to simply not to use JBoss* (or any other J2EE system).

Revelation One: The Pentaho Preconfigured Installation is Not the Pentaho Platform

The official documentation and whitepapers make a fine (and really not obvious) distinction between the "Pentaho Platform" and the "Pentaho Server". The server is the big and heavy-weight thing, and actually relies on a working J2EE infrastructure to run. So from now on, we will ignore this big pink elephant. The interesting pearl is the Pentaho Platform buried in the Server.

The Platform is a couple of JARs, which are almost entirely made of of infrastructure and glue code. The Platform only has one purpose: To orchestrate the various components into a BI related symphony. If I would have to describe the platform in one single sentence, it would be: "The platform is a runtime environment for a XML-based process-language that additionally provides auditing, logging, configuration and other infrastructure needs to allow to run BI-Jobs."

Revelation Two: The Platform does not require J2EE at all.

Now we are leaving the heavy-weight area and dive deep into the sacred land of resource-efficiency! Although the platform provides several implementations of its interfaces to integrate seamlessly into J2EE environments, at the same time it also ships with implementations that are not tainted by any J2EE related code. These clean implementations make it possible to integrate the platform into all kinds of Java-Applications.

For me, running the platform outside of a J2EE server allows me to debug the components I write from inside my IDE. I do not have to deal with a heavy-weight server that starts up and shuts down in 5 or more minutes. I do not have to dive through layers over layers of application server code before I come to the parts of the application that interest me. I do not have to deal with HTTP requests. I do not have to deal with configuring a server before I can work. I can start my work immediately.

When I have to deal with XActions and have to find out why the $%&&$ the thing is not working, I also tend to be faster to simply attach a debugger and see whats going on under the covers instead of performing an pen-and-paper analysis of the XAction file itself. Run, listen for the crash, jump to the crash, and search the burning ruins for hints on what happened. Fast and simple and since I started using the platform as embedded tool, I never had to deal with setting up JNDI datasources in JBoss or any other J2EE system and I never had to write a single XML-deployment descriptor again. This is how heaven must be like.

But having the platform as a embedded toolkit opens a whole new world of opportunities. Maybe you have to provide bursting capabilities (that is: generating lots of reports and sending them out to a predefined list of recipients. Much like what spammers do daily but clean and family friendly) then the platform can do this for you with minimal efforts. Maybe you need to integrate reporting in your application and at the same time you have to ensure (and prove later) that the reports have been generated and have been distributed correctly. Or in an extreme case you need to query a web-service to provide parameters to query on a OLAP server to feed a Kettle transformation to run a sequence of reports that are distributed via email, then the embedded platform allows you to run that XAction as easily as a simple report itself.

Revelation Three: Code!

Up to the Platform 1.2.0, there was a sub-project called the Pentaho-SDK, which contained a couple of examples on how to execute XActions in the standalone mode. A SDK on a OpenSource project (where the full sources are always available) was some sort of strange beast, so this project ceased to exist and only the SVN server knows where it's spirit went. However, the death of the SDK cut of the audience that just wanted to run the platform and who did not want to deal with all the code of the platform.

So here we start again.

(1) Setup the project

• Grab the latest sources and copy all JARs from "thirdparty/lib" and all its subdirectories and copy them into your project's lib directory.
• Add all the jars to your projects CLASSPATH.
  
• Build the platform and add the generated JARs to your classpath as well.
• Grab a configured copy of the solution directory.
• Configure JNDI so that the components know how to access the database(s).
• (Remove BIRT from the system-listeners, as it does not seem to initialize in standalone mode.)

or

• Download the preconfigured standalone environment :) (scroll down)

(2) Java: Initialize the platform.

Initializing the platform is easy, all you have to do is to provide a standalone-context and point it to your solution-directory.

  public static boolean initialize()
  {
    try
    {
      // We need to be able to locate the solution files.
      // in this example, we are using the relative path in our project/package.
      final File solutionRoot = 
          new File("/home/src/pentaho/pentaho-demo/pentaho-solutions/");
      final File applicationRoot = new File("/home/src/pentaho/pentaho-demo/");
      final StandaloneApplicationContext context =
          new StandaloneApplicationContext(solutionRoot.getAbsolutePath(), 
		applicationRoot.getAbsolutePath());

      // Initialize the Pentaho system
      return PentahoSystem.init(context);
    }
    catch (Throwable t)
    {
      // of course, you should have some better 
      // error handling than I have ;)
      t.printStackTrace();
      return false;
    }
  }

(3) Execute your XAction. The XAction-Path should be relative to the solution-repositories root-directory. The parameters must be given in the HashMap and must match the declared parameters of the XAction. By adding more code it is possible to provide a UI on top of this process that queries the parameters in the same way as the Pentaho-Server's HTML-UI does it.

    final String xactionPath = 
        "samples/steel-wheels/reports/Income Statement.xaction";
    final HashMap parameters = new HashMap();
    parameters.put ("output-type", "pdf");

    final FileOutputStream out = new FileOutputStream ("/tmp/report.pdf");
    try
    {
      ISolutionEngine engine = SolutionHelper.execute
          ("Just a description used for logging ", "User (only for logging)", 
		xactionPath, parameters, out);
      List messages = engine.getExecutionContext().getMessages();
      engine.getExecutionContext().dispose();

      // out contains whatever the XAction produced.
    }
    catch (Exception e)
    {
      e.printStackTrace();
    }
    finally
    {
      out.close();
    }

(4) Clean up. Always shut down the platform before you exit the application. You want to be sure that all data is written into the databases and that all buffers are flushed.

    PentahoSystem.shutdown();
    System.exit(0);

So go ahead, download the package and start walking the lightweight path.

• Pentaho-Standalone (ZIP-Package)
• Pentaho-Standalone (TAR.GZ-Package)

* Nitpickers corner: JBoss as used in this article actually represents all the evilness found in all J2EE servers. No matter whether you choose JBoss, WebSphere, BEA or whatever J2EE-Servers you prefer, they are heavyweight machinery and not meant to be used for developing applications. Once you are finished developing, they surely form a superior runtime environment for your J2EE code, but everything that makes them good in production makes them horrible for development. Slow startups, heavy footprint and lots of lots of XML descriptors - efficient development should look different than that.

Starting this year and continuing next year, Germany entered a series of state-government elections that will find its culmination in the federal elections next year. OK, given the fact that both major parties were to weak to dominate the politics and (fortunately) to stubborn to put their differences aside, the last three years were a low-intensity election campaign.

After the technocratic reign of Gerhard Schöder (who not only managed to trash the social security systems by granting tax breaks for the rich and boosting the replacement of ordinary jobs into a new low-wage job sector (low wage jobs are exempt from paying taxes and from contributing to the social security system), the Social Democrats got massively slapped for their right-wing politics. As the "social" party's actions were indistinguishable from the actions the conservative and neo-conservative parties would have done if in power, these parties got slapped in 2005 as well (for not only allowing but actually actively supporting this policy). With both major parties throttling each other, the government was finally busy just blaming each other than doing yet another reform. This way, the country at least got a break.

But with the formation of a new left-wing party to fill the gap the declining social-democratic and conservative parties, everything is open now. The new party already is strong enough to win a massive number of seats in the parliaments - and it's movement still growing. To the traditional parties that used to be in power since the end of the war, this possible loss of influence is horrible. Loosing 20% of their market share within two years is a shocking experience. And shocked people tend to react in strange ways. But listen:

Of course, (despite the fact that banks are crashing and the weak dollar is a deadly thing for the export-oriented German economy) we are at the beginning of a boom. (Remember: Its election time, so truth is one of the few things you can't expect from the propaganda system right now. If you'd only listen to the media coverage, you would have known that we have been at the beginning of the boom since the dot-com bubble crashed.) And so, officially, the number of unemployed people is as low as never before in the last 20 years. Right now, the official number tells that 3.413.921 are officially unemployed. This is the number you'll hear throughout repeated by politicans and media all the time. But reading the statistic tells a different story: 1.531.940 people were "employed" in so called trainings and employment incentives and therefor do not count as unemployed. The effect of these trainings is proven to be equal to nothing, but hey, we cannot afford the negative image of 5 million unemployed people.

(Compared to March 2005, the number of people receiving well-fare or unemployment support did not change much; it went up from 7.977.502 in March 2005 to 8.027.980 in April 2008. But we are in a boom! You just have to believe in it and it will become true.)

According to the report of April 2008 a total number of 8.027.980 persons depended on additional well-fare subsidiaries for their survival. Given the official 40 million people that form the German workforce, this means that 20% of the people out there either have no jobs or cannot live from the minimum wage jobs they have. Of course, Germany does not have minimum wage laws, as this would harm the market. Therefore in some regions where hourly rates of 2 to 3 Euro for full-time jobs have become common.

The social-democratic*  changes of the poor-laws already introduced lowest-wage work throughout the country. Nowadays, it is common (especially in the declining traditional industries) to find yourself replaced by temporary workers or subsidized workers from one of the many employment incentives. Why pay 15 or 20 Euro, if you can get workers that are forced to work (quitting or getting fired means you would not get well-fare support anymore) and that are guaranteed to not complain? And of course, these forced-labor workers have no union-rights, no job-security or any other protection a regular worker might enjoy.

The negative impact of that policy now starts to become widely visible. The costs for the social security explode, as the income generating high-wage jobs erode and cost-generating low-wage jobs grow like cancer.

In that situation and with a clear road to destruction ahead, what is the most stupid thing you can do? Right: More of the same bad medicine. The latest plans (as usual paid for by the Bertelsmann foundation, a neo-conservative think-tank of the worst kind) outline a model where everyone who receives well-fare is forced into a low-wage job. But as there are not enough of these jobs, the industry should be allowed to participate on these programs. Of course, the industry would never ever think of replacing their regular workers with these slaves. Never, especially as the plan forces the industry to offer the products created by this system not under a minimum price that is equal to the price of the product when created by regular workers. Or in other words: they get a guaranteed profit margin for all the troubles they have.

Of course, this only speeds up the decline of the whole social system, and makes sure that the Inverted-Robin-Hood principle comes true: Take it from the poor, give it to the rich. Or as all criminals know: It is easier to extract small amounts of money from a million persons than to extract millions from a bunch or rich persons.

The study itself talks about the positive effects a similar system had in the US. Hmmm .. funny, Michael Moore tells otherwise. And even such old studies like Gabriel Kolbs "Wealth and Power in America: An Analysis of Social Class and Income Distribution" (1962) tell us, that only one third of the unemployed are covered by social security and that 35-40% of the families in the US were forced to live below the maintainance standard (which is the minimum income needed to maintain a sustainable living) a while 12-25% even lived below the emergency standard (which is the minimum income needed to survive for a short time) (Source: Wikipedia). Well, if the future our politicans want to achieve looks like the past and current state of the U.S., then our society will be back in the same dark times as before the war, with extreme differences between the poor and the rich and no or only minimal connection or exchange between these two worlds.

What a brave new world we build.

* I use the term 'social-democrat' loosely here, as the words 'social' and 'democratic' should not to be mixed with the party or social system we have today. Calling the 'social-democratic' party social is like calling the roman-catholic church a progressive think-tank.

Charting in the Classic-Engine is currently implemented as a set of report functions and expressions. A chart-collector function produces the data-set and a chart-expression then uses the dataset and produces a JFreeChart object for it. The JFreeChart object is then directly renderable by the reporting-engine and produces some nice Vector-graphics output.

For historical reasons, these charting expressions have been held captive by the Pentaho-Platform. In the early days, when the land was rough and largely unsettled, a important project required to include charts. As some of the demos in the old JFreeReport-extensions project showed how to integrate JFreeChart into the reporting engine, it was the most obvious way to head into that direction. And so the first chart expression was born.

Over the years, more and more chart types were added and whenever a new feature was needed, it was added as well. At some point, the chart expressions started to depend on some core classes of the Pentaho-Platform, and thus what started as "strange packaging strategy" suddenly became a severe dependency issue, as now the chart-expressions were unable to run without a configured platform in the back. This rendered these expressions unusable outside of the platform and thus unusable for anyone who used the reporting-engine as embedded component.

Fast forward. Some months/years later, the trouble-maker dependency has been removed. But sitting in the same project as the platform code, there are only policies and documentation notes that protect us from having such a rouge dependency again. And in the real world, policies and documentation hints do not actively protect the code - they just make the reasoning for having to fix the newly introduced flaw easier.

But the new found peace was still an uneasy one: Changes in the way the reporting engine handled functions and generated content propagated slowly into these expressions (as the release cycles of the Engine and Platform do not (and cannot) match). At the same time, new properties of the expressions were added, which somehow appeared to work (well, they did not crash :) ) with the older engine but due to stricter validity checks on the existing contracts were bluntly rejected in the later engine and report-designer. And there the release cycle hit again: We did not see the troublesome contribution until the first bug-report came in. At that particular point, a fix would have either broken the engine's or the platform's backward compatibility promise.

And of course: Adding code to a large monolithic project is a lot more cumbersome than adding code to a small and simple (the Chart-Expressions are not simple. Although they have not yet reached the gnomish-threshold, the massive amount of knobs and levers and other properties in them makes them a complex system) focused project.

So let's see what we have:

• The chart-expressions share no code with the platform and do not depend on the mere existence of the platform
• The shifted release cycle gives me headaches
• Adding code to a heavyweight project is no fun

So let's fix these problems.

There is now a new project in the reporting-engine Subversion-Repository under svn://source.pentaho.org/pentaho-reporting/engines/classic/legacy-charts
The project contains a copy of the chart-expression formerly found in the platform. With Platform 1.8, the platform JAR will no longer contain the chart-expressions and the sources for them will be removed from the platform-project. The platform then switches to the expressions found here.

The first official release of these expressions will be named version 0.8.10 (to be in sync with the reporting-engine).

The freed chart-expressions take 72 KB (compared to ~450 KB of the pentaho-plugin.jar) and stick with JDK 1.4 (they _should_ even work with JDK 1.3), instead of requiring the bloatware JDK 1.5.

As initial bonus for early adopters, all chart-expressions now can control the item-label's visibility and handle null-values and invalid datasets a lot more gracefully than before.

Me too.


NO MORE SWT!




Grab it at: svn://source.pentaho.org/pentaho-reporting/libraries/mqleditor

Three days of hunting a bug in an application. An application that took the paradigm of loose coupling, combined it with Java's already incredible (bordering to insane) powerful reflection and introspection mechanisms, and then added a set of generated classes (by using on-the-fly-inline-assembly) on top of it. An application that could have done the very same thing with boring, almost dangerously conservative tailored-to-fit data-structures (the total opposite of the weird over-generalization found there). Whenever I have to use a Java DISASSEMBLER to debug a simple problem, then there is something seriously wrong.

But since yesterday I have hope again. After being in a second hand bookshop in Frankfurt, I found salvation. There it was, buried between old books of dubious origin: A original 1970's print of John Seymors Guide to Selfsufficiency. John Seymor wrote a whole series of books describing how to create a self-sufficient living style including do-it-yourself farming and house hold crafts. These texts are introductory writings and surely not suitable to grab the book and start being farmer the next day - but they give a good overview for the novice. After the book, you know at least where to start to search for more information.

Sadly, John Seymors book omits the most important and most profitable art of farming: How to plant and harvest peanut-butter. So I have to continue my journey for the secrets of Peanut-Butter-Farming ..