Free the Slaves: Pentaho Charting Expressions are now a independent project
Charting in the Classic-Engine is currently implemented as a set of report functions and expressions. A chart-collector function produces the data-set and a chart-expression then uses the dataset and produces a JFreeChart object for it. The JFreeChart object is then directly renderable by the reporting-engine and produces some nice Vector-graphics output.
For historical reasons, these charting expressions have been held captive by the Pentaho-Platform. In the early days, when the land was rough and largely unsettled, a important project required to include charts. As some of the demos in the old JFreeReport-extensions project showed how to integrate JFreeChart into the reporting engine, it was the most obvious way to head into that direction. And so the first chart expression was born.
Over the years, more and more chart types were added and whenever a new feature was needed, it was added as well. At some point, the chart expressions started to depend on some core classes of the Pentaho-Platform, and thus what started as "strange packaging strategy" suddenly became a severe dependency issue, as now the chart-expressions were unable to run without a configured platform in the back. This rendered these expressions unusable outside of the platform and thus unusable for anyone who used the reporting-engine as embedded component.
Fast forward. Some months/years later, the trouble-maker dependency has been removed. But sitting in the same project as the platform code, there are only policies and documentation notes that protect us from having such a rouge dependency again. And in the real world, policies and documentation hints do not actively protect the code - they just make the reasoning for having to fix the newly introduced flaw easier.
But the new found peace was still an uneasy one: Changes in the way the reporting engine handled functions and generated content propagated slowly into these expressions (as the release cycles of the Engine and Platform do not (and cannot) match). At the same time, new properties of the expressions were added, which somehow appeared to work (well, they did not crash :) ) with the older engine but due to stricter validity checks on the existing contracts were bluntly rejected in the later engine and report-designer. And there the release cycle hit again: We did not see the troublesome contribution until the first bug-report came in. At that particular point, a fix would have either broken the engine's or the platform's backward compatibility promise.
And of course: Adding code to a large monolithic project is a lot more cumbersome than adding code to a small andsimple (the Chart-Expressions are not simple. Although they have not yet reached the gnomish-threshold, the massive amount of knobs and levers and other properties in them makes them a complex system) focused project.
So let's see what we have:
There is now a new project in the reporting-engine Subversion-Repository under svn://source.pentaho.org/pentaho-reporting/engines/classic/legacy-charts
The project contains a copy of the chart-expression formerly found in the platform. With Platform 1.8, the platform JAR will no longer contain the chart-expressions and the sources for them will be removed from the platform-project. The platform then switches to the expressions found here.
The first official release of these expressions will be named version 0.8.10 (to be in sync with the reporting-engine).
The freed chart-expressions take 72 KB (compared to ~450 KB of the pentaho-plugin.jar) and stick with JDK 1.4 (they _should_ even work with JDK 1.3), instead of requiring the bloatware JDK 1.5.
As initial bonus for early adopters, all chart-expressions now can control the item-label's visibility and handle null-values and invalid datasets a lot more gracefully than before.
For historical reasons, these charting expressions have been held captive by the Pentaho-Platform. In the early days, when the land was rough and largely unsettled, a important project required to include charts. As some of the demos in the old JFreeReport-extensions project showed how to integrate JFreeChart into the reporting engine, it was the most obvious way to head into that direction. And so the first chart expression was born.
Over the years, more and more chart types were added and whenever a new feature was needed, it was added as well. At some point, the chart expressions started to depend on some core classes of the Pentaho-Platform, and thus what started as "strange packaging strategy" suddenly became a severe dependency issue, as now the chart-expressions were unable to run without a configured platform in the back. This rendered these expressions unusable outside of the platform and thus unusable for anyone who used the reporting-engine as embedded component.
Fast forward. Some months/years later, the trouble-maker dependency has been removed. But sitting in the same project as the platform code, there are only policies and documentation notes that protect us from having such a rouge dependency again. And in the real world, policies and documentation hints do not actively protect the code - they just make the reasoning for having to fix the newly introduced flaw easier.
But the new found peace was still an uneasy one: Changes in the way the reporting engine handled functions and generated content propagated slowly into these expressions (as the release cycles of the Engine and Platform do not (and cannot) match). At the same time, new properties of the expressions were added, which somehow appeared to work (well, they did not crash :) ) with the older engine but due to stricter validity checks on the existing contracts were bluntly rejected in the later engine and report-designer. And there the release cycle hit again: We did not see the troublesome contribution until the first bug-report came in. At that particular point, a fix would have either broken the engine's or the platform's backward compatibility promise.
And of course: Adding code to a large monolithic project is a lot more cumbersome than adding code to a small and
So let's see what we have:
- The chart-expressions share no code with the platform and do not depend on the mere existence of the platform
- The shifted release cycle gives me headaches
- Adding code to a heavyweight project is no fun
There is now a new project in the reporting-engine Subversion-Repository under svn://source.pentaho.org/pentaho-reporting/engines/classic/legacy-charts
The project contains a copy of the chart-expression formerly found in the platform. With Platform 1.8, the platform JAR will no longer contain the chart-expressions and the sources for them will be removed from the platform-project. The platform then switches to the expressions found here.
The first official release of these expressions will be named version 0.8.10 (to be in sync with the reporting-engine).
The freed chart-expressions take 72 KB (compared to ~450 KB of the pentaho-plugin.jar) and stick with JDK 1.4 (they _should_ even work with JDK 1.3), instead of requiring the bloatware JDK 1.5.
As initial bonus for early adopters, all chart-expressions now can control the item-label's visibility and handle null-values and invalid datasets a lot more gracefully than before.
MQL-Editor - civilized this time
Are you tired of having to work with crappy UI-toolkits like the Sucker's Widget Toolkit? Do you prefer a working out-of-the-box experience that runs the same way no matter what Operating System or hardware you use? Tired of having to cope with compatibility and native library issues?
Me too.
That's why I spend the day rewriting the MQL-Editor in Swing. The original editor was written using the abomination called SWT. But getting SWT code to run cross-platform is a nightmare and usually ends up in a lot of screaming and throwing stracktraces on each other. As I wasn't able to get the thing running on my Linux-Box without exceptions. After once more tracing it down to the class of "SWT does not work with Swing" problems we faced over the last years, I cannot bear it anymore.
NO MORE SWT!
As the MQL-Editor is a thin layer on top of the Pentaho-MetaData system, rewriting this functionality is faster than trying to cope with the bad design of SWT. And above all: It is guaranteed to work within 24 hours, and it solves all the SWT-vs.-Swing integration problems by killing SWT. That's a clear message, isn't it?
So here we are. The Swing version covers all the functionality of the original dialog and adds some extra validation of the input (so that it is impossible to continue with invalid queries).
Grab it at: svn://source.pentaho.org/pentaho-reporting/libraries/mqleditor
Debugging, Farming ... still no peanut-butter ..
Three days of hunting a bug in an application. An application that took the paradigm of loose coupling, combined it with Java's already incredible (bordering to insane) powerful reflection and introspection mechanisms, and then added a set of generated classes (by using on-the-fly-inline-assembly) on top of it. An application that could have done the very same thing with boring, almost dangerously conservative tailored-to-fit data-structures (the total opposite of the weird over-generalization found there). Whenever I have to use a Java DISASSEMBLER to debug a simple problem, then there is something seriously wrong.
But since yesterday I have hope again. After being in a second hand bookshop in Frankfurt, I found salvation. There it was, buried between old books of dubious origin: A original 1970's print of John Seymors Guide to Selfsufficiency. John Seymor wrote a whole series of books describing how to create a self-sufficient living style including do-it-yourself farming and house hold crafts. These texts are introductory writings and surely not suitable to grab the book and start being farmer the next day - but they give a good overview for the novice. After the book, you know at least where to start to search for more information.
Sadly, John Seymors book omits the most important and most profitable art of farming: How to plant and harvest peanut-butter. So I have to continue my journey for the secrets of Peanut-Butter-Farming ..
But since yesterday I have hope again. After being in a second hand bookshop in Frankfurt, I found salvation. There it was, buried between old books of dubious origin: A original 1970's print of John Seymors Guide to Selfsufficiency. John Seymor wrote a whole series of books describing how to create a self-sufficient living style including do-it-yourself farming and house hold crafts. These texts are introductory writings and surely not suitable to grab the book and start being farmer the next day - but they give a good overview for the novice. After the book, you know at least where to start to search for more information.
Sadly, John Seymors book omits the most important and most profitable art of farming: How to plant and harvest peanut-butter. So I have to continue my journey for the secrets of Peanut-Butter-Farming ..
The total brain-dump
Extract the brains of all reporting and charting developers, put them in a large blender, turn it on, wait, turn the blender off, and fill the brains back into their original containers to extract the new ideas. Repeat this process as often as needed.
The last two weeks were by all accounts insane. It all started very innocent, with a call to fly over to Orlando to bring the features of version 0.8.10 to the people of Pentaho.
The scenery was well set up. The charting team finished the first major step of the new charting engine, so that we now have a solid and well-laid out fundament for generating charts. Mike created Mantle, a new UI for the Pentaho Platform. Finally the solid infrastructural backend the platform provides becomes a face that is appropriate in the wake of the 21st century. Using Mantle instead of the old UI feels like driving a Ferrari vs. a horse chariot. You have to see it, feel it! In the meantime, the Reporting-Ease-Of-Use sprints were still on their way, changing the Report-Designer from a developer-centric tool into a by far more business user friendly designer.
The two weeks were filled with dialogs like this: 'Look, thats one of the new things we put into [project]!' - 'Hey, I could use that in [Project] to do do [mind boggling feature].' - 'Wait, when you do that, I could use that to do [feature] over here.'.
So what did we create?
After these two weeks, I really have to wonder: If we, with just a few developers, are able to drive development so fast, why do Monster-companies like Business Objects, Cognos and so on (with thousands of employees) appear nowadays nothing more than sitting ducks for targeting practice. Not that I would complain about it, as 65 million years ago, swift and agile critters already out-paced slow and huge adversaries. Let's repeat that game ..
The last two weeks were by all accounts insane. It all started very innocent, with a call to fly over to Orlando to bring the features of version 0.8.10 to the people of Pentaho.
The scenery was well set up. The charting team finished the first major step of the new charting engine, so that we now have a solid and well-laid out fundament for generating charts. Mike created Mantle, a new UI for the Pentaho Platform. Finally the solid infrastructural backend the platform provides becomes a face that is appropriate in the wake of the 21st century. Using Mantle instead of the old UI feels like driving a Ferrari vs. a horse chariot. You have to see it, feel it! In the meantime, the Reporting-Ease-Of-Use sprints were still on their way, changing the Report-Designer from a developer-centric tool into a by far more business user friendly designer.
The two weeks were filled with dialogs like this: 'Look, thats one of the new things we put into [project]!' - 'Hey, I could use that in [Project] to do do [mind boggling feature].' - 'Wait, when you do that, I could use that to do [feature] over here.'.
So what did we create?
- Mantle no longer needs a explicit XAction to run reports stored in the new unified fileformat. As long as your report does not require complex preprocessing or uses bursting, you no longer need to write and maintain a separate XAction.
- Mantle now provides a Pageable HTML preview for the reports of the platform.
- The engine now has a clean and controlled way of defining parameters. The report definition contains all information needed to build the most marvelous UIs on top of it. Which brings us to:
- Mantle can parametrize reports and generates a sensible, fully and easily customizable (without using XSLT! I cant bear that stuff!) parameter UI.
- The new Charting System now becomes integrated into the Classic-Engine. (For now, this support will be called experimental, as we need the freedom to twist the code and XMLs whenever we feel the need for it.)
- The Interactivity-Extensions blew away the innocent (and also the not so innocent, of course) bystanders. The ability to inject any HTML/Script code into the resulting HTML files now allows a whole class of new reports. The Swing-UI allows similar features.
- The Report-Designer now comes with a fully featured formula-editor, that makes Office-(Open and MS alike)-users feel at home.
- We all agreed (with full heart) that SWT is a big WTF and should not have been born in the first place. SWT brings the insanity of low-level APIs combined with the inability to provide sensible platform independence (the main argument for running Java!). Swing is our future here!
After these two weeks, I really have to wonder: If we, with just a few developers, are able to drive development so fast, why do Monster-companies like Business Objects, Cognos and so on (with thousands of employees) appear nowadays nothing more than sitting ducks for targeting practice. Not that I would complain about it, as 65 million years ago, swift and agile critters already out-paced slow and huge adversaries. Let's repeat that game ..
Bloatware JDK
When it comes to the question of "what JDK does your application runs on", my answer "JDK 1.2.2" usually causes a lot of raised eyebrows, along with questions like "That's pretty old? Why don't you update? There are so many new features in the new <put the latest version of the JDK in here>!"
Well, until now I haven't found a feature that was compelling enough to upgrade.
"Perfection is achieved, not when there is nothing more to add, but when there is nothing left."
Starting with JDK 1.3, Sun wend on a strange and weird expedition with its JDK.
The classes in the JDK define the core functionality set of the Java Language. In the old days of JDK 1.2.2, this core feature set was a sensible minimum for working with Java. Anything beyond that functionality would have to be added as extension-library or application-library (the difference between the two choices is minimal; extensions are managed by the boot-classloader, are always trusted and always available to all applications. Extensions are added by copying the JARs into the $JRE/lib/ext directlry. Application libraries are untrusted by default, are application specific and are therefore only loaded if specified on the classpath. There is no limitation in what functionality you can achieve in either way, its just "do you have to set the classpath or not".
In JDK 1.3 Sun started to copy JNDI and a Corba-ORB into the JDK. Before that, JNDI was a separate extension library, while the ORB was a external product. So there was no reason, other than feeling the need to have some new features, to pump up the size of the JDK.
The reasoning for the Corba-ORB is particulary funny. From the README: "This eliminates the requirement for licensing and distributing third-party Java ORBs." The last time Microsoft tried that reasoning, they got sued to the ground. Well, Sun could as well just have made that ORB available under a free (as in beer, I dont want to be so demanding to ask for "as in speech"-freedom yet) license with the same effect. But (I guess) CORBA was what all the "cool kids" used at that time, so the JDK would have to follow.
I yet have to see Corba-based applications outside the scope of heavy enterprise applications (which normally use the Enterprise Editon of the JDK anyway)
With JDK 1.4, Sun started to mess around with the language specs (and never stopped since then). The formerly extension libraries JAXP and Java-ImageIO were added to the JDK. Looking at the very sucessful open-source market, Sun also added a Logging API (which is a cheap and horrible to configure copy of Log4J), regular expressions (ignoring the fact that there are plenty of well-established RegExp-libraries available), a new Printing-API and a preferences API.
Again, most of these libraries existed as separate projects before, and none of these libraries requires special attention from the JDK to run. So there is absolutely no reason for them to be a core functionality.
The printing API is not that bad, but there was and is no reason for that being a core functionality. The API could have been created as a ordinary JDK-extension, and thus not adding a megabyte or two to the JDK. As JNDI failed to get acceptance from the users, Sun tried again, and added a Preferences-API. Giving developers a sensible API for storing application preferences is a good thing, no question. But that whole API smells just like a simplified version of the JNDI system. So instead of providing a sensible JDNI default implementation, they prefered to add something new. Sweet, more space wasted.
Then there was JDK 1.5 (or 5.0, as Dot-Net was approaching, and so Sun had to show that Java was more mature. Sooner or later we will see a "Java XP", I guess. For the first time, Sun did not add new libraries (updating previously added libraries does not count as something new, despite all marketing claims), yet still managed to mess around with the language to introduce a new layer of possible bug-injection points.
But heaven did not last long. JDK 1.6, (I guess after running out of libraries to add without looking stupid,) now ships with a whole database system. Embedded databases are perfectly suited for desktop applications, sure, but for ages they roamed freely through forrests and over plains, and people used them like any other library. So what's the point in adding yet another 20MB of Jars to the already huge JDK? What makes this move even more ridiculous: Java-DB is not even part of the JDK 1.6, it is installed in a separate location and has to be activated manually. For now, the JRE also does not ship with that database - but you can bet it will later!
I really wonder - what's the reasoning behind that? Do they want to copy each aspect of Dot-NET (including the 5GB runtime called "a Windows installation")? Or is this just a "I'll scratch your back" move towards IBM (which orignally developed Java-DB (aka Derby, aka DB/2-Embedded)?
So make the insanity obvious, just look at the numbers:
JDK 1.2.2_17; Windows; International Version: 7.18 MB
JDK 1.3.1_20; Windows; International Version: 7.94 MB
JDk 1.4.2_16; Windows; Multilingual Version: 14.94 MB
JDk 1.5.0_14; Windows; Multilingual Version: 16.10 MB
JDK 1.6.0_5; Windows; Multilingual Version: 15.18 MB
The next JDK (JDK 1.6.0-update 10), finally tries some corrective steps. The JDK will no longer install all bloated core packages; most packages will now be downloaded and installed on demand (Which will give a funny new class of ClassLoader-errors ("HTTP 404: Class not found").
"The JRE is being modularized, so that bits and pieces of it can be downloaded as needed. In the current prototype, the download needed to support a typical Swing program is between 3 and 4MB."
At that point, we will be back where we started, at the size of JDK 1.2.2. Sanity wins, finally.
Well, until now I haven't found a feature that was compelling enough to upgrade.
"Perfection is achieved, not when there is nothing more to add, but when there is nothing left."
Starting with JDK 1.3, Sun wend on a strange and weird expedition with its JDK.
The classes in the JDK define the core functionality set of the Java Language. In the old days of JDK 1.2.2, this core feature set was a sensible minimum for working with Java. Anything beyond that functionality would have to be added as extension-library or application-library (the difference between the two choices is minimal; extensions are managed by the boot-classloader, are always trusted and always available to all applications. Extensions are added by copying the JARs into the $JRE/lib/ext directlry. Application libraries are untrusted by default, are application specific and are therefore only loaded if specified on the classpath. There is no limitation in what functionality you can achieve in either way, its just "do you have to set the classpath or not".
In JDK 1.3 Sun started to copy JNDI and a Corba-ORB into the JDK. Before that, JNDI was a separate extension library, while the ORB was a external product. So there was no reason, other than feeling the need to have some new features, to pump up the size of the JDK.
The reasoning for the Corba-ORB is particulary funny. From the README: "This eliminates the requirement for licensing and distributing third-party Java ORBs." The last time Microsoft tried that reasoning, they got sued to the ground. Well, Sun could as well just have made that ORB available under a free (as in beer, I dont want to be so demanding to ask for "as in speech"-freedom yet) license with the same effect. But (I guess) CORBA was what all the "cool kids" used at that time, so the JDK would have to follow.
I yet have to see Corba-based applications outside the scope of heavy enterprise applications (which normally use the Enterprise Editon of the JDK anyway)
With JDK 1.4, Sun started to mess around with the language specs (and never stopped since then). The formerly extension libraries JAXP and Java-ImageIO were added to the JDK. Looking at the very sucessful open-source market, Sun also added a Logging API (which is a cheap and horrible to configure copy of Log4J), regular expressions (ignoring the fact that there are plenty of well-established RegExp-libraries available), a new Printing-API and a preferences API.
Again, most of these libraries existed as separate projects before, and none of these libraries requires special attention from the JDK to run. So there is absolutely no reason for them to be a core functionality.
The printing API is not that bad, but there was and is no reason for that being a core functionality. The API could have been created as a ordinary JDK-extension, and thus not adding a megabyte or two to the JDK. As JNDI failed to get acceptance from the users, Sun tried again, and added a Preferences-API. Giving developers a sensible API for storing application preferences is a good thing, no question. But that whole API smells just like a simplified version of the JNDI system. So instead of providing a sensible JDNI default implementation, they prefered to add something new. Sweet, more space wasted.
Then there was JDK 1.5 (or 5.0, as Dot-Net was approaching, and so Sun had to show that Java was more mature. Sooner or later we will see a "Java XP", I guess. For the first time, Sun did not add new libraries (updating previously added libraries does not count as something new, despite all marketing claims), yet still managed to mess around with the language to introduce a new layer of possible bug-injection points.
But heaven did not last long. JDK 1.6, (I guess after running out of libraries to add without looking stupid,) now ships with a whole database system. Embedded databases are perfectly suited for desktop applications, sure, but for ages they roamed freely through forrests and over plains, and people used them like any other library. So what's the point in adding yet another 20MB of Jars to the already huge JDK? What makes this move even more ridiculous: Java-DB is not even part of the JDK 1.6, it is installed in a separate location and has to be activated manually. For now, the JRE also does not ship with that database - but you can bet it will later!
I really wonder - what's the reasoning behind that? Do they want to copy each aspect of Dot-NET (including the 5GB runtime called "a Windows installation")? Or is this just a "I'll scratch your back" move towards IBM (which orignally developed Java-DB (aka Derby, aka DB/2-Embedded)?
So make the insanity obvious, just look at the numbers:
JDK 1.2.2_17; Windows; International Version: 7.18 MB
JDK 1.3.1_20; Windows; International Version: 7.94 MB
JDk 1.4.2_16; Windows; Multilingual Version: 14.94 MB
JDk 1.5.0_14; Windows; Multilingual Version: 16.10 MB
JDK 1.6.0_5; Windows; Multilingual Version: 15.18 MB
The next JDK (JDK 1.6.0-update 10), finally tries some corrective steps. The JDK will no longer install all bloated core packages; most packages will now be downloaded and installed on demand (Which will give a funny new class of ClassLoader-errors ("HTTP 404: Class not found").
"The JRE is being modularized, so that bits and pieces of it can be downloaded as needed. In the current prototype, the download needed to support a typical Swing program is between 3 and 4MB."
At that point, we will be back where we started, at the size of JDK 1.2.2. Sanity wins, finally.