Reporting Tales - development tag

Purifying action: Report-Designer 2.0 is on the way

Tue, 29 Jul 2008 20:45:00 GMT

Three weeks! It now has been three weeks since the purification started. Three weeks of furious deleting, refactoring, screaming, crying, contemplating suicide, restructuring, analyzing and finally fixing the report-designer's codebase.

Are we finished now? Not yet. The current SVN version is now in a state where all the major parts are in place, but the overall experience is still .. experimental.

My development theme for the new report-designer is "back to the roots". Almost all external dependencies are gone now. Well, some minor dependencies are still waiting for their extermination:

JGoodies Formlayout - a creature spawned in the depths of the seventh circle of hell - is still used in many of the old dialogs. As this layout manager is as hard to maintain as the infamous Gridbag-Layout of the core JDK, there are no advantages of using it. Whenever we now touch one of the old dialogs, we will replace this nightmare with a plain Gridbag-Layout. GridbagLayout may be ugly, but at least we don't have to ship a Jar for it or force other developers to learn yet another useless trick.
IntelliJ IDEAs NotNull-annotations. The idea of marking parameters, fields and return-values as nullable seems to be a good idea at first. But sadly, the implementation of said idea is horrible. When using IntelliJ, violations of the annotation constraints easily remain undetected - the compiler does not warn you and so you won't see errors until you get the Exception at runtime. In addition, anyone not using IntelliJ wont have much use of these annotations anyway. Therefore, for the new code and all classes touched during the redesign: Let's kill these annotations. Now we replace them with strict coding guidelines. We start with explict checks on all public or protected methods, wherever we do pass object-references around. On a architecture level we prevent NullPointerExceptions by having a strict object-model with strict type-checks. And of course, this includes that we do not pass private collections around, as it seems all so common these days.
DOM4J and Jaxen: These classes are used by the Pentaho-Version-Checker to parse a structurally simple XML document. One could use a combined total of 500KB of libraries or one could use down-to-earth XML-parser implementations that use JDK-1.5 functionality only. Guess what the old code does and guess with what we will end up with.

Very early in the game, we already removed the Pentaho-Platform, as using a ?? MB collection of libraries to read a single properties-file was a bit ridiculous. We removed all magical code-generation (that was done via asm-lib and reflection) as all of these uses were easily replaceable by a single well-defined interface. We removed the PullParser and all the XML-magic as XML parsing in JDK 1.5 is really well-defined (heck, it was well defined in the age of JDK 1.2, for the usecases we need here). We removed SWT as I cant stand libraries that crash the Virtual Machine just because they cannot cope with the processor architecture or operating system. I still believe in the "100% pure Java" world Sun advertised ages ago. And we removed Castor, as a OR-mappers used for parsing a non-complex XML file is a WTF on its own.

Inside the code itself, we sliced through the event handling and created a clear event-dispatching path. Although it must be fun to virtually connect every object with every other object, I do not want to spend the few remaining years of my sanity on maintaining such a beast. Almost all cases of invoking methods by reflection were removed by simply creating interfaces containing the method to be called. But that road was no fun to walk - so at the end we ripped out the old data-model and rendering and editors and ... and so the new report-designer works directly against the reporting-engine's datastructures and with components that follow a strict separation of concerns.

And now, with all magic killed, we leave the dark age and enter the age of enlightening. With a sane and maintainable architecture to start with, we will finally be able to spend less time on maintenance and more time on developing new features. After all, bug-hunting is not a fun activity, so spending less time there leaves us more time for the fun stuff.

The new report-designer data-model is a thin wrapper around the visual objects of the reporting-engine. The model itself only provides some caching of the layout, change tracking and event-notification and nothing else. There is no point in wrapping around datasources or functions, as long as we set the social rule that after these objects have been altered we always notify the model of it.

The report-designer's code now no longer contains copies of existing reporting-engine functionality. Datasources, layouting, expressions - it all comes from the one and only principal source now.

And the result of all of this: The report-designer now automatically provides access to all features of the reporting engine as soon as the reporting engine implements them. We can now guarantee that whatever the report-designer shows you in the edit view will be the same result the reporting engine will show you. Everything the reporting-engine can do, can now be done with the report-designer.

We now also started to define the first set of plugin-points for the report-designer, so that anyone can hook-in own editors, datasources and actions without dealing with all the complexity of the whole project.

Taking small steps to cross the tab

Thu, 26 Jun 2008 18:27:00 GMT

After a long silence, let's have something positive today: Pentaho Reporting now officially talks to Mondrian and any other OLAP4J datasources.

We now ship with two flavors of MDX access. The existing MDX capabilities are covered by the BandedMDXDataFactory, while the crosstabbing functionality will rely on a new DenormalizedMDXDataFactory.

The BandedMDXDataFactory takes a two dimensional MDX-Query-Result and maps the multidimensional dataset into a flat table. The approach is reasonable if you want to access the cube row-by-row, but it fails badly as soon as your query has more than two dimensions or if your query-result displays a ragged hierarchy. The report-designer used this mode for a very long time to provide at least some access to Mondrian-DataSources. The banded mode is still great if you need banded reporting over MDX datasources.

However, with Version 0.8.11 of the reporting engine, we finally have to provide real crosstabbing capabilities.

At that point, the pre-chewed data provided by the BandedMDXDataFactory is totally unsuitable for anything sophisticated. You cannot reconstruct a cow from a steak. In the same way we cannot use the banded data to reconstruct the axis and hierarchy information provided by the real MDX-ResultSet. At the same time, the complex (and in some points ambiguous) nature of the data-processing that happens inside the BandedMDXDataFactory makes it next to impossible to use plain queries as source for a crosstabbed report.

The goals for our crosstab-implementation are straightforward:

It has to work on existing data-sources using only TableModels as input (Don't over-architect)
The internal data-source structures must be simple so that any source-system is capable of providing the data in the correct format. (Don't exclude anyone.)
Provide only simple aggregation as built-in functions (Don't copy Mondrian.)
Make sure that functions and expressions work exactly like in relational reports. (Don't be special.)

The new denormalized MDX-DataFactory provides a streaming view over the MDX-Cells. Any datasource can provide a similar view by simply joining the fact-table with all dimensions (and by sorting them according to the desired axis structure). The denormalized view now makes it possible to treat MDX-Columns and Rows (and any of the other 253 possible axises) as relational groupings, which just happen to be displayed in a non-banded manor.

Now with the data-problem solved, displaying the data will be quite easy, even for huge result-sets.

Classic-Engine 0.8.10 and beyond

Tue, 03 Jun 2008 12:33:00 GMT

This weekend, we finally released Version 0.8.10 of the Pentaho Reporting Classic Engine. This release is yet another infrastructure release (yes, sounds boring) that prepares the ground for going to 1.0.

Aside from the already covered Unified Fileformat and the full support for all kinds of meta-data, this release also ships with a totally revamped parametrization API, support for Barcodes (great job, Mimil!) and Sparkline support.

The next development cycle will be a shorter one. In the upcoming Version 0.8.11 we will finally add crosstabbing and Pivot-tables, speak with Mondrian datasources, provide a sensible interface for the rich-text capabilities, add free-form subreports and will add a first version of the multi-column support. If everything goes right, this version will enter its Release-Candidate state at the end of the month.

One major change already happened on this version: All engine and library classes contained in org.jfree-packages now have been moved into corresponding org.pentaho-packages. This move was necessary so that we do no longer pollute the org.jfree-namespace. At the same time, it allows us to move the Classic and the Flow-Engine into separate packages, so that they can co-exist in the same Java Virtual Machine.

As usual: Users of the XML fileformats are safe from any changes, the XML report definitions continue to work unchanged. API users will have to migrate their code to the new package space. But as the change involves only moved packages, a update of the import-statements should be the majority of the conversion work.

As the APIs of the libraries seem to be stable and sane now, along with the release of 0.8.11, all libraries will be labeled 1.0 versions.

The total brain-dump

Mon, 21 Apr 2008 05:10:00 GMT

Extract the brains of all reporting and charting developers, put them in a large blender, turn it on, wait, turn the blender off, and fill the brains back into their original containers to extract the new ideas. Repeat this process as often as needed.

The last two weeks were by all accounts insane. It all started very innocent, with a call to fly over to Orlando to bring the features of version 0.8.10 to the people of Pentaho.

The scenery was well set up. The charting team finished the first major step of the new charting engine, so that we now have a solid and well-laid out fundament for generating charts. Mike created Mantle, a new UI for the Pentaho Platform. Finally the solid infrastructural backend the platform provides becomes a face that is appropriate in the wake of the 21st century. Using Mantle instead of the old UI feels like driving a Ferrari vs. a horse chariot. You have to see it, feel it! In the meantime, the Reporting-Ease-Of-Use sprints were still on their way, changing the Report-Designer from a developer-centric tool into a by far more business user friendly designer.
The two weeks were filled with dialogs like this: 'Look, thats one of the new things we put into [project]!' - 'Hey, I could use that in [Project] to do do [mind boggling feature].' - 'Wait, when you do that, I could use that to do [feature] over here.'.

So what did we create?

Mantle no longer needs a explicit XAction to run reports stored in the new unified fileformat. As long as your report does not require complex preprocessing or uses bursting, you no longer need to write and maintain a separate XAction.
Mantle now provides a Pageable HTML preview for the reports of the platform.
The engine now has a clean and controlled way of defining parameters. The report definition contains all information needed to build the most marvelous UIs on top of it. Which brings us to:
Mantle can parametrize reports and generates a sensible, fully and easily customizable (without using XSLT! I cant bear that stuff!) parameter UI.
The new Charting System now becomes integrated into the Classic-Engine. (For now, this support will be called experimental, as we need the freedom to twist the code and XMLs whenever we feel the need for it.)
The Interactivity-Extensions blew away the innocent (and also the not so innocent, of course) bystanders. The ability to inject any HTML/Script code into the resulting HTML files now allows a whole class of new reports. The Swing-UI allows similar features.
The Report-Designer now comes with a fully featured formula-editor, that makes Office-(Open and MS alike)-users feel at home.
We all agreed (with full heart) that SWT is a big WTF and should not have been born in the first place. SWT brings the insanity of low-level APIs combined with the inability to provide sensible platform independence (the main argument for running Java!). Swing is our future here!

After these two weeks, I really have to wonder: If we, with just a few developers, are able to drive development so fast, why do Monster-companies like Business Objects, Cognos and so on (with thousands of employees) appear nowadays nothing more than sitting ducks for targeting practice. Not that I would complain about it, as 65 million years ago, swift and agile critters already out-paced slow and huge adversaries. Let's repeat that game ..

Version 0.8.10: All Features on board

Mon, 31 Mar 2008 20:29:00 GMT

Finally, after weeks after weeks of coding, version 0.8.10 is feature-complete. During the next weeks, we will enter a stabilization cycle fixing bugs after bugs until we can safely declare the engine ready for production use again.

Bugs we found in the 0.8.10 codeline that also exist in 0.8.9 will be fixed in both branches. When we finally reach GA-state, we will produce two releases; 0.8.9-5 to bring the fixes to the Pentaho-Platform 1.6/1.7 users, while Platform 1.8 surely switches to 0.8.10 as soon as we can get it integrated.

The feature set of this new version is impressive now.

We ship with 6 data-source-types now:

TableModels
Static Datasources( calling Java-Methods using reflection)
SQL (using plain drivers, JNDI or java-beans as source for the JDBC-connections)
Hibernate HQL
Kettle-Transformations
MQL-queries (using the Pentaho-Meta-Data system)

And of course, we have some more exciting new features:

The brand new ODF-based Unified-File-Format
Complete End-to-End Meta-Data system integration
A structural Meta-Data layer to make all of our elements and expressions introspectable
Attribute-Expressions to compute Element-Attributes at runtime
Sparkline-Elements
CLOB support
Complete SVG-Support using Batik
a interactive SwingPreview (Hyperlinks for drill-down and drill-across reports)
a greatly extended HTML-Export that allows to embed own raw-content for interactive HTML-files
Group-wide Keep-Together and group- and report-wide styles

and as mentioned earlier in this channel:

A re-born Report-Wizard-Core

If everything works as planned, we shall see the final release in four weeks from now.