Reporting Tales

June 29, 2009

No need to look sour: We’ll show how Citrus came to the Pentaho-Report-Designer

Filed under: Uncategorized — Thomas @ 20:09

On Wednesday 3PM London time (or 10AM Orlando time) we will give a Community Web-Ex about how Reporting Changes in the upcoming Citrus-Release.

Aside from officially announcing the switch from our well-loved, most-hated-by-sales “0.8.x” version numbering schema, we will shed some light what new features we have and on how the new features will make your reporting life easier. During the first part, we will give an short overview about the new stuff (and the reasons behind it) and also explain the changed relationship of the new reporting engine and the BI-Platform. As we all know, that there is no other truth than working code, the second part will demonstrate how to utilize parameters, the new datasources and the Rich-Text engine (along with the magic wizard), to build a simple interactive ad-hoc reporting solution. Building this solution should be fast, so we’ll have plenty of time for questions and answers at the end of the Web-Ex.

The web-ex information will be available via the Pentaho-Forum.

June 14, 2009

Scripted data-sources: And yet another usecase for XActions gone

Filed under: Uncategorized — Tags: — Thomas @ 17:55

On the holy quest to cut back on the usage of XActions for everydays life, we come a lot closer to the goal now.

XActions are cool for solving the more complex cases of BI-Tasks. But for just getting the data for a friggin’ report they are the literal hammer trying to squash the mosquito. Yes, it works, but it leaves a mess behind. (Sorry, Marc, I tried to love XActions, but it felt like making love to a cactus. I just cant stand programming in XML for ordinary daily tasks.)

So we now have the Scriptable-Datasource working. The datasource itself was working since months in the engine, but as I learned the hard way: No feature is used when there is no GUI. And since last week the UI for defining those datasources is working. The scriptable datasource uses Apache’s Bean-Scripting-Framework to let you instantiate Swing TableModels during the report processing. The datasource has access to the current DataRow (the thing that holds the data for the current report-state), and thus is fully parametrizable. At the same time, you are free to pull in any third party code (like code that accesses your backend systems or instantiates EJBs) so that you are no longer limited to plain databases or precompiled JARs (via the Java-Call datafactory) to get your data in. And the best of all: Assuming that you have the right set of libraries on the classpath, you can use a large set of scripting languages to get your job done.

Nothing would be perfect without an example, right? So here we go:

import javax.swing.table.DefaultTableModel;

String[] columnNames = new String[5];
columnNames[0] = "Year";
columnNames[1] = "Q1";
columnNames[2] = "Q2";
columnNames[3] = "Q3";
columnNames[4] = "Q4";

DefaultTableModel model = new DefaultTableModel(columnNames, 3);
model.addRow(new Object[]{ new Integer(2001), new Integer(10), new Integer(10), new Integer(14), new Integer(21)});
model.addRow(new Object[]{ new Integer(2002), new Integer(14), new Integer(34), new Integer(10), new Integer(12)});
model.addRow(new Object[]{ new Integer(2003), new Integer(10), new Integer(11), new Integer(14), new Integer(15)});
return model;


And now back to the drawing board, as I still have to get the attribute-expression support for the “query”, “query-timeout” and “query-limit” attributes back in place. At that point, you will be able to compute the query that used for the report. But that’s definitely a topic for another blog entry.


June 2, 2009

Real Rich-Text support is making it’s way into the engine

Filed under: Uncategorized — Thomas @ 20:07

One of the last commited items out of my infinite bag of things to add to the reporting system was the support for real Rich-Text content. (PRE-166)

Rich-Text basically means: Formatted text. But although formatted text may be nice, our layouting system easily can do a bit more than that. As the current layout-system of the classic-engine is a spin-off of the layout system of the frozen Flow-Engine, we can render full-featured documents at nearly no additional costs. And of course, implementing a minimalistic system that supports little more than <b>, <i>, <ul> is boring. If I wanted an non-challenging job, I would become Banker, or even Central-Banker.

Today, surprisingly after only 6 hours of work, the first RTF document appeared happily in the report-designer’s report-preview. And even more surprising to me: There are only two features missing to make it totally complete: Justified text-alignment and line-indentions. Solving the justified-text-alignment problem is a to large to solve within the Citrus timeframe (at least if I want to have it safe and scalable. Text-processing is the stuff that takes ages to compute and eats memory like trolls eat children). But text-indention is easy to bring in (it’s yet another “almost there” feature).

Now how does it work?

Rich-Text processing is disabled by default and has to be enabled by setting the “rich-text-type” attribute to the desired document type. This way we preserve backward compatiblity (as old reports have no such setting, so they default to text/plain documents) and performance (as plain-text does not need to go through the Rich-Text parser). If the rich-text-flag is set, any content that could contain a document (CLOB, BLOB, char- and byte-arrays and InputStreams and Readers, and of course javax.swing.text.Document instances) are passed into the layouter as raw-objects. The layout-preprocessor then translates these objects into real Document instances by calling RichTextProcessor implementations. These implemenations are pluggable, so that you can easily bring your own rich-text format into the engine. The built-in RichTextProcessors use the Swing-EditorKit parsers for RTF and HTML to create the Documents and then translate these documents into Bands, Label and Content-(formerly known as Image)-elements.

The original element is then treated as Band and the generated Band/Elements are processed as if they were childs of this artifical band.

Of course, introducing artificial Bands and Elements did have side-effects. Not on the engine’s core, which did not required a single change. But we now have a new Element-Type called “external-element-field”, which allows to use the same trick outside of the layouter. If the element returns a “org.pentaho.reporting.engine.classic.engine.core.Element” instance, that instance is printed as if the element was a band and the returned instance were the band’s only child. So far, the only one real use for it is to load external Sub-Reports from an URL or path, but after Citrus we may weave some more magic in this area.

If you haven’t checked out our Citrus-Reporting yet, then you are missing an amazing new world. So don’t hesitate and grab the latest build!

May 4, 2009

Apple’s Terminal: Its hard to find something more retarded

Filed under: Uncategorized — Tags: — Thomas @ 15:03

Compared with a real X11-Terminal application, the Apple Terminal sucks. Big Time.

“Whats wrong with it?” you may ask.

Lets start simple. Command-line applications cannot receive mouse events when running inside the Apple Terminal. And the keyboard is messed up (yeah, I’m running Apple, so I probably are supposed to use the Mouse to enter certain keys, like Page-Up and Page-Down.).

Since switching from a real System to the Mac (superior hardware, but dont get me started about the soft side), I really started to appreciate the power of the X11-Terminals you find in all Linux distributions. I’m a command line guy, so working with a Mac feels like being a Catholic Priest sitting in a Nudist Colony - Sin whereever you look.

Gettin Midnight Commander was easy, once I found MacPorts. But working with it was pain. No, torture. I can live without a mouse - but without a Page-Up and -Down key, browsing directories or looking at files becomes a nightmare. And without a insert-key, selecting files is not fun either. I was about to give up and end my days using muCommander. MuCommander is a Midnight-/Norton Commander clone, but copying files over the network is horrible slow and there is no command line either, so navigating complex directory structures is a keystroke hell.

But then a heavenly voice spoke to me: “Thomas. You are using Apple software.” And I remembered. Apple software usually comes good looking but not neccessaily “complete” - more like prove of concept things than real applications. So maybe its not the Mac OS (its Unix after all - and thus hard to get wrong) - its the application. Fireing up X11 and xterm convinced me: Yes, that Apple was a foul one.

Its not that mouse support is a experimental one. Its has been there since at least 9 years now. Have no rush, Apple, I can wait another 9 years. Who needs Mouse support anyway. (Note that I say that to a audience that did not manage to make UI elements like comboboxes navigatable via the keyboard. I have seen Comboboxes on my C64’s GEOS system and yes, they where usable via a keyboard.)

But not getting the keyboard right is a killer. Running X11 and XTerm on the same machine gets it right out of the box. So its not that they dont have the tools. They dont have either a clue how to adapt the XTerm code to their own codebase or they dont care. (Guess what I think what the reason is.)

I would happily drag the XTerm application on the launcher bar. BUT I CANT. The launcher does not launch just anything - it must be a blessed application bundle. So to start the XTerm, I need a command line, which Xterm would provide. Thanks again, Apple.

(If anyone knows a sensible terminal application that supports ncurses mouse-events, please drop me a line. I start to get desperate.)

May 1, 2009

Its May - and Pentaho Report Designer is rushing towards the release ..

Filed under: Uncategorized — Thomas @ 20:44

Its May, and thus all the animals big and small go wild on their eternal quest to spread their genes. Thats fine and nice, and unless they make their mess in my backyard (not that I have a backyard) - so all is OK by my standards. But not just animals feel the need to go all wild - the report designer feels this urge too. Look how it peacocks around, spreads its features to attract data.

Today, the last feature implementation sprint ends, and thus we are now in a fairly feature complete state. There are bits and bolts that need adjustment, and a whole family of bugs needs to be sent to the next, hopefully better world, where all code is written in C++ with lots of places to hide and procreate. During the comming six weeks we will now concentrate on hardening the code to make this the best report-designer the BI space has to offer.

If you haven’t seen the new breed, then now is a good time to do so, and you will see that comparing Citrus to the older releases is like comparing monkeys to humans - some features are still recognizable, but the effects of evolution are clearly visible.

Despite the fact that I talked about feature completeness a few lines earlier, I’m not to cheap to contradict myself immediately saying that one big feature will make it into the engine during the next days: Rich text support is comming to the reporting engine. Technically, we already supported this for a very long time - but without a sane user-model to create rich.text this was not a usable feature at all. Now that we finally came up with a decent specification, we can solve that task once and for all.

rdw-citrus-m5

The last three weeks saw great changes (again). The Report-Design-Wizard, which now has a slick (or in CEO speak: sexy) design and a groovy Mortal Combat like logo. Managing datasources and queries got easier - a right click on the query now allows you to select that query as the active one or to edit it directly. The platform integration now allows to schedule the XAction-less reports and Will Gorman sneaked some more crosstab-editing in. The inspector is back from his vacation, and tells you (whether you want to hear it or not) about all the wrongs in the report-definition. And last but not least, a new display mode now highlights overlapping elements that would be removed when the report gets exported to any of the table-export output types.


prd-citrus-m5

So if you haven’t done it yet, then go to our Hudson server right now and check out the latest Citrus release.

April 15, 2009

Not hacker-proof: Getting a Social Security Number in Ireland

Filed under: Uncategorized — Thomas @ 11:29

Right now I’m sitting in the local Welfare office to get a PPS number (thats how the Social Security Number is called here), otherwise I would not be able to legally earn any money here. Not that I would have problems earning money illegaly, its just that the local law-enforcement community gets a bit bitchy after a while in that case.

As Ireland, like all english speaking countries lacks organization and oversight when it comes to registering and controlling people, there exists no central citizen registration. Yeah, if they would accept a few lessons from Germany or France leaders like Bush (now Obama) or Blair could learn a valuable lesson on how to create a police state efficently. A social security number makes you a “real” citizen here - its needed to get a Job and maybe for all kind of other identifying purposes as well.

To get one of these numbers, all you need is a proof of address and a proof of identity.

Getting the proof of address is easy - just give them a utility bill. As there are no checks on your identity, any name will work, as long as your landlord believes your claim, that your mother named you “Donald Duck” . Given some basic people-skills, that should be easy to do.

The proof of identity is a bit more difficult. If you claim to be neither British nor Irish citizen, the Welfare-office requires a passport. Getting hold of one named “Donald Duck” is probably a bit too difficult to be carried out by novice con-artists. But getting a fake Windsor or scottish accent is not hard, and then you can take the easy route: British or Irish citizens need a birth certificate and a photographic ID (any ID!).

Birth-certificates are easy to “produce”. Either copy one - heck, they are plain paper and not meant to be tamper proof - or reapply for one choosing a suitable photographic ID. Be aware that this particular step may upset the law-enforcement community again, so be sure to do it right or to not do it. (Of course, as a good citizen I do not encourage any illegal activities of any kind. And if you have to do such things, then for heavens sake make sure that you at least prepare well.)

So how to get the photographic ID? Make one yourself. There are plenty of companies that offer services to produce company staff IDs or IDs for staff members for local events. They dont even check much on the background - so as long as you dont want to produce IDs from well-known locale companies you shall be fine. Just make sure, the ID you produce has a “official” looking name and appearance. If everything fails, register at the local library, in case they have a photo-ID for their members.

So once you have the ID, the birth certificate and the fake scottish accent, you are fine to apply for a PPS number. Welcome, newborn irish citizen “Donald Duck”.

April 9, 2009

Opening documents with their default application

Filed under: Uncategorized — Tags: — Thomas @ 12:42

When developing Java, 99% of all tasks can be solved in a platform independent manor right out of the box. But then there is that last tiny bit of tasks, that is inherently platform specific, as it has to deal with the weirdness of the host-OS.

For the Pentaho Report-Designer, we support the three major desktop platforms - Linux, Windows and Mac OS. When ever we generate a document (be it when we preview the report or when we do a actual report-run), we have to show the user the newly generated document. The cheesy way of doing this is to assume a reasonable set of default applications. For PDF, we would for instance default to launching the “Acrobat Reader”. “Hey, I dont use closed-source applications from monopolistic vendors on my machine!” you may scream out in terror. And yes, assuming a set of “standard” applications when there is no standard is not a good thing.

Luckily, there is hope. Each operating system has a magic way of launching the “right” application for a given document. Of course, its not documented at all. So calling on numerous years of messing around with operating systems, here’s my cook-book for opening documents.

1. Linux (and also Solaris)

On most modern distributions, the mapping between mime-types and corresponding applications happens based on the rules expressed in a file called /etc/mailcap(5). The command-line utility run-mailcap(8) reads the mailcap file and automatically launches the right application for the given file. So running the default application is easy:

run-mailcap <your-file>

or in Java:

Runtime.getRuntime().exec(new String[]{"run-mailcap", pathToYourFileAsString });

2. Windows

On Windows, file associations are stored in the registry. The registry is a binary file, so parsing that one is no fun. Of course, its also available via native C functions, but shipping platform dependent libraries is so 1995! Luckily, Microsoft drives a course of “never delete a documented function” and thus, the old tricks from Windows 95 still work on all recent Windows versions. Back in the old days, the most powerful toy to mess around with a system was the “rundll32″ utility. This utility can be used to invoke functions from a library (dll-file), without having to write a C-programm (I’d rather read out Perl-regular expressions in the public than to go back to C again).

The Shell32.dll contains a nice function to run the default application called “ShellExec_RunDLL”. So let’s abuse that.

rundll32.exe "SHELL32.DLL,ShellExec_RunDLL" <yourfilehere>

3. Mac OSX

MacOS is a weird thing. Applications are stored in bundles, well, directories with a lot of binary gibberish to know what entry in there is executable. Its equally inaccessible as the Windows Registry and the Apple Developer Connection is a great place to never find anything (who would have guessed that the entry called “CFBundle” describes the application directory structure). The File-open dialog of course, treats these application bundles as if they where files, but sadly no one told the Unix-shell that these things are executable.

So we have to find magic again, this time by using the open(1) command to parse whatever makes a bundle so special and then to execute the correct native application inside the bundle.

If you just want to open the file with the default application, use

open <yourfile>

If you know the application bundle and want to open the file with that specific application, you cant do that directly. Again you have to use the open(1) command, but this time with a extra parameter:

open -a <pathtotheapplicationbundle> <yourfile>


April 4, 2009

Subversion version troubles

Filed under: Uncategorized — Thomas @ 15:33

For all our development, we use Subversion as our sourcecode management tool. Usually, it works quite well, until you make a mistake that leads to using various versions of this tool across your systems.

The situation:

As a command line addict, I’m heavily dependent on the command-line tools for my daily work. I have three systems I work with on a regular base:

  • A Ubuntu desktop for the main development work while being in Germany, that ships with Subversion 1.4.4.
  • A Windows based notebook, for presentations and meetings, that is made usable via Cygwin and that comes with Subversion 1.4.
  • A MacBook for development and work in the field, that comes with subversion 1.4 as well.

So the situation is easy. All systems use the same command line version, and thus, Thomas should be happy. (That was back in mid-March.)

Let’s start the disaster that keeps me busy for the next days: “apt-get dist-update”

Subversion is now 1.5 on Ubuntu (via hardy-backports). And all of the sudden, my Windows and Mac systems are locked out, as Subversion *always* auto-updates the repository on any write access. Great! (The Ubuntu system is the only system with a sane filesystem (case-insensitive, unlimited path length) and a sane set of file-server capabilities (smb, nfs)).

Lets try to solve the problem. Upgrade the Windows system. Thats relatively easy, thanks to Cygwins update function. I end up with Subversion 1.5 for now. Luckily I dont use Torture-SVN on Windows, so that system is done. And more luckily, the Cygwin package maintainers did not yet publish packages for Subversion 1.6. But the next update promises to be fun.

Lets go to the Mac. Oh, SVN is bundled with XCode. So there is no update option available. Yeah, monolithic bundling makes life so easy! And no sane person would use public package systems with repositories that can hold updated versions. Luckily a Mac is nothing more than a Unix with a large set of proprietary extensions, so there is hope. So lets dig into the core.

MacPorts would want to install SVN 1.6. Of course, this version is not compatible with the older versions. And of course, there are no packages available elsewhere. And of course, there is no “downgrade”. Failure.

Next try: Lets see whether there is a pre-compiled download at the subversion site itself. After some digging in the archived parts of the site, yes, there is a Subversion 1.5.6 download. Download, install, done? Not really. Now its time for commandline configuration first, as the old 1.4-client is still installed. A quick edit later, the command line is happy.

I don’t even want to think about the monumental task to update the various VMWare images that sit on top of these physical machines. Life’s no fun when being forced to play admin.

Lessons learned:

  • Never upgrade
  • If you upgrade, take a week off afterwards to clean up the carnage - or
  • Wait with the upgrade until all your systems have all your packages available (or until the old packages dont work anymore, whichever comes last).

March 16, 2009

Crossing the border: 500k lines of Java Code

Filed under: Uncategorized — Thomas @ 17:44

For a long time we sneaked around the magical barrier, and chopped down old code to keep the project small. But now, with the Citrus report-designer at our side, we no longer fear.

Today, Ohloh finally reported that we now have over 500.000 lines of Java code in our project (trunk only, not counting the various branches). We are getting enterprisy now.

In the meantime, the report-designer has undergone yet another face-lifting.

The editor canvas now is a beauty on its own. The old and deliberatly ugly Swing-Wizard is gone and has been replaced by a much cleaner Xul-Wizard User-Interface.

Curious to see that new look? Then go grab the latest CI build and see it yourself.

February 27, 2009

Pentaho Reporting Citrus Milestone 2 is seen in the wild

Filed under: Uncategorized — Thomas @ 21:36

Yesterday, the 5th sprint (a bug-fixing sprint) finished. There was some serious action going on and now the product is a lot more fun to use than before.

How much fun? Watch the video of the End-of-sprint demonstration.

Why dont you download it and see how it works. If - no, when - you find bugs, please tell us via either the forum or JIRA.

Download the milestone from our Sourceforge page.


Powered by WordPress