db4o and Groovy Saturday, July 08, 2006

I want to say thanks to Christof and the rest of the gang at db4o as they were kind enough to send me a hardcover copy of The Definitive Guide to db4o. The book showed up this week. I have not had time to read all of it at this point but at a glance it looks like the authors have covered a lot of interesting territory inlcuding replication with Hibernate, which I expect is going to be a big win for db4o in the eyes of a lot of folks.

I have worked up some interesting Groovy integration capabilities for db4o. As soon as I have some time to clean some of that up and get it documented I will be sharing that.

I wish continued success to all of the guys at db4o.

New Groovy Spec Lead Wednesday, June 14, 2006

Today the official announcement was made that Guillaume Laforge is the new Groovy JSR-241 Spec lead.

The whole Groovy community is anxious about the next few months hoping that 1.0 will be finished up. Congrats and good luck to Guillaume!

Groovy And JScience Sunday, June 04, 2006

I have been spending a lot of time investigating Groovy lately. I wanted to dig in to writing a Groovy Category so I started writing some code to support things like this...

def duration = 30.minutes + 2.hours

Before I got too far along there I discovered that John Wilson had already implemented a lot of that in his TimeCategory which is part of Google Data Support.

I had recently read a JNB article that Lance Finney had written on JScience and that gave me another idea. I decided to write a Groovy Category that would use JScience to support things like this...

def len = 2.kilometers + 500.meters

Since I wanted to do this as an exercise I decided to not even do any research to find out if anyone had already implemented something like this. I wanted to build this for my own benefit as a learning exercise. I started by expressing some of my own requirements in the form of a Groovy Unit Test like this...

class JScienceTest extends GroovyTestCase {

  void testConversions() {
    use(JScienceCategory) {
      def len = 2.kilometers
      assertEquals 2.0,    len.kilometers,  0.00001
      assertEquals 2.0E3,  len.meters,      0.00001
      assertEquals 2.0E5,  len.centimeters, 0.00001
      assertEquals 2.0E6,  len.millimeters, 0.00001
      assertEquals 2.0E9,  len.micrometers, 0.00001
      assertEquals 2.0E12, len.nanometers,  0.00001
      
      def len2 = 400.centimeters
      assertEquals 4.0E-3, len2.kilometers,  0.00001
      assertEquals 4.0,    len2.meters,      0.00001
      assertEquals 4.0E2,  len2.centimeters, 0.00001
      assertEquals 4.0E3,  len2.millimeters, 0.00001
      assertEquals 4.0E6,  len2.micrometers, 0.00001
      assertEquals 4.0E9,  len2.nanometers,  0.00001
    }
  }
  
  void testAddition() {
    use(JScienceCategory) {
      def len = 2.kilometers + 500.meters
      assertEquals 2.5, len.kilometers, 0.00001
    }
  }
  
  void testSubtraction() {
    use(JScienceCategory) {
      def len = 3.kilometers - 500.meters
      assertEquals 2.5, len.kilometers, 0.00001
    }
  }
  
  void testMultiplication() {
    use(JScienceCategory) {
      def len = 3.kilometers * 2
      assertEquals 6.0, len.kilometers, 0.00001
    }
  }
  
  void testDivision() {
    use(JScienceCategory) {
      def len = 42.meters / 3
      assertEquals 14.0, len.meters, 0.00001
    }
  }
}

My Groovy Kung Fu is still developing and I had not done anything at all with JScience before this. Even with those limitations I was able to get all of those tests passing in about an an hour.

Would you rather do this in Java...

Measure length = Measure.valueOf(50, SI.CENTI(SI.METER)).plus(Measure.valueOf(25.0, SI.METER));
Measure lengthInCentimeters = length.to(SI.CENTI(SI.METER));
System.out.println("length in centimeters is " + lengthInCentimeters.getEstimatedValue());

Or do this in Groovy...

def length = 50.centimeters + 25.meters
println "length in centimeters is ${length.centimeters}"

Isn't Groovy fun? :)

Operator Overloading In Groovy Tuesday, May 30, 2006

I have been doing some Groovy development lately. I just spent a few minutes spinning around a quirky behavior that makes perfect sense to me now but at first had me scratching my head. Take a look at this...

def sqlDate = new java.sql.Date(System.currentTimeMillis())
sqlDate += 2

That code is creating an instance of java.sql.Date and adding 2 days to it. My moment of confusion comes from the fact that after adding the 2 days the sqlDate reference no longer points to a java.sql.Date object but instead points to a java.util.Date object. Hmm... What is going on?

The operator overloading is being inherited into java.sql.Date from java.util.Date. The plus method in java.util.Date is returning a java.util.Date, which makes perfect sense. Since that method is inherited into java.sql.Date and not overridden then when it is invoked it returns a java.util.Date.

Try this...

def sqlDate = new java.sql.Date(System.currentTimeMillis())
println sqlDate.class
sqlDate += 2
println sqlDate.class

db4o is pretty interesting Monday, December 05, 2005

I have done a lot of work with a number of Java persistence solutions including JDBC, JDO, Hibernate, JSR-220 Persistence and Prevayler (have just tinkered with Prevayler... no real work). Just recently I started investigating yet another persistence solution called db4objects (db4o). db4o has versions for Java, .NET and Mono. I have only investigated the Java version. In the time that I have spent investigating db4o I have found some pretty interesting stuff.

One thing that stands out is that db4o allows you to persist plain 'ol Java objects as plain 'ol Java objects. POJOs need not extend any magic base class or implement any special interface. POJOs need not have any special id field. POJOs need not have any special constructor. There is no requirement for a no-arg constructor or even a public constructor. db4o doesn't require any object descriptors (XML or otherwise) and doesn't require you to mark persistent classes up with annotations. db4o does not require that persistent fields have Java Bean compliant getters and setters. db4o pretty much will take your objects as they come.

db4o can run in embedded mode which means that all of db4o is inside of your applications process. There doesn't need to be a separate db4o process running somewhere. There can be if that suits your deployment needs, but there doesn't need to be.

db4o has some pretty robust schema evolution capabilities that allow fields to be added, removed and renamed. There is support for moving classes to new packages.

db4o is an OO database. db4o is not an object to relational mapping tool, the db is an OO db.

One of the issues I initially had concerns about was performance once the database accumulated large numbers of objects. To test some of this, I created a fairly simple object model to represent music cds. The classes look something like this...

public class CDArtist {
  private String name;
  private List<CD> cds = new ArrayList<CD>();
  // constructor and methods snipped...
}


public class CD {
  private String title;
  private List<CDTrack> tracks = new ArrayList<CDTrack>();
  // constructor and methods snipped...
}

public class CDTrack {
  private String title;
  private int trackNumber;
  private CD cd;
  // constructor and methods snipped...
}

I downloaded a gaboodle of data from freedb.org and wrote a simple parser to turn that data into instances of my classes and started dropping those into my db4o database. At present I have about 75,000 artists, 185,000 cds and over 2,000,000 tracks in the database. I realize that for a lot of situations even those 2,000,000+ tracks don't really amount to a lot of data but that is what I am currently working with. It is enough data to exercise some of the things I wanted to look at.

db4o supports 3 query techniques...

• Query By Example


• Native Query


• S.O.D.A.

I will show a simple example of each of these here and include some performance figures.

Query By Example (QBE)

The following code uses QBE to retrieve all CDs in the database that contain a track with the name "The Trooper".

Db4o.configure().objectClass(CDTrack.class).maximumActivationDepth(0);
Db4o.configure().objectClass(CDTrack.class).objectField("title").indexed(true);

// my custom factory
ObjectContainer db = ObjectContainerFactory.get().createObjectContainer();

CDTrack myCandidateTrack = new CDTrack(null, 0, "The Trooper");
List<CDTrack> results = db.get(myCandidateTrack);
for (CDTrack track : results) {
  CD theCD = track.getCd();
  System.out.println(theCD);
}

db.close();

That query completes in less than 300-400 milliseconds. That is querying over 2,000,000 tracks, identifying the ones that match the name "The Trooper" and retrieving the cd that the track belongs to (note the call to track.getCd() inside of the loop).

S.O.D.A.

Db4o.configure().objectClass(CDTrack.class).maximumActivationDepth(0);
Db4o.configure().objectClass(CDTrack.class).objectField("title").indexed(true);

// my custom factory
ObjectContainer db = ObjectContainerFactory.get().createObjectContainer();

Query query = db.query();
query.constrain(CDTrack.class);
query.descend("title").constrain("The Trooper");
ObjectSet<CDTrack> results = query.execute();
for (CDTrack track : results) {
  CD theCD = track.getCd();
  System.out.println(theCD);
}

db.close();

That S.O.D.A. query executes in about 100 milliseconds. Again querying over 2,000,000 tracks and retrieving the matching tracks and their containing cds.

Native Query

This is what a native query might look like in db4o.

Db4o.configure().objectClass(CDTrack.class).maximumActivationDepth(0);
Db4o.configure().objectClass(CDTrack.class).objectField("title").indexed(true);

// my custom factory
ObjectContainer db = ObjectContainerFactory.get().createObjectContainer();

List<CDTrack> results = db.query(new Predicate<CDTrack>() {
  public boolean match(CDTrack candidate) {
    return candidate.getTitle().equals("The Trooper");
  }
});
for (CDTrack track : results) {
  CD theCD = track.getCd();
  System.out.println(theCD);
}

db.close();

This approach has some nice benefits. One is that you get real compile time type safety. The query isn't some arbitrary string that might or might not be legal at runtime. The query is real Java code that gets compiled. That is nice. However, that Predicate looks a little suspect to me. Judging from looking at the code it seems that the db4o engine is going to have to create all of my CDTrack objects and pass each of them one a time to my match(CDTrack) method so I can decide which of them match my criteria. Since I have over 2,000,000 tracks that can't be efficient. The code above executes in about 100-150 milliseconds. I am still querying those 2,000,000+ tracks and retrieving all the same stuff I was retrieving in the previous examples. What is going on here at runtime is that db4o is doing some slick class loading voodoo and figuring out what my Predicate would do, then it optimizes all of that away by turning my Predicate into a S.O.D.A. query. Run the code in a debugger and find that my match(CDTrack) method never actually gets called. There are limits here. The optimizer does a good job of figuring out what you intended to do but you can do things in your Predicate that the optimizer can't figure out in which case the Predicate cannot be optimized away and then the engine will have to create all of those CDTracks and pass them to the match method. This is easy enough to sort out at development time if you need to make sure the Predicate will be optimized. While experimenting with this don't try to put a System.out.println call or logging calls in your Predicate to monitor if your Predicate is getting called or not. Those fall in the category of things that the optimizer can't handle and a side effect of them being there is that the method will not get optimized away. There is a callback mechanism you can hookup to retrieve notifications that indicate when a Predicate is optimized and when it isn't.

ObjectContainer db = ObjectContainerFactory.get().createObjectContainer();

((YapStream)db).getNativeQueryHandler().addListener(new Db4oQueryExecutionListener() {
  public void notifyQueryExecuted(Predicate filter, String msg) {
  }
});

That callback will be notified when db4o first deals with any particular Predicate. The msg argument will by "DYNOPTIMIZED" if the query has been dynamicallly optimized. msg will be "UNOPTIMIZED" if the query could not be optimized. msg will be "PREOPTIMIZED" if the query had been pre optimized. db4o has some bytecode manipulation tools to preoptimize Predicates but I have not investigated that.

They are working on some Hibernate replication modules that will allow a db4o database to be kept in synch with a relational database via Hibernate. That code is still in development and I haven't looked at any of that.

The discussion forums are pretty active and as far as I can tell there is a lot of momentum behind the effort right now.

db4o is distributed under a couple of different license. There is a GPL version available for open source projects, experimenting and internal projects. There is also a commercial license available for commercial, non-open source applications. As far as I know, the GPL version is the same software as the commercial version. The restrictions have to do with distribution, not the software itself.

No reasonable person is going to claim that db4o is the silver bullet of persistence but it is interesting stuff and probably makes a lot of sense for a lot of applications. If nothing else, it is a good thing to be aware of.

Is Sun Pimping The Java Name? Friday, April 29, 2005

Does the name "Java" really belong on these products?

Sun Java Desktop System

Sun Java Workstation W1100z

Sun Java Workstation W2100z

Free IntelliJ IDEA License Thursday, March 03, 2005

A few weeks ago the guys at JetBrains/IntelliJ announced the availability of IntelliJ IDEA licenses that they were making available free of charge to qualifying open source developers. See http://www.jetbrains.com/idea/opensource/ for details.

I already own a current license for IntelliJ IDEA and that license allows me to do whatever development I like (personal, commercial, open source, whatever). I can even install that license on as many machines as I like as long as I am the only person using them. Contrast that with the more limited open source license which is only allowed to be used for open source development and may be limited further to only being used on development of open source projects that the guys at IntelliJ approve. I am not exactly sure about that last part, but the license is limited to open source work. Anyway, because I already own a less restrictive license, the free license doesn't really buy me anything, but I wanted to participate in their program, so I submitted a request based on my involvement with JarSpy. Shortly after sending the request I got an email response letting me know that they have received lots of requests and that each one needs to be evaluated individually and that will be time consuming. I wasn't too worried about it so I saved the email and went on with my business.

Fast forward to today. Today I received an email from them letting me know that they have approved my request and are issuing me a free license. Whooo Hoo! In a way, this is of absolutely no consequence whatsoever. In another way, I am still glad that they are extending this offer and that they are not being overly restrictive about it.

I hope that this helps IntelliJ IDEA continue to grow in its popularity and subsequently leads to its continued life of innovation.

JDO 2.0 Has Been Approved! Tuesday, March 01, 2005

I am pleased to report that JDO 2.0 has been approved! This is great news for the JDO community. The initial public review ballot was voted down back in January with a vote of 10 against, 5 for and 1 abstainer. The reconsideration ballot passed with a vote of 0 against, 12 for, 1 no vote and 3 abstainers.

I don't think this vote hurts the JSR 220 effort but I think it does a lot of good for JDO users and JDO vendors.

Free IntelliJ IDEA License Tuesday, February 08, 2005

The guys at IntelliJ are giving away free licenses for IDEA to open source projects. I haven't read all of the details, but at a glance this looks awesome.

Details available here.

Bad News For JDO Wednesday, January 19, 2005

I am very disappointed to see the results of the JSR 243 public review ballot. I believe this means that the Expert Group has 30 days to submit an updated draft for a revote. If a revised draft is not submitted in that timeframe or if the revote fails, the JSR will be closed. If anyone thinks that is not correct, please share what you understand about the process.

I knew that JSR 220 was eventually going to succeed JDO 2.0 but as I understand it, there won't be anything "real" coming from JSR 220 until 2006.

Given the vote count and some of the comments submitted by the voters, it doesn't to me look very likely that the draft will be revised in such a way to change voter's minds. I think that if the vote is going to turn around it would have to be in response to community feedback but I don't know how much of that there will be.

I really am disappointed in this whole thing. ;(

Someone say it ain't so...

Already I am seeing talk of org.jdo.* popping up.

JarSpy Plugin For IntelliJ Tuesday, January 11, 2005

Tonight I was looking for the IntelliJ Plugin Wiki but I couldn't remember the url so I did the obvious; I put "intellij plugin" into google. To my surprise, the first link in the search results referred to my JarSpy Plugin For IntelliJ. I can't say that I had forgotten ever having written that, but I can say that I haven't had any reason to think about it for a long time. A couple of years ago I got an email from one of the billions of JarSpy fans out there and the email was asking me for a JarSpy plugin for IntelliJ. I was an IntelliJ user at the time (I still am) and I was interested in taking a look at their plugin API so I went at it. It didn't take long to write the plugin (although I do remember wishing there was better documentation for their API, maybe that has improved since then).

JarSpy has turned out to be one of those little things that continues to popup for me from time to time. Just a couple of weeks ago at my regular poker game one of my friends, who I had never had any reason to discuss JarSpy with, said something like "hey, I was using JarSpy today and...".

It has probably been more than a year since I made any code changes to JarSpy. I have some interesting ideas that I would like to add. Maybe I will get some time to play with that soon.

Ant Docbook Styler Suite Thursday, December 30, 2004

If you do any publishing with Docbook you should take a look at the Ant and Docbook Styler Suite. I recently stumbled across that and have so far been very impressed. Installation is as simple as executing a target in the build file provided with the distribution and using the library is as simple as adding a task in your Ant build file to execute a target in their build file. You have to provide a handful of property values so their task knows where to find your docbook source, what to generate, etc. but it really is a snap to use. Their 3 or 4 page instructions (included in the distribution of course) are very clear and got me up and running within 10 minutes or so.

I have looked at other Ant + Docbook solutions and they are all a wirey mess of dependencies and you have to download Docbook stylesheets separately and plugging in FO support is another twisted mess, and on and on. The Styler Suite was much much simpler and seems to do a very nice job.

If you are a docbook author but not a Java developer, this may turn out being more headache than it is worth but if you already have Ant and Java installed, using this is a cinch.

Thanks to Dawid Weiss for the slick tool!

Code Cracker Lives On

I bought my first Palm Pilot back when the Palm IV series first came out. I think it was 1998. Soon after I bought the device I was interested in writing software to run on the thing. I picked up a copy of OReilly's Palm Programming: The Developer's Guide, and I was on the ground running. I remember spending a weekend just playing around with the API and I built a very simple little game that played like the old MasterMind game that most of us played as a kid. It wasn't that I wanted to write a MasterMind game, that was just something simple that I could use to help learn the PalmOS API. I expected to just throw the thing away when I was done but I found myself enjoying playing the game when I had a minute or 2 to kill (like standing around in front of the microwave at work waiting for my lunch to cook) so I held on to it. I gave the application a proper name, CodeCracker, created a very simple web page that explained a little about the game and I put it on my personal web site and made the game available for download. I remember being surprised at how many people found the page and even more surprised at how many people bothered to take time to send a thank you note to me. A couple of years later I was cleaning up a hard drive and stumbled across the source code. I decided to put the source code on SourceForge for safe keeping and it has been there ever since.

Just today I received an email from a guy in Italy saying thanks for the game and offering some suggestions for how to improve the game.

I spent a weekend 6 or 7 years ago playing around with writing something for the PalmOS and a very simple game evolved. It is sort of interesting how these little things linger on.

CodeCracker Lives On! ;)

Hostility Directed At Prevayler Monday, December 27, 2004

Prevayler seems to be getting a lot of blog attention all of a sudden. I don't know what kicked off the spike in attention, but something seems to have done just that.

Prevayler is yet another framework for POJO persistence. What makes Prevayler unique is the fact that it takes an approach that your entire data model is in memory all of the time. Because it is always reading from RAM, performance benefits are touted as high as 9000 times faster than Oracle and 3000 times faster than MySQL (I won't go into all of the details behind those numbers, see their site for more details). One of the costs of this approach is directly related to RAM limitations. If everything is loaded into RAM all of the time then the amount of RAM is an obvious potential road block.

Some of the blogging rants I have seen go to great lengths to elaborate on the flaws of this approach. This guy rants with some hostility for several pages about Prevayler's shortcomings while this guy seems to take a more calm pragmattic look. Some of the attacks I have read are really pointed at the idea that Prevayler is a ridiculous idea that could never replace our relational databases. I don't think Prevayler is proposing that they are really a threat to Oracle or any other DB vendor. I think they are proposing a solution that makes sense for some systems but not all. I don't know if they are right or not, but I think that is what they are proposing.

I am not a Prevayler proponent and I am not taking a position to defend Prevayler. I have played around with their tool a little and it is interesting but I am really not a Prevayler evangalist. What I find interesting about the attacks on Prevayler is that a lot of them seem to be very passionate and fired up. If Prevayler was really a totally insane ridiculous idea I don't think anyone would even pay attention. Maybe this is just another manifestation of the idea of how emotionally connected Enterprises (that means people) get to their databases and even the idea that it might not be right is enough to launch the missiles.

Naughty Or Nice? Saturday, December 25, 2004

Apparently, this guy sold his kids' Christmas presents on eBay because the kids were naughty. My wife and I sometimes joke about selling our boys on eBay when they misbehave (that is right, the boys... not their toys.). I never considered selling their stuff. ;)

As I type this post I am looking out over a sea of toys and schtuff all wrapped up and spread around our family room. In about 6 hours from now I expect a certain 7 year old boy to be jumping up and down on my bed yelling something about Santa Clause having been in our house. I better get to bed.

Happy Holidays To All!

JDO 2.0 Sunday, December 19, 2004

The Public Draft of JSR-243 is available for download at http://jcp.org/en/jsr/detail?id=243. This JSR is defining the JDO 2.0 spec.

There is a lot of new and cool stuff being proposed for 2.0. One item in particular that I am interested in is the whole detachment idea. The idea is that we would be able to detach a persistent object from the data store, modify the object and then later re-attach that object and commit the changes. The changes could be commited by a different PersistenceManager and could even be committed in an entirely different VM.

Another nicety proposed for 2.0 is the formalization of O/R metadata for defining indexes, foreign keys and constraints. Some JDO implementations already support this sort of thing via vendor extensions in the meta data. ObjectDB uses this approach. Other implementations suggest that you let their tool create the tables and columns and then you go in after the fact and define your indexes, constraints etc. in the database yourself. TJDO takes this approach. It will be nice to have this formalized in the metadata for portability.

J2SE 6.0 Snapshots Available Thursday, December 16, 2004

I just noticed that it looks like J2SE 6.0 snapshot releases are being made available at https://j2se.dev.java.net/. Have I finished upgrading to J2SE 5.0 yet??? ;)

What Is Wrong With Our Web Apps? Friday, December 10, 2004

Last night Rob Smith gave a great presentation on Tapestry at the St. Louis Java SIG. I liked a lot of the things that I learned about Tapestry but as I listened and learned I was thinking about what sorts of problems does Tapestry really solve for me. Then I started thinking about how that relates to many other web app frameworks that are out there. Just off the top of my head without spending too much time thinking about it or Googling or anything like that, I came up with this list of tools that help us build web apps...

• JSP


• Servlet


• Java Server Faces


• Velocity


• Tapestry


• Struts


• Spring


• Tiles

Several of those can be thought of as being direct competitors of each other.

These technologies all try to help improve our web apps by making it easy to do things like separate the presentation layer from the data. Some provide tools to help with state management. Some help with scalability. Those are all important things to address and sometimes are difficult to address. If the tools can help solve those problems that is a good thing of course.

Web apps are hard to build. Few would still entertain fantasies about some magic web tool that solves all of the Enterprise's problems, is easy for everyone in the Enterprise to use and doesn't call for much development time. We all know that can't happen. If we didn't know that in the beginning, we know it now.

I believe that a lot of things are well suited as web apps and for the purposes of this conversation I am using the term "web app" to mean an application that uses a web browser as its primary front end. However, I am wondering if a lot of us are spending a lot of time and effort (that sounds like money to me) building web apps that really shouldn't be web apps. At last year's No Fluff Just Stuff Java Symposium in St. Louis I remember a comment that Bob Lee made during one of the expert panels. Bob said something very close to "Can we all just agree to stop building browser based apps?".

I don't think that web apps are bad. I think a lot of things work very well as web apps. In addition, as a Software Engineer I find working with a lot of the web app frameworks to be a lot of fun. I am just wondering if we are collectively trying to push pegs into the wrong shaped holes too often.

More and more I am feeling like we really need a fundamentally different (and better) way to build distributed systems. I can't exactly put my finger on it, but somewhow I feel like we are solving a lot of the wrong problems and that in 6 months or 6 years or however long it takes us to come up with this next generation of ideas we are going to look back and wonder how we ever managed to operate our businesses any other way.

What do you think about any of that? Let me throw out some specific questions for thought.

What problems do you have in building web apps that are not solved very well by the existing frameworks?

What types of projects have you been involved with that were architected as web apps that should not have been and what made them poor candidates to be built as a web app?

What types of projects have you been involved in that were not architected as web apps that should have been and what made them so well suited to be web apps?

What do you think is next?