Code Smart

Ruby script to list process info in Windows

2011-10-07T13:07:00.000-07:00

Below is a Ruby script to list process id and its command line in Windows.

To run it:
ruby pid.rb <process-name>
ie: ruby pid.rb java: will list java.exe and javaw.exe processes


class ProcessInfo
  attr_accessor :cmdLine, :pid
  
  def self.parseWmic(cmd)
    result = `#{cmd}`
    raise("Error: " + result) unless $? == 0
    processes = []
    pinfo = nil
    result.split(/\r?\n/).each do |line|
      next if line =~ /^\s*$/
      if line =~ /CommandLine=(.*)/i
        pinfo = ProcessInfo.new
        pinfo.cmdLine = $1
      elsif line =~ /ProcessId=(\d+)/i
        pinfo.pid = $1
        processes << pinfo unless pinfo.pid.to_i == $$.to_i
      end
    end
    return processes
  end
  
  def self.queryProcess(processName)
    return self.parseWmic("wmic process where \"name like '" + processName + 
                "%'\" get ProcessID,Commandline /format:list")
  end
  
  def self.queryJavaProcess
    return self.queryProcess('java')
  end  
  
  def to_s
    @pid.to_s + " " + @cmdLine
  end	
end	

puts ProcessInfo.queryProcess(ARGV[0])

Equivalent of ps -ef | grep java in Windows

2011-10-06T16:54:00.000-07:00

If you want to list a process and its command line, process id in Windows, this is the command:

 wmic process where "name='java.exe'" get ProcessID, Commandline /format:list

That is the equivalent of "ps -ef | grep java" in Linux. Then you could use taskkill /f /pid <PID> to kill it.

Install JDK on Windows without a UI

2011-09-23T16:46:00.000-07:00

in CMD.exe shell:

./jdk-6u27-windows-x64.exe /s INSTALLDIR=\c:\jdk\jdk1.6.0_27-x64

This will allow you to install the JDK automatically without any GUI prompt, ideal for automation script.Note the forward slash before the c: drive is necessary.

Force JBoss to load your lib jars

2011-04-07T11:51:00.000-07:00

If you're running JBoss and you want it to favor librarires that go with your WAR instead of using what JBoss bundled with. Add this jboss-web.xml to your WEB-INF dir


<jboss-web>
  <class-loading java2classloadingcompliance="false">
  <loader-repository>
      org.terracotta:archive=all-in-one.war
      <loader-repository-config>
          java2ParentDelegation=false
      </loader-repository-config>
  </loader-repository>
  </class-loading>
</jboss-web>

Substitute "org.terracotta" for your package name and "all-in-one.war" for your WAR

QA Engineer needed at Terracotta

2010-11-04T10:51:00.000-07:00

If you're like testing softwares, writing applications to find bugs and figuring out performances, Terracotta is looking for QA engineers.

Very bizarre problem when running JDK 32 bit on a 64 bit Linux box

2010-07-15T16:47:00.001-07:00

I just noticed a very weird problem if you use a 32bit Sun JDK on a 64bit RedHat Linux.

The "user.home" system properties, returned by System.getProperty("user.home"), has the value of "?". Same thing happens with "user.name" property.

I'm not sure why this is but seems like a bug too me.


-bash-3.00$ /shares/jdk/hotspot1.6.0_16/bin/java ShowSystemProps | grep "user"
user.country = US
user.dir = /home/hhuynh
user.home = ?
user.timezone = 
user.name = ?
user.language = en

Correct values if I used 64bit JDK


-bash-3.00$ /shares/jdk/hotspot1.6.0_16_x64/bin/java ShowSystemProps | grep "user"
user.country = US
user.dir = /home/hhuynh
user.home = /export2/homes/hhuynh
user.timezone = 
user.name = hhuynh
user.language = en

This bug leads to Maven creating the local repository under the current working dir: "./?/.m2/repository" and leads to many confusing problems down the road.

Simple Bash script to rotate log

2010-07-14T16:04:00.000-07:00

If you have some program that doesn't handle its own log rotation, you could use the below script to rotate the log. Ideally, you would call this script in a daily crob job.


#!/bin/bash
logfile=$1
if [ ! -f $logfile ]; then
  echo "log file not found $logfile"
  exit 1
fi
timestamp=`date +%Y%m%d`
newlogfile=$logfile.$timestamp
cp $logfile $newlogfile
cat /dev/null > $logfile
gzip -f -9 $newlogfile

Maven gotcha: "post-integration-test" phase

2009-08-24T18:36:00.000-07:00

In Maven, "post-integration-test" phase can be used to do test clean up, for example, to shut down Cargo after the tests finished.

If you run "maven install" then the phases would be in this order:


pre-integration-test:  start Cargo (web server)
integration-test:      start tests
post-integratino-test: stop Cargo

However, if you just run "maven integration-test" to run the tests directly, "post-integration-test" phase is never called, which might lead to your cleanup process not being run.

So the workaround is to bind your clean up tasks to "integration-test" itself which will be run right after the tests run during that phase.

Maven Javadoc workaround: Unable to find package java.lang in classpath or bootclasspath

2009-08-08T21:53:00.000-07:00

Seen this before?


Embedded error: Error rendering Maven report: Exit code: 1 - com.sun.tools.javac.util.FatalError: Fatal Error: Unable to find package java.lang in classpath or bootclasspath
 at com.sun.tools.javac.comp.MemberEnter.importAll(MemberEnter.java:123)
 at com.sun.tools.javac.comp.MemberEnter.visitTopLevel(MemberEnter.java:509)
 at com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:446)
 at com.sun.tools.javac.comp.MemberEnter.memberEnter(MemberEnter.java:387)
 at com.sun.tools.javac.comp.MemberEnter.complete(MemberEnter.java:819)
 at com.sun.tools.javac.code.Symbol.complete(Symbol.java:386)
 at com.sun.tools.javac.code.Symbol$ClassSymbol.complete(Symbol.java:758)
 at com.sun.tools.javac.comp.Enter.complete(Enter.java:451)
 at com.sun.tools.javac.comp.Enter.main(Enter.java:429)
 at com.sun.tools.javadoc.JavadocEnter.main(JavadocEnter.java:53)
 at com.sun.tools.javadoc.JavadocTool.getRootDocImpl(JavadocTool.java:152)
 at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:330)
 at com.sun.tools.javadoc.Start.begin(Start.java:128)
 at com.sun.tools.javadoc.Main.execute(Main.java:41)
 at com.sun.tools.javadoc.Main.main(Main.java:31)
javadoc: error - fatal error

This error has baffled me for the last time. Though I don't understand why the plugin or javadoc would trigger this error, I found a workaround: adding rt.jar to the "bootclasspath":

<bootclasspath>${java.home}/lib/rt.jar</bootclasspath>


         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-javadoc-plugin</artifactId>
            <version>2.6</version>
            <configuration>
               <minmemory>128m</minmemory>
               <maxmemory>512m</maxmemory>
               <!-- fix error can't find java.lang package -->
               <bootclasspath>${java.home}/lib/rt.jar</bootclasspath>
            </configuration>
            <reportSets>
               <reportSet>
                  <reports>
                     <report>aggregate</report>
                  </reports>
               </reportSet>
            </reportSets>
         </plugin>

Starting Terracotta Server as a Windows service

2009-06-23T09:11:00.000-07:00

There's an easy way to set up Terracotta server as Windows service using the open source Java Service Wrapper.

I've used their Integration Method 1, which uses a wrapper WrapperSimpleApp to run Terracotta server main class com.tc.server.TCServerMain.

First, just download and install the latest version wrapper-windows-x86-32-3.3.5. Then make copy of conf/wrapper.conf and name it TerracottaServer.conf.

Fill out these needed properties:


# for logging
wrapper.java.command.loglevel=INFO

# Method 1 main class
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp

# classpath
# (1) is for WrapperSimpleApp
# (2) is for Terracotta Server - You'll need to fix up the path for your own use
wrapper.java.classpath.1=../lib/wrapper.jar
wrapper.java.classpath.2=d:/work/builds/terracotta-3.0.1/lib/tc.jar

# Java Additional Parameters
## NOTE: -server option only works with a JDK, not with JRE
wrapper.java.additional.1=-server 
wrapper.java.additional.2=-XX:+HeapDumpOnOutOfMemoryError
wrapper.java.additional.3=-Dcom.sun.management.jmxremote
wrapper.java.additional.4=-Dtc.install-root=d:/work/builds/terracotta-3.0.1

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=512

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=512

# Application parameters. This is where you specify TC server main class
wrapper.app.parameter.1=com.tc.server.TCServerMain

That's pretty much all you need. There are Batch scripts in the "bin" folder of the wrapper installation where you can install/uninstall your service. Just make sure you modify those scripts to point to TerracottaServer.conf file you made earlier.

Once you have the service install, you can start/stop the service by using Windows services manager or by using the scripts. There will be a log of the run under "/logs"

I've made a copy of the scripts and the TerracottaServer.conf for easy tryout.

Executing a Maven plugin from a Maven plugin

2009-06-03T22:44:00.000-07:00

I've searched for a way to call a Maven plugin from inside another plugin but couldn't find any. So I've put on my hacker hat and made it happen. It might not be the best way to do it or even orthodox but here goes.

For example, Terracotta has a Maven plugin, namely "tc" and I want to add a 'help' goal to it which works exactly like the help plugin

With the help plugin, you can do this:


mvn help:describe org.terracotta.maven.plugins:tc-maven-plugin -Ddetail

But it's a little verbose. I want to do

 
mvn tc:help

which would accomplish the same thing. To be able to do that, I have my "tc" plugin depended on "help" by declaring it as a dependency in the pom.


    <dependency>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-help-plugin</artifactId>
     <version>2.1</version>
    </dependency>

Then I looked in the source of the "help" plugin, specifically DescribedMojo.java which handles to goal "help:describe" as shown above. So basically, if I can construct a DescribeMojo instance and fill in the needed info, you'll be able to execute it just like your own mojo. The trick is, DescribeMojo has all of it fields "private" and no setters.
So I hacked the hell out of it and dug into its privates by using reflection. This is a no-no in so many books but I feel I'm working on a known version (2.1) of the "help" plugin so the chance of its API change is slim.


/**
 * Print help for all known goals
 * 
 * @author hhuynh
 * 
 * @goal help
 */
public class HelpMojo extends AbstractMojo {
  .....

  /**
   * @parameter expression="${project.remoteArtifactRepositories}"
   * @required
   * @readonly
   */
  private List remoteRepositories;

  /**
   * The goal you want to see help. By default help prints for all goals
   * 
   * @parameter expression="${goal}"
   */
  private String goal;

  public void execute() throws MojoExecutionException, MojoFailureException {
    DescribeMojo describeMojo = new DescribeMojo();
    setValue(describeMojo, "artifactFactory", artifactFactory);
    setValue(describeMojo, "pluginManager", pluginManager);
    setValue(describeMojo, "projectBuilder", projectBuilder);
    setValue(describeMojo, "project", project);
    setValue(describeMojo, "settings", settings);
    setValue(describeMojo, "session", session);
    setValue(describeMojo, "localRepository", localRepository);
    setValue(describeMojo, "remoteRepositories", remoteRepositories);

    setValue(describeMojo, "plugin", "org.terracotta.maven.plugins:tc-maven-plugin");
    setValue(describeMojo, "detail", true);
    setValue(describeMojo, "goal", goal);
    
    describeMojo.execute();
  }

  private void setValue(Object o, String field, Object value) throws MojoFailureException {
    Class c = o.getClass();
    Field _field;
    try {
      _field = c.getDeclaredField(field);
      _field.setAccessible(true);
      _field.set(o, value);
    } catch (Exception e) {
      throw new MojoFailureException(e.getMessage());
    }
  }
}

The 3 important fields I had to fill out are:

plugin: full name of the plugin you want to print help

detail: I want full description by default

goal: If specified, it only prints help for that goal

There are other component fields that Maven will normally inject them into your own mojo automatically. For example, the field "remoteRepositories" is needed by DescribeMojo and since we construct it by hand, we have to inject the value of this field ourselves. But you can get the value of it for free by Maven so all I need to do is just declaring it and passing it along.

Voila, you got a fully functional DescribeMojo instance and all that left is to call "execute()" on it.

So that's how the hack is done.

Sharing data of Spring beans with Terracotta

2009-04-27T00:28:00.000-07:00

This is a quick demo showing how you would share data of Spring beans between different JVMs using Terracotta.

The example is based on the Voting Booth of Spring By Example website (a great website to learn Spring by the way)

In this example, I have 2 beans defined in application-context.xml


 <bean id="recorder" class="org.terracotta.demo.LocalVoteRecorder" />

 <bean id="votingBooth" class="org.terracotta.demo.VotingBooth">
  <property name="voteRecorder" ref="recorder" />
 </bean>

So simply, the 'votingBooth' bean has a reference to the 'recorder' bean which has a map of candidates and their votes.


public class VotingBooth {
  VoteRecorder recorder = null;

  public void setVoteRecorder(VoteRecorder recorder) {
    this.recorder = recorder;
  }

  public void vote(Candidate candidate) {
    recorder.record(candidate);
  }
  
  public void printResult() {
    recorder.tallyVotes();
  }
}

/* ************* */
public class LocalVoteRecorder implements VoteRecorder {
  final static Log                      logger      = LogFactory
                                                        .getLog(LocalVoteRecorder.class);
  private final Map<Candidate, Integer> votesRecord = new ConcurrentHashMap<Candidate, Integer>();

  public void record(Candidate candidate) {
    int count = 0;

    if (!votesRecord.containsKey(candidate)) {
      votesRecord.put(candidate, count);
    } else {
      count = votesRecord.get(candidate);
    }
    count++;
    votesRecord.put(candidate, count);
  }

  public void tallyVotes() {
    for (Candidate candidate : votesRecord.keySet()) {
      logger.info("Candidate '" + candidate + "' has "
          + votesRecord.get(candidate) + " votes.");
    }
  }
}

Spring will do its magic and inject the right data to into main program at runtime. To demo, I have the main function vote 5 times between 2 known candidates then print out the result in the end.


public class Main {

  final static Log         logger     = LogFactory.getLog(Main.class);
  final static Random      rand       = new Random();
  final static Candidate[] candidates = new Candidate[] {
      new Candidate("Hung"), new Candidate("Nick") };

  public static void main(String[] args) {
    ApplicationContext applicationContext = new ClassPathXmlApplicationContext(
        "/application-context.xml");

    VotingBooth votingBooth = (VotingBooth) applicationContext
        .getBean("votingBooth");

    // vote 5 times
    for (int i = 0; i < 5; i++) {
      Candidate luckyCandidate = candidates[rand.nextInt(2)];
      votingBooth.vote(luckyCandidate);
      logger.info("voted for candidate " + luckyCandidate);
    }
  
    votingBooth.printResult();
  }
}

So if you run this Spring app as is, it will print the result of that run and exit. Quite simple.

Now let say we want to maintain a voting record that would last beyond the life cycle of that run. And also, any subsequent runs should add to the vote tally. In other words, the data is shared between the nodes (JVMs) and persisted.

This is where Terracotta comes in. The data we're interested in sharing will be marked as "ROOT". In this case, it's the "votesRecord" field of the 'recorder' bean. This field will be transparently clustered by Terracotta, its value will be saved in Terracotta server and it will be faulted back into a new node as needed. There's no thirdparty function calls nor networking code, just plain Java. Here is what needed in Terracotta configuration file tc-config.xml:


  <application>
    <dso>
      <instrumented-classes>
        <include>
          <class-expression>org.terracotta.demo.*</class-expression>
        </include>
      </instrumented-classes>
      <roots>
        <root>
          <field-name>org.terracotta.demo.LocalVoteRecorder.votesRecord</field-name>
        </root>
      </roots>
    </dso>
  </application>

That piece of config tells Terracotta to instrument all classes under the org.terracotta.demo package and make "votesRecord" a root.

Since Terracotta has Maven plugin, I just use its plugin and run the example. The pom is here if you're interested. First, you need to start Terracotta server:

% mvn tc:start

Then you can run the example with Terracotta enable simply by:

% mvn tc:run

On first run, I got this result: Nick got voted twice


[INFO] [vote] INFO  [main] voted for candidate Nick
[INFO] [vote] INFO  [main] voted for candidate Hung
[INFO] [vote] INFO  [main] voted for candidate Hung
[INFO] [vote] INFO  [main] voted for candidate Nick
[INFO] [vote] INFO  [main] voted for candidate Hung
[INFO] [vote] INFO  [main] Candidate 'Hung' has 3 votes.
[INFO] [vote] INFO  [main] Candidate 'Nick' has 2 votes.
[INFO] Finished node vote
[INFO] DSO processes finished
[INFO] Skipping stopping DSO Server
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------

On second run: Nick got voted one more time brings his total votes to 3


[INFO] [vote] INFO  [main] voted for candidate Hung
[INFO] [vote] INFO  [main] voted for candidate Hung
[INFO] [vote] INFO  [main] voted for candidate Nick
[INFO] [vote] INFO  [main] voted for candidate Hung
[INFO] [vote] INFO  [main] voted for candidate Hung
[INFO] [vote] INFO  [main] Candidate 'Hung' has 7 votes.
[INFO] [vote] INFO  [main] Candidate 'Nick' has 3 votes.

This shows that the votes are indeed shared between 2 separate runs. To see the shared data live, you can use Terracotta Developer Console to look at the roots:

Run this Maven command to start the console:

% mvn tc:dev-console

You can check out the whole project from Subversion: http://svn.terracotta.org/svn/forge/projects/TerracottaSpringDemo

Create unique temp filename with Batch script

2009-04-14T14:45:00.001-07:00

For Windows batch scripts, you could use %RANDOM% to generate a random number and use that as part of the temp file name but that doesn't guarantee uniqueness since random number can be repeated. By adding another variable to the mix, seconds and fraction of seconds, the file name would be unique:



call :GETTEMPNAME
echo "Temp file name is %TMPFILE%"
goto :EOF

:GETTEMPNAME
set TMPFILE=%TMP%\mytempfile-%RANDOM%-%TIME:~6,5%.tmp
if exist "%TMPFILE%" GOTO :GETTEMPNAME 

:EOF

Using Ant as a library

2009-01-19T19:01:00.000-08:00

Ant has a lot of predefined tasks that could be a great help. You can use them directly from your Java code. Here are some examples using Ant to download a file, unzip a package and exec some shell command.

You would need to have
+ ant-1.7.1.jar
+ ant-launcher-1.7.1.jar
+ commons-io-1.3.2.jar (for exec example, found at http://commons.apache.org/io)

in your classpath.


public class AntDemo {
  
  /**
   * Download a file at sourceUrl to dest
   *
   */
  public static void download(URL sourceUrl, File dest) {
    Project dummyProject = new Project();
    dummyProject.init();
    
    Get antGet = new Get();
    antGet.setProject(dummyProject);
    antGet.setVerbose(true);
    antGet.setSrc(sourceUrl);
    antGet.setDest(dest);
    antGet.execute();
  }

  /**
   * Unzip a zip file
   *
   */
  public static void unzip(File src, File dest) {
    Project dummyProject = new Project();
    dummyProject.init();

    Expand antUnzip = new Expand();
    antUnzip.setProject(dummyProject);
    antUnzip.setSrc(src);
    antUnzip.setDest(dest);
    antUnzip.execute();

    /* ant doesn't preserve permission bits
       need to restore them manually */

    Chmod chmod = new Chmod();
    chmod.setProject(dummyProject);
    chmod.setDir(new File(src.getAbsolutePath().replaceAll(".zip", "")));
    chmod.setPerm("a+rx");
    chmod.setIncludes("**/**");
    chmod.execute();
  }

  
  /**
   * Run a shell command and return the output as String
   *  
   */
  public static String exec(String command, List params, File workDir) {
    File outputFile;
    try {
      outputFile = File.createTempFile("exec", ".out");
    } catch (IOException e) {
      throw new RuntimeException(e);
    }

    Project dummyProject = new Project();
    dummyProject.init();

    ExecTask execTask = new ExecTask();
    execTask.setProject(dummyProject);
    execTask.setOutput(outputFile);
    execTask.setDir(workDir != null ? workDir : new File(System
        .getProperty("user.dir")));
    execTask.setExecutable(command);
    if (params != null) {
      for (String param : params) {
        execTask.createArg().setValue(param);
      }
    }

    execTask.execute();

    FileReader reader = null;
    try {
      reader = new FileReader(outputFile);
      return IOUtils.toString(reader);
    } catch (Exception e) {
      throw new RuntimeException(e);
    } finally {
      IOUtils.closeQuietly(reader);
      outputFile.delete();
    }
  }

  public static void main(String[] args) {
    List params = Arrays.asList(new String[] { "Hello", "World" });
    System.out.println(exec("echo", params, null));
  }
}

Just note that you'll need a dummy project for the Ant task. Otherwise you'll get an NullPointerException.

There are a lot of tasks that you could use. The list is here http://ant.apache.org/manual/tasksoverview.html

Java Webstart Demos with Terracotta enabled

2009-01-04T23:57:00.001-08:00

For easy deployment, Terracotta enabled Java applications can also be launched via web start. There are certain issues to look out for but it's certainly possible. I wrote a small demo to demonstrate it.

Issue 1: Terracotta installation required on user's computer before running.

This is solved easily by downloading a Terracotta zip file and extract it to a known location on user computer. In my demo, I install it to $HOME/terracotta_ws.

Issue 2: Java Webstart applications don't allow modification of boot classpath.

Since you can't append or pre-append to boot classpath of the webstart VM, which Terracotta requires, you are forced to spawn an external JVM for your app. This is solvable by signing your jars with a keystore, that will allow your webstart app to spawn external process. This is a good blog showing how to sign your jars.

Note that only jars that are part of your main webstart GUI frame and its libraries are required to be signed. Jars that are part of your Terracotta applications are not required.

When I use "webstart app", I mean the control app that is downloaded and run by Java webstart.

Wehn I use "main app" or "TC enabled app", I mean the Terracotta enabled applications that are spawned in external processes by the "webstart app"

Issue 3: You can't declare your main application jars as part of resources to download in JNLP file.

The main reason is that Java Webstart obfuscates those jars and stores them in a cache with variable folder name. So your webstart app has to handle the downloading and installing the main app's binaries (zip file) to a known location. This is also doable. Take a look at the JNLP file I have:


<jnlp spec="1.0+" codebase="http://localhost:8080/webstartdemo"
 href="demo.jnlp">
 <information>
  <title>TC webstart demo</title>
  <vendor>Terracotta</vendor>
  <description>Demo of how to start Terracotta from webstart
  </description>
  <description kind="short">TC Webstart example</description>
  <offline-allowed />
 </information>
 <resources>
  <j2se version="1.5+" />
  <jar href="TCWebStartDemo.jar" download="eager" />
  <jar href="ant-1.7.1.jar" />
  <jar href="ant-launcher-1.7.1.jar" />
  <jar href="miglayout-3.6.2-swing.jar" />

  <property name="tcws.codebase" value="http://localhost:8080/webstartdemo" />
  <property name="tcws.terracotta.zip" value="terracotta-trunk-nightly-rev11190.zip" />
  <property name="tcws.demo.list" value="sharededitor,mandelbrot" />
  <property name="tcws.sharededitor.mainClass" value="demo.sharededitor.Main" />
  <property name="tcws.mandelbrot.mainClass" value="mandelbrot.Main" />

 </resources>
 <application-desc main-class="demo.MainFrame" />
 <security>
  <all-permissions />
 </security>
</jnlp>

From line 13 to 16, those jars are for the webstart app which will be downloaded automatically by java webstart. And those are the ones that need to be signed.
Line 25 indicates the main class of the webstart app.

From line 18 to 22, that is how I communicate to my webstart app (via system properies) about where to download Terracotta and the demos (Terracotta enabled apps). Each demo is a zip file which contains a tc-config.xml, a "lib" folder with all the jars it requires.

You can play with the demo here.

The webstart app Eclipse project can be checked out at SVN repo http://svn.terracotta.org/svn/forge/projects/labs/TCWebStartDemo

Your main interests should be the 2 classes:

DemoManager: download and install TC and demos

MainFrame: webstart app that spawns processes

Enjoy the demos :)

Redirecting System.out and System.err to JTextPane or JTextArea

2009-01-02T20:34:00.000-08:00

In Swing, if you want to redirect System.err and System.out to a JTextPane or a JTextArea, you only need to override the write() methods of OutputStream to append the text to the text pane instead.

The example below shown how to do it with JTextPane. For JTextArea, it's quite simpler, just call JTextArea.append()

This is useful when you spawn external process from a Swing application and want to see its output in your own text component.

Jan 04 2009: Revised regarding Eric Burke's comment below.
JTextPane version:


  private void updateTextPane(final String text) {
    SwingUtilities.invokeLater(new Runnable() {
      public void run() {
        Document doc = textPane.getDocument();
        try {
          doc.insertString(doc.getLength(), text, null);
        } catch (BadLocationException e) {
          throw new RuntimeException(e);
        }
        textPane.setCaretPosition(doc.getLength() - 1);
      }
    });
  }

  private void redirectSystemStreams() {
    OutputStream out = new OutputStream() {
      @Override
      public void write(final int b) throws IOException {
        updateTextPane(String.valueOf((char) b));
      }

      @Override
      public void write(byte[] b, int off, int len) throws IOException {
        updateTextPane(new String(b, off, len));
      }

      @Override
      public void write(byte[] b) throws IOException {
        write(b, 0, b.length);
      }
    };

    System.setOut(new PrintStream(out, true));
    System.setErr(new PrintStream(out, true));
  }

JTextArea version:


  private void updateTextArea(final String text) {
    SwingUtilities.invokeLater(new Runnable() {
      public void run() {
        textArea.append(text);
      }
    });
  }

  private void redirectSystemStreams() {
    OutputStream out = new OutputStream() {
      @Override
      public void write(int b) throws IOException {
        updateTextArea(String.valueOf((char) b));
      }

      @Override
      public void write(byte[] b, int off, int len) throws IOException {
        updateTextArea(new String(b, off, len));
      }

      @Override
      public void write(byte[] b) throws IOException {
        write(b, 0, b.length);
      }
    };

    System.setOut(new PrintStream(out, true));
    System.setErr(new PrintStream(out, true));
  }

[+] Old versions


  private void redirectSystemStreams() {
    OutputStream out = new OutputStream() {
      @Override
      public void write(int b) throws IOException {
        Document doc = textPane.getDocument();
        try {
          doc.insertString(doc.getLength(), String.valueOf((char) b), null);
        } catch (BadLocationException e) {
          throw new IOException(e);
        }
        textPane.setCaretPosition(doc.getLength() - 1);
      }

      @Override
      public void write(byte[] b, int off, int len) throws IOException {
        Document doc = textPane.getDocument();
        try {
          doc.insertString(doc.getLength(), new String(b, off, len), null);
        } catch (BadLocationException e) {
          throw new IOException(e);
        }
        textPane.setCaretPosition(doc.getLength() - 1);
      }

      @Override
      public void write(byte[] b) throws IOException {
        write(b, 0, b.length);
      }
    };

    System.setOut(new PrintStream(out, true));
    System.setErr(new PrintStream(out, true));
  }

JTextArea version:


  private void redirectSystemStreams() {
    OutputStream out = new OutputStream() {
      @Override
      public void write(int b) throws IOException {
        textArea.append(String.valueOf((char) b));
      }

      @Override
      public void write(byte[] b, int off, int len) throws IOException {
        textArea.append(new String(b, off, len));
      }

      @Override
      public void write(byte[] b) throws IOException {
        write(b, 0, b.length);
      }
    };

    System.setOut(new PrintStream(out, true));
    System.setErr(new PrintStream(out, true));
  }

Link checker with Java - better version

2008-12-18T14:56:00.000-08:00

I posted an old blog regarding checking for broken links with Java. However, that version uses 3rd party library httpunit and doesn't handle forwarding case. If a url forwards to another url, http response code would be 302 and the old version would consider that link is broken.

The new version would follow the forwarding url and check it. If the forwarded url is dead, it would be detected.

So here is the new code, without any extra 3rd party API:


  private static boolean isLive(String link) {
    HttpURLConnection urlConnection = null;
    try {
      URL url = new URL(link);
      urlConnection = (HttpURLConnection) url.openConnection();
      urlConnection.setRequestMethod("HEAD");
      urlConnection.setConnectTimeout(5000); /* timeout after 5s if can't connect */
      urlConnection.setReadTimeout(5000); /* timeout after 5s if the page is too slow */
      urlConnection.connect();
      String redirectLink = urlConnection.getHeaderField("Location");
      if (redirectLink != null && !link.equals(redirectLink)) {
        return isLive(redirectLink);
      } else {
        return urlConnection.getResponseCode() == HttpURLConnection.HTTP_OK;
      }
    } catch (Exception e) {
      return false;
    } finally {
      if (urlConnection != null) {
        urlConnection.disconnect();
      }
    }
  }
  
  public static void main(String[] args) {
    System.out.println(isLive("http://google.com"));
    System.out.println(isLive("http://somefakelink.com"));
  }

Simple demo of clustering Ehcache

2008-12-18T00:49:00.000-08:00

To cluster a cache between 2 or more JVM's, it's quite simple if you use Terracotta underneath. It will handle the networking, the shared memory, transparently for you. There's no extra API but a configuration file.

Terracotta also has maven plugin support so you could grab the demo project below and run without downloading anything.

The demo is only one Java source file. It is meant to be run in 2 VM's.

I used a cyclic barrier to synchronize between the 2 VM's. It will block until both VM hit the "await()" call. The first node will create a cache and populate it with 2 elements. The second node then queries out of the cache to demonstrate that it could get to the shared data.

package demo;

import java.util.concurrent.CyclicBarrier;

import net.sf.ehcache.Cache;
import net.sf.ehcache.CacheManager;
import net.sf.ehcache.Element;
import net.sf.ehcache.store.MemoryStoreEvictionPolicy;

public class EhcacheDemo {

  private CacheManager  manager = CacheManager.create();
  private CyclicBarrier barrier = new CyclicBarrier(2);

  public void run() throws Exception {
    Cache testCache;

    System.out.println("Waiting for the other node...");
    if (barrier.await() == 0) {
      System.out.println("Creating the testCache...");
      testCache = new Cache("test", 100, MemoryStoreEvictionPolicy.LFU, true,
          null, false, 60, 30, false, 0, null);
      manager.addCache(testCache);
    }

    barrier.await();
    testCache = manager.getCache("test");

    if (barrier.await() == 0) {
      System.out.println("First node: adding 2 elements...");
      testCache.put(new Element("k1", "v1"));
      testCache.put(new Element("k2", "v2"));
    } else {
      System.out.println("Second node: querying 3 elements...");
      System.out.println(testCache.get("k1"));
      System.out.println(testCache.get("k2"));
      System.out.println(testCache.get("k3"));
    }

    System.out.println("Done!");
  }

  public static void main(String[] args) throws Exception {
    new EhcacheDemo().run();
  }

}

Note that second node is also querying for non-existed element "k3" as a sanity check.

So how does Terracotta know what I want to share? From the above code, I tell Terracotta to share the cache "manager" and "barrier" by marking them as root in Terracotta configuration file tc-config.xml


  <application>
    <dso>
      <instrumented-classes/>
      <roots>
        <root>
          <field-name>demo.EhcacheDemo.manager</field-name>
        </root>
        <root>
          <field-name>demo.EhcacheDemo.barrier</field-name>
        </root>
      </roots>
    </dso>
  </application>

After you download the project, just run this Maven commands:


mvn install
mvn tc:run

The output would be something like this:


[INFO] [node1] Waiting for the other node...
[INFO] [node0] Waiting for the other node...
[INFO] [node0] Creating the testCache...
[INFO] [node1] First node: adding 2 elements...
[INFO] [node0] Second node: querying 3 elements...
[INFO] [node1] Done!
[INFO] [node0] [ key = k1, value=v1, version=1, hitCount=2, CreationTime = 1229592169906, LastAccessTime = 1229592170081 ]
[INFO] [node0] [ key = k2, value=v2, version=1, hitCount=2, CreationTime = 1229592170070, LastAccessTime = 1229592170145 ]
[INFO] [node0] null
[INFO] [node0] Done!

Compile and test with different JDK with a Maven project

2008-09-21T22:38:00.001-07:00

If you ever have the need to compile and run tests of your Maven project with different JDK, here are some tips:

1. The cheapest way

Maven by default use your JAVA_HOME to compile the project and then run the tests.
So if want to run your tests with different jdk, just set your JAVA_HOME to the jdk of your desire. Here's a neat treat in Bash to set an env variable locally to the current process.

Assume you have environment variables JAVA_HOME_15 and JAVA_HOME_16 pointing to their respective jdk1.5 and jdk1.6, you can then do this:


%> JAVA_HOME=$JAVA_HOME_15 mvn install   # run with 1.5
%> JAVA_HOME=$JAVA_HOME_16 mvn install   # run with 1.6

2. The Maven way

(hairy verbose and messy pom.xml profiles)

This only works with Maven-2.1.0-M1. Failed with 2.0.9 for me

You have to specify which JDK you want for 2 of these plugins: maven-compiler-plugin and maven-surefire-plugin.


  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
          <verbose>true</verbose>
          <fork>true</fork>
          <executable>${jdk}/bin/javac</executable>
          <compilerVersion>1.5</compilerVersion>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <jvm>${jdk}/bin/java</jvm>
          <forkMode>once</forkMode>
        </configuration>
      </plugin>
    </plugins>
  </build>

So you see I use the property ${jdk} Now we just need to define a profile for each of the JDK we want and set the ${jdk} appropriately.


  <profiles>
      <profile>
      <id>default_jdk</id>
      <activation>
          <activeByDefault>true</activeByDefault>
      </activation>
      <properties>
          <jdk>${env.JAVA_HOME}</jdk>
      </properties>
    </profile>
    <profile>
      <id>jdk15</id>
      <activation>
          <activeByDefault>false</activeByDefault>
      </activation>
      <properties>
          <jdk>${env.JAVA_HOME_15}</jdk>
      </properties>
    </profile>
    <profile>
      <id>jdk16</id>
      <activation>
          <activeByDefault>false</activeByDefault>
      </activation>
      <properties>
          <jdk>${env.JAVA_HOME_16}</jdk>
      </properties>
    </profile>    
  </profiles>

Above I defined 3 profiles. The first one is the default one. It's a trick to set ${jdk} to $JAVA_HOME so that Maven knows what to use if you don't specify any profile to run.

Here how you would activate each profile:


%> mvn install          # no profile so by default run tests with JAVA_HOME
%> mvn install -Pjdk15  # run with jdk1.5
%> mvn install -Pjdk16  # run with jdk1.6

Note that in the profile I use env variables JAVA_HOME_15 and JAVA_HOME_16 also. You can of course fill in an absolute path to your JDK.

So there you have it. Took me awhile to figure out.

Bug bite

2008-03-26T15:56:00.000-07:00

It's a very subtle bug but results in great headache. I didn't notice my bug until someone pointed it out. Good lesson learned: Always have tests when you refactor.

See if you could spot it.


// BEFORE
private LockID[] getAllLockIDs(LockID id) {
  LockID[] lids = new LockID[transactionStack.size() + 1];
  lids[0] = id;
  for (int i = 1; i < lids.length; i++) {
    TransactionContext tc = (TransactionContext) transactionStack.get(i - 1);
    lids[i] = tc.getLockID();
  }
  return lids;
}

   
// AFTER
private List getAllLockIDs(LockID id) {
  List lids = new ArrayList();
  lids.add(id);
  for (int i = 1; i < transactionStack.size(); i++) {
    TransactionContext tc = (TransactionContext) transactionStack.get(i - 1);
    lids.add(tc.getLockID());
  }
  return lids;
}

Pod Racing: How to synchronize threads across multiple JVMs

2008-02-07T09:56:00.001-08:00

Sometimes, you want to start the job in multiple JVMs only when all the VM has started. This scenario often comes up with multiplayer games. Let's take a pod racing game for example. The players need to start at the same time and each game server handles 1 player (for simplicity's sake)

Starting from Java 5, there's a CyclicBarrier to synchronize threads and is a perfect tool for this. However, it only handles threads in 1 VM. D'oh! Here's where Terracotta comes in. If you make that CyclicBarrier a shared object, Terracotta will handle the clustering for you, transparently (no API). All of a sudden, your CyclicBarrier object will be seen across all the VMs participated in the game.

Here's an example of how to do it. First, let's take a look at the PodRacer class

package demo;

import java.util.concurrent.CyclicBarrier;

public class PodRacer {
  public final static int     COUNT   = 2;
  private final CyclicBarrier barrier = new CyclicBarrier(COUNT);
  private String              name;

  public PodRacer(String name) {
    this.name = name;
  }

  public void ready() {
    System.out.println(name + ": ready");
  }

  public void set() throws Exception {
    /* in Terracotta world, all threads in different VMs will block here */
    barrier.await();
  }

  public void go() throws Exception {
    System.out.println(name + ": go");
    Thread.sleep((int) (Math.random() * 5000) + 10);
    System.out.println(name + ": arrived at " + System.currentTimeMillis());
  }

  public static void main(String[] args) throws Exception {
    PodRacer racer = new PodRacer(System.getProperty("racer.name", "unknown"));
    racer.ready();
    racer.set();
    racer.go();
  }
}

Notice I hardcoded number of racers here to 2 but it can be made dynamic. If you run this class just like a normal Java program, it will block at racer.set() call because there's only 1 thread arriving at the barrier whereas it requires 2 for the barrier to be lifted. And if you don't use Terracotta underneath, it doesn't matter how many VMs you start, it will just block there.

Next, I'll add Terracotta into the mix and share the barrier object. That can be achieved by marking it in Terracotta configuration file tc-config.xml


<?xml version="1.0" encoding="UTF-8"?>
<con:tc-config xmlns:con="http://www.terracotta.org/config">
  <servers>
    <server host="%i" name="localhost">
      <dso-port>9510</dso-port>
      <jmx-port>9520</jmx-port>
      <data>terracotta/server-data</data>
      <logs>terracotta/server-logs</logs>
    </server>
  </servers>
  <clients>
    <logs>terracotta/client-logs</logs>
  </clients>
  <application>
    <dso>
      <instrumented-classes/>
      <roots>
        <root>
          <field-name>demo.PodRacer.barrier</field-name>
        </root>
      </roots>
    </dso>
  </application>
</con:tc-config>

And that's it. When I start 2 JVMs running PodRacer class with Terracotta, I will efficiently have 2 threads calling racer.set() on the shared barrier

To make it even easier to try out with Terracotta, there's a Maven 2 plugin that will handle starting up your project with Terracotta enabled. Here's the pom.xml of my project:


<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>PodRacing</groupId>
  <artifactId>PodRacing</artifactId>
  <version>0.0.1</version>
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
          <source>1.5</source>
          <target>1.5</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.terracotta.maven.plugins</groupId>
        <artifactId>tc-maven-plugin</artifactId>
        <version>1.0.3</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>bootjar</goal>
            </goals>
          </execution>
        </executions>
        
        <configuration>
          <processes>
            <process nodeName="racer1" 
                     className="demo.PodRacer"
                     jvmargs="-Dracer.name=Anakin"/>
            <process nodeName="racer2" 
                     className="demo.PodRacer"
                     jvmargs="-Dracer.name=Sebulba"/>
          </processes>
          
        </configuration>
      </plugin>
    </plugins>
  </build>
  <pluginRepositories>
    <pluginRepository>
      <releases />
      <snapshots />
      <id>terracotta</id>
      <url>http://download.terracotta.org/maven2</url>
    </pluginRepository>
  </pluginRepositories>
</project>

See how I defined 2 racers through the process element in the plugin configuration. That will start 2 distinct VMs with Terracotta enabled. The output looks something like this:


[INFO] Starting DSO nodes 
[INFO] Starting node racer1: c:\jdk\jdk1.6.0_02\jre/bin/java.exe -Dcom.tc.l1.modules.repositories=file:/C:/Users/hhuynh/.m2/repository/ -Dtc.nodeName=racer1 -Dtc.numberOfNodes=2 -Dtc.config=d:\work\workspace\projects\PodRacing\tc-config.xml -Dtc.classpath=file:/c:/Users/hhuynh/AppData/Local/Temp/tc-classpath37736.tmp -Dtc.session.classpath=/C:/Users/hhuynh/.m2/repository/org/terracotta/tc-session/2.5.0/tc-session-2.5.0.jar -Dcom.tc.l1.modules.repositories=file:/C:/Users/hhuynh/.m2/repository/ -Xbootclasspath/p:d:\work\workspace\projects\PodRacing\target\dso-boot.jar -Dracer.name=Anakin -cp d:\work\workspace\projects\PodRacing\target\classes; demo.PodRacer
[INFO] Starting node racer2: c:\jdk\jdk1.6.0_02\jre/bin/java.exe -Dcom.tc.l1.modules.repositories=file:/C:/Users/hhuynh/.m2/repository/ -Dtc.nodeName=racer2 -Dtc.numberOfNodes=2 -Dtc.config=d:\work\workspace\projects\PodRacing\tc-config.xml -Dtc.classpath=file:/c:/Users/hhuynh/AppData/Local/Temp/tc-classpath37737.tmp -Dtc.session.classpath=/C:/Users/hhuynh/.m2/repository/org/terracotta/tc-session/2.5.0/tc-session-2.5.0.jar -Dcom.tc.l1.modules.repositories=file:/C:/Users/hhuynh/.m2/repository/ -Xbootclasspath/p:d:\work\workspace\projects\PodRacing\target\dso-boot.jar -Dracer.name=Sebulba -cp d:\work\workspace\projects\PodRacing\target\classes; demo.PodRacer
[INFO] ------------------------------------------------------------------------

[INFO] [racer2] Sebulba: ready
[INFO] [racer1] Anakin: ready
[INFO] [racer1] Anakin: go
[INFO] [racer2] Sebulba: go
[INFO] [racer2] Sebulba: arrived at 1202406914274
[INFO] Finished node racer2
[INFO] [racer1] Anakin: arrived at 1202406915125
[INFO] Finished node racer1

Ouch, Anakin sucked!

Download the Maven project and try it out. Have fun.

P.S. More about sharing memory between JVMs here http://unserializableone.blogspot.com/2006/10/share-precious-heap-memory-accross.html or visit Terracotta

Monkey tales: Continuous Integration at Terracotta

2007-11-27T07:59:00.000-08:00

As a platform to cluster and scale Java applications, Terracotta usages are greatly varied and resourceful. Further more, Terracotta supports Linux, Windows, and Solaris (also OSX for dev), compiles and runs with Sun JDKs (1.4, 1.5, 1.6) and IBM. As for application servers, we support Tomcat, JBoss, Jetty, Websphere, Weblogic, etc. The list goes on and on. That puts a lot of burden on testings and build infrastructure and we can't think of a better way than continuous integration.

CruiseControl (CC) comes in handy and works really well for us. It basically checks out new changesets from the repo periodically then compile and run all tests. Our custom built system, tcbuild, is JRuby + Ant combo. It has the ability to run defined group of tests such as system tests, unit tests, container tests, crash tests, etc.

We create a CC project for each group of tests, namely check_unit, check_system, check_container, etc that varies in jdk, appserver, test mode. There will be 57 projects with all the permutations.

And each CC project will do:
- check out latest revisions
- compile
- run tests in that project
- report
....

And we do this on multiple boxes that run Redhat, Suse 10, Windows 2003, Solaris 9, 10. We call these monkey machines.

So what happens when there's a bad build (compiling error), or tests fail? CC will email a specific team depends on which group of tests that failed. There's also a mailing list that will keep track of every failure.

This works great but there's a drawback. Whenever there's a compile error, we got spammed and we got it bad. It will fail on all 57 CC runs on each monkey box.

To solve this problem, we devised a two-tier monkey scheme. We would have a monkey box, called "monkey police" that would check out the latest revision(s) every 5 minute. It will compile and run a group crucial system tests. If the build fails to compile or any of the test fails, that means that build is sorely broken. In this case, the police will email the last person(s) and the engineer mailing list about the error. If everything works out fine, the police will mark the latest revision as good and save it onto a shared file. The monkey troops (other test machines) will read from that shared file and only do "svn update" up to the latest known good revision before they run their tests.

So before, the troops will do "svn update" and it will pull down every changes. With this scheme, the troops will do "svn update -r 2333" where "2333" is a good revision that the monkey police has tested.

And we have one monkey police per branch. So when there's build error or fatal test failure, we'll know about it right away and it won't disturb our test machines in the mean time.

Why bother with a bad checkin? :)

This has worked great for us and I thought I'd share the experience.

Solution to classpath too long (aka input line too long) problem in Windows

2007-10-15T10:59:00.000-07:00

If you use Java in Windows, you're bound to run into classpath too long problem when your classpath grows. Windows has a limit (1KB - 2KB) of characters you can have on one single command line. The infamous "Input line is too long" is very annoying. Here's a trick to get around it.

"java.exe" command also scan for classes from the environment variable "CLASSPATH". If you can break your classpath into separate folders and jars, you can concatenate them like this:


setlocal

set CLASSPATH=c:/my/first/jar/somejar.jar;%CLASSPATH%
set CLASSPATH=c:/my/second/jar/someotherjar.jar;%CLASSPATH%
set CLASSPATH=c:/path/to/a/folder;%CLASSPATH%
.......
.......

java com.mycompany.Main

endlocal

By using CLASSPATH env variable, you don't need to pass it in to "java" command. The setlocal/endlocal pair ensures the CLASSPATH is "local" to this process and won't pollute the systemwise value.

--------
Updated:

Travis has pointed out that there is a limit on environment variable in Windows. I found it limited to 8K. So this isn't an absolute solution but it should sustain you for awhile. As suggested by D, you could shorten the path by using virtual drive. Further more, JDK6 has supported wildcard (*) in classpath.


Virtual drive:
Say you have bunch of jars under c:/path/to/lib

subst z: c:/my/path/to/lib
set CLASSPATH=z:/jar1.jar:%CLASSPATH%

Or with JDK6:

set CLASSPATH=c:/path/to/lib/*:%CLASSPATH%

Check for broken links in Ruby, Bash script and Java

2007-10-08T19:38:00.000-07:00

I found myself writing 3 different versions of a function to check for broken links in Ruby, Bash and Java. Just want to document here in case anyone interested. Note that these functions use HEAD method (as opposed to GET/POST). It won't download a big file just to see if it's live.

Ruby

require 'net/http'
require 'uri'

def isLive?(url)
  uri = URI.parse(url)
  response = nil
  Net::HTTP.start(uri.host, uri.port) { |http|
    response = http.head(uri.path.size > 0 ? uri.path : "/")
  }  
  return response.code == "200"
end

puts isLive?("http://google.com")
puts isLive?("http://asdfasdf.com")

Bash

#!/bin/bash

function isLive {
  wget -q --spider $1
}

isLive "http://google.com/somefakelink"

if [ $? -eq 0 ]; then
  echo "Good link"
else 
  echo "Broken link"
fi

Java: Edited (December 18 2008): I've written a better Java version that handles link forwarding and doesn't use 3rd party API here


/* need httpunit-1.6.jar http://httpunit.sourceforge.net */

private boolean isLive(String link) {
  try {
    WebRequest request = new HeadMethodWebRequest(link);
    WebConversation wc = new WebConversation();
    WebResponse response = wc.getResource(request);
    return response.getResponseCode() == 200
  } catch (Exception e)
    return false;
  }
}

Toggle code portion in your blog

2007-09-18T00:47:00.001-07:00

I'm just testing out my javascript to toggle code sections in my blog. It's a useful feature to help keep your blog appears short while still provides enough details.

Show code

The javascript I use:

function toggle(id) {
  var e = document.getElementById(id);
  e.style.display = e.style.display == "none" ? "block" : "none";
}

Then create an anchor to control the toggling:
<a href="javascript:void(0)" onclick="toggle('thecode')">Show code</a>
<div id="thecode" style="display:none"> my code goes here <div>

Hope this works :)

Distributed EhCache as second level cache under Hibernate

2007-09-17T18:05:00.000-07:00

EhCache is a one of the great options for Hibernate second level cache. By making it distributed, multiple web applications will be able to share the same cache thus enhance your overall performance and availability. To enable the distributed cache, Terracotta 2.4.3 has a built in support for EhCache 1.3.0 and 1.2.4. I will go through an example of how this be done.

The stack could be visualized like this:


-----------        -----------
Tomcat 1            Tomcat 2
-----------        -----------
Hibernate           Hibernate
------------------------------
        EhCache
       TERRACOTTA
------------------------------

Terracotta is the driving force though its presence is transparent to your web app thanks to bytecode instrumentation. First, let take a look at enabling EhCache in Hibernate configuration file
[+]hibernate.cfg.xml

... database setting ...
<property name="cache.use_query_cache">true</property>
<property name="cache.provider_configuration_file_resource_path">ehcache.xml</property>
<property name="cache.use_second_level_cache">true</property>
<property name="cache.provider_class">org.hibernate.cache.EhCacheProvider</property>

<mapping resource="Event.hbm.xml" />
<mapping resource="Person.hbm.xml" />

It's just the same standard setting that you would use for non-distributed case. You want to turn on query cache mode, point Hibernate to ehcache.xml and finally specify a provider. The ehcache.xml can be as simple as this:
[+]ehcache.xml

<ehcache>
  <diskStore path="user.dir"/>
  <defaultCache
      maxElementsInMemory="10000"
      eternal="false"
      overflowToDisk="false"
      timeToIdleSeconds="300"
      timeToLiveSeconds="300"
      diskPersistent="false"
      diskExpiryThreadIntervalSeconds="120"
      memoryStoreEvictionPolicy="LRU"/>
  
  <cache name="org.hibernate.cache.StandardQueryCache"
      maxElementsInMemory="100"
      eternal="false"
      timeToIdleSeconds="120"
      timeToLiveSeconds="120"
      overflowToDisk="false"/>
  
  <cache name="org.hibernate.cache.UpdateTimestampsCache"
      maxElementsInMemory="5000"
      timeToIdleSeconds="120"
      timeToLiveSeconds="120"
      eternal="true"/>
</ehcache>

One thing I'd like to point out is that Terracotta persists heap memory to disk efficiently (and fault them in as needed) so "overflowToDisk" becomes redundant in StandardQueryCache. As a matter of fact, Terracotta doesn't honor this option. Now that the cache is set up, in our entity mapping files, we need to let Hibernate know which entities we'd like to cache during runtime. In my example, I have an Event table (title, date) and a Person table (firstname, lastname) that have a many-to-many relationship through a join table PERSON_EVENT. With that in mind, let's examine Event.hbm.xml and Person.hbm.xml:
[+]Mapping files


Event.hbm.xml:
<class name="events.Event" table="EVENTS">
<cache usage="read-write" />
<id name="id" column="EVENT_ID">
  <generator class="native" />
</id>
<property name="date" column="EVENT_DATE" />
<property name="title" />

<set name="participants" table="PERSON_EVENT" lazy="true"
  inverse="true" cascade="lock">
  <cache usage="read-write" />
  <key column="EVENT_ID" />
  <many-to-many column="PERSON_ID"
    class="events.Person" />
</set>
</class>

Person.hbm.xml:
<class name="events.Person" table="PERSON">
   <cache usage="read-write" />
   <id name="id" column="PERSON_ID">
       <generator class="native"/>
   </id>

   <property name="firstname"/>
   <property name="lastname"/>
  
   <set name="events" table="PERSON_EVENT">
       <cache usage="read-write" />
       <key column="PERSON_ID"/>
       <many-to-many column="EVENT_ID" class="events.Event"/>
   </set> 

</class>

Event.java:
public class Event {
 private Long   id;
 private String title;
 private String date;
 private Set    participants = new HashSet();

 /* getters and setters */
}

Person.java:
public class Person {
 private Long   id;
 private String firstname;
 private String lastname;
 private Set    events = new HashSet();

 /* getters and setters */
}

The details might be distracting but if you're familiar with Hibernate, this should be as simple as it gets :)

Now we get past all the settings, the fun stuff begins with our servlets. I've created 2 servlets, one called CreateEvents that will populate data into our table. The other, QueryEvents, will query and display the cache hit statistic.
[+]CreateEvents.java


public class CreateEvents extends HttpServlet {

 /* (1) */
 public void init() throws ServletException {
   super.init();
   generateData();
 }

 protected void doGet(HttpServletRequest req, HttpServletResponse resp)
     throws ServletException, IOException {
   /* (2) */
   if (req.getParameter("fn") != null && req.getParameter("ln") != null) {
     EventManager.createAndStorePerson(req.getParameter("fn"), req
         .getParameter("ln"));
   }

   resp.setContentType("text/html");
   resp.getWriter().println("<html><body>");
  
   /* (3) */
   Statistics stats = HibernateUtil.getSessionFactory().getStatistics();
   stats.setStatisticsEnabled(true);
   QueryStatistics queryStats = stats.getQueryStatistics("from Event");
  
   /* (4) */
   resp.getWriter().println(
       "Events created: " + EventManager.query("Event") + "<br/>");
   resp.getWriter().println(
       "Event query cache miss: " + queryStats.getCacheMissCount() + "<br/>");
   resp.getWriter().println(
       "Event query cache hit: " + queryStats.getCacheHitCount() + "<br/>");
   resp.getWriter().println("</body></html>");
 }

 private void generateData() {
   HibernateUtil.dropAndCreateDatabaseSchema();

   /* people */
   Long p1 = EventManager.createAndStorePerson("Ichigo", "Kurosaki");
   Long p2 = EventManager.createAndStorePerson("Abarai", "Renji");
   Long p3 = EventManager.createAndStorePerson("Ishida", "Uryu");

   /* events */
   Long e1 = EventManager.createAndStoreEvent("Event 1", "2007-09-30");
   Long e2 = EventManager.createAndStoreEvent("Event 2", "2007-12-01");

   /* participants in e1 */
   EventManager.addPersonToEvent(p1, e1);
   EventManager.addPersonToEvent(p2, e1);
   EventManager.addPersonToEvent(p3, e1);

   /* participants in e2 */
   EventManager.addPersonToEvent(p2, e2);
   EventManager.addPersonToEvent(p3, e2);
 }

}

Some default data (3 persons and 2 events) are created during init() phase. With (2), I added an option to add additional Person to the database so later we can use it to demonstrate cache invalidating. To be able to get statistic of cache hit and miss, (3), a query statistic object is created. It will give us the hit/miss count in (4), for the query "select * from Event".
With QueryEvents.java, we will ask Hibernate for list of persons, and events, which would prove to us whether the cache is used or database is used:
[+]QueryEvents.java


public class QueryEvents extends HttpServlet {

 protected void doGet(HttpServletRequest req, HttpServletResponse resp)
     throws ServletException, IOException {

   resp.setContentType("text/html");
   resp.getWriter().println("<html><body>");

   Statistics stats = HibernateUtil.getSessionFactory().getStatistics();
   stats.setStatisticsEnabled(true);
   QueryStatistics queryStats = stats.getQueryStatistics("from Event");

   /* (1) query Event */
   List events = EventManager.query("Event");
   resp.getWriter().println(
       "Events found: " +  events + "<br/>");
   resp.getWriter().println(
       "Event query cache miss: " + queryStats.getCacheMissCount() + "<br/>");
   resp.getWriter().println(
       "Event query cache hit: " + queryStats.getCacheHitCount() + "<br/>");

   /* (2) query People */
   queryStats = stats.getQueryStatistics("from Person");
   resp.getWriter().println("<br/>" +
       "People found: " + EventManager.query("Person") + "<br/>");
   resp.getWriter().println(
       "Person query cache miss: " + queryStats.getCacheMissCount() + "<br/>");
   resp.getWriter().println(
       "Person query cache hit: " + queryStats.getCacheHitCount() + "<br/><br/>");

   /* (3) reassociate transient Event objects to this hibernate session */
   /* so we can query for participants */
   Session session = HibernateUtil.getSessionFactory().getCurrentSession();
   session.beginTransaction();
   for (Iterator it = events.iterator(); it.hasNext();) {
     session.lock((Event) it.next(), LockMode.NONE);
   }
  
   resp.getWriter().println("Participants of Even 1: " + ((Event)events.get(0)).getParticipants() + "<br/>");
   resp.getWriter().println("Participants of Even 2: " + ((Event)events.get(1)).getParticipants() + "<br/>");
  
   session.close();   
   resp.getWriter().println("</body></html>");
 }

}

I ran these two servlets in 1 Tomcat to make sure everything working correctly:
By hitting http://localhost:8080/Events/create (mapped to CreateEvents servlet)


Events created: [Event 1: 2007-09-30, Event 2: 2007-12-01]
Event query cache miss: 1
Event query cache hit: 0

Since it's the first time we query for events, that's why we have one cache miss. Hibernate went to db to get the data.
Now we hit http://localhost:8080/Events/query (mapped to QueryEvents servlet), the result is:


Events found: [Event 1: 2007-09-30, Event 2: 2007-12-01]
Event query cache miss: 1
Event query cache hit: 1

People found: [Ichigo Kurosaki, Abarai Renji, Ishida Uryu]
Person query cache miss: 1
Person query cache hit: 0

Participants of Even 1: [Abarai Renji, Ishida Uryu, Ichigo Kurosaki]
Participants of Even 2: [Abarai Renji, Ishida Uryu]

As expected, the Event query is now hit the cache, raising the hit count to 1. And since it's the first time we query for Person, the cache miss is 1. The participants list proves that the event pojos coming from the cache are valid and can be re-associated to this Hibernate session. Now, if we run CreateEvents servlet on Tomcat 1 and QueryEvents on Tomcat 2, the distributed cache should give us the same result.
This is where Terracotta comes in. There is no change in settings needed in any of the Hibernate mappings files, nor ehcache.xml. There is no code change either. What you need to do is to run Tomcat with Terracotta enabled. The process involved setting 3 java system properties to Tomcat jvm


-Xbootclasspath/p:"path/to/Terracotta/bootjar"
-Dtc.install-root=/path/to/Terracotta/install
-Dtc.config=/path/to/tc-config.xml

Detailed instructions can be found here
Luckily, Terracotta has a nice Session configuration tool that will help you set up 2 Tomcats (or Weblogic) cluster. All you need is to import your WAR file. I created a Evetns.war file that contains both of my serlvets and all the needed jars. I need to configure tc-config.xml to let Terracotta knows that I'm using Hibernate, EhCache by adding those modules (1). Also, classes that will be shared need to be instrumented (2). Terracotta also supports sharing of session by declaring your webapp name (3). However, I'm not clustering any session in this example.
[+]tc-config.xml


<?xml version="1.0" encoding="UTF-8"?>
<con:tc-config xmlns:con="http://www.terracotta.org/config">
 <servers>
   <server host="%i" name="localhost">
     <dso-port>9510</dso-port>
     <jmx-port>9520</jmx-port>
     <data>terracotta/server-data</data>
     <logs>terracotta/server-logs</logs>
   </server>
 </servers>
 <clients>
   <logs>terracotta/client-logs</logs>
   ( 1 )
   <modules>
 <module name="clustered-hibernate-3.1.2" version="1.0.0" />
 <module name="clustered-ehcache-1.3" version="1.0.0" />
   </modules>
 </clients>
 <application>
   <dso>
     ( 2 )
     <instrumented-classes>
       <include>
         <class-expression>events.Event</class-expression>
       </include>
       <include>
         <class-expression>events.Person</class-expression>
       </include>
       <include>
         <class-expression>events.EventManager</class-expression>
       </include>       
     </instrumented-classes>
     ( 3 )
     <web-applications>
       <web-application>Events</web-application>
     </web-applications>
   </dso>
 </application>
 <system>
   <configuration-model>development</configuration-model>
 </system>
</con:tc-config>

The session configurator will start up Terracotta server and 2 Tomcats. We can now access CreateEvents servlet on the first Tomcat at port 9081 by hitting http://localhost:9081/Events/create and hit QueryEvents on the second Tomcat at http://localhost:9082/Events/query.

The result I got is:

 Events/create:
Events created: [Event 1: 2007-09-30, Event 2: 2007-12-01]
Event query cache miss: 1
Event query cache hit: 0

Events/query:
Events found: [Event 1: 2007-09-30, Event 2: 2007-12-01]
Event query cache miss: 0
Event query cache hit: 1

People found: [Ichigo Kurosaki, Abarai Renji, Ishida Uryu]
Person query cache miss: 1
Person query cache hit: 0

Participants of Even 1: [Abarai Renji, Ishida Uryu, Ichigo Kurosaki]
Participants of Even 2: [Abarai Renji, Ishida Uryu]

If I reload Events/query, the statistic is as expected:


Event query cache miss: 0
Event query cache hit: 2

Person query cache miss: 1
Person query cache hit: 1

To test that the cache is invalidated, after hitting Events/query, in the cache we now have a list of 3 persons. If we create one new person, by hitting http://localhost:9081/Events/create?fn=John&ln=Smith, what we have in the cache now is stale data. Of course, thanks to Terracotta, the second Tomcat + Hibernate is aware of this situation and stale data will be invalidated. Which leads to a cache miss (instead of a hit) when we reload http://localhost:9082/Events/query


Event query cache miss: 0
Event query cache hit: 3

People found: [Ichigo Kurosaki, Abarai Renji, Ishida Uryu, John Smith]
Person query cache miss: 2
Person query cache hit: 1

As you can see, Event query cache hit continues to rise, when we now have a cache miss in Person query since the cached data is made invalid. Hibernate had to hit the database for new fresh data.

I hope I didn't bore you with too much details but I think it's important to each steps. Terracotta is greatly beneficial if you choose to use EhCache as distributed cache with Hibernate.

You can download the project here and give it a try.

Simple directory browser for Amazon S3

2007-09-06T13:12:00.000-07:00

If you have used Amazon S3, you might be annoyed at the lack of support for directory browsing. If you hit your repo url, all you get is an XML file listing your content. So I wrote this small javascript to read that XML file, then display as a directory tree just like you're browsing an FTP site.

For example, say Terracotta has a S3 repo at http://download.terracotta.org. If you click on that link, you'll get an XML file. Now I want to be able to see what we have under http://download.terracotta.org/maven2, you'll get nothing but an error trying to go there.

My javascript is under the page http://download.terracotta.org/maven2/index.html

Just right click on that page and read the source, the javascript is pretty simple. Could have been made better or fancier but I'm no javascript guru :)

Maybe it could be of use for you.

Note: S3 doesn't serve index.html file automatically (after all, it's not a web server) so you have to hit the index.html explicitly.

Lightweight Google Geocoder with Java

2007-08-26T21:25:00.000-07:00

I often look for tools before deciding to build them myself for obvious reasons: saving time and labor. I was looking for a Java implementation of Google geocoder but there seems to be only this one GeoGoogle. My first impression is that the tool is a little heavy for its job. Further more, it chooses to deal with XML format and all the parsing and schema validation is a bit too much for my taste. To use Google geocode service, all you need is to send an HTTP request along with your address. The response will be in either XML or JSON format. It will include details about the address you sent such as city, state, zipcode, etc, but the most important things are longitude and latitude. So I decided to go with JSON and wrote this tool. Hopefully, someone else might find it useful. Many thanks to the folks that wrote JSON-LIB.


public class GGcoder {
  private static final String URL         = "http://maps.google.com/maps/geo?output=json";
  private static final String DEFAULT_KEY = "YOUR_GOOGLE_API_KEY"; 

  public static GAddress geocode(String address, String key) throws Exception {
    URL url = new URL(URL + "&q=" + URLEncoder.encode(address, "UTF-8")
        + "&key=" + key);
    URLConnection conn = url.openConnection();
    ByteArrayOutputStream output = new ByteArrayOutputStream(1024);
    IOUtils.copy(conn.getInputStream(), output);
    output.close();

    GAddress gaddr = new GAddress();
    JSONObject json = JSONObject.fromString(output.toString());
    JSONObject placemark = (JSONObject) query(json, "Placemark[0]");

    final String commonId = "AddressDetails.Country.AdministrativeArea";

    gaddr.setFullAddress(query(placemark, "address").toString());
    gaddr.setZipCode(query(placemark,
        commonId + ".SubAdministrativeArea.Locality.PostalCode.PostalCodeNumber")
        .toString());
    gaddr.setAddress(query(placemark,
        commonId + ".SubAdministrativeArea.Locality.Thoroughfare.ThoroughfareName")
        .toString());
    gaddr.setCity(query(placemark,
        commonId + ".SubAdministrativeArea.SubAdministrativeAreaName").toString());
    gaddr.setState(query(placemark, commonId + ".AdministrativeAreaName").toString());
    gaddr.setLat(Double.parseDouble(query(placemark, "Point.coordinates[1]")
        .toString()));
    gaddr.setLng(Double.parseDouble(query(placemark, "Point.coordinates[0]")
        .toString()));
    return gaddr;
  }

  public static GAddress geocode(String address) throws Exception {
    return geocode(address, DEFAULT_KEY);
  }

  /* allow query for json nested objects, ie. Placemark[0].address */
  private static Object query(JSONObject jo, String query) {
    try {
      String[] keys = query.split("\\.");
      Object r = queryHelper(jo, keys[0]);
      for (int i = 1; i < keys.length; i++) {
        r = queryHelper(jo.fromObject(r), keys[i]);
      }
      return r;
    } catch (JSONException e) {
      return "";
    }
  }

  /* help in query array objects: Placemark[0] */
  private static Object queryHelper(JSONObject jo, String query) {
    int openIndex = query.indexOf('[');
    int endIndex = query.indexOf(']');
    if (openIndex > 0) {
      String key = query.substring(0, openIndex);
      int index = Integer.parseInt(query.substring(openIndex + 1, endIndex));
      return jo.getJSONArray(key).get(index);
    }
    return jo.get(query);
  }

  public static void main(String[] args) throws Exception {
    System.out.println(GGcoder.geocode("650 Townsend st, San Francsico, CA"));
    System.out.println(GGcoder.geocode("94103"));
  }
}


/* Class to hold geocode result */
public class GAddress {
  public String address;
  public String fullAddress;
  public String zipCode;
  public String city;
  public String state;
  public double lat;
  public double lng;

  /* getters and setters */

Make use of Java dynamic proxy

2007-08-16T08:56:00.000-07:00

I didn't know about Java dynamic proxy before but finally got a chance to learn and found a good use for it. The test I was writing had the smell of repeating code:


private void testPutWithoutSynch(params...) throws Exception {
  ...
  
  map.put("k1", "v1");
  
  assertions...
}

private void testPutWithSynch(params...) throws Exception {
  ...
  
  synchronized(map) {
    map.put("k1", "v1");
  }

  assertions...
}

The test pattern repeated with other operations that you can do with Map like puttAll, remove, etc... Do_set_up() is actually some pre setup steps that I need to perform according to parameters list. As you can see, I have to repeat the test method twice for each Map operation I want to test, with and without the synchronize block.

So there is this code smell and I didn't know get rid of it. Then Tim Eck, my coworker, showed me how to use Java proxy. The idea is to wrap my map in a proxy, and use reflection to invoke methods of the map. By using a invocation handler, I can have a variable to control whether or not I want to use synchronize for that particular operation.


class Handler implements InvocationHandler {
  private final Map map;
  private boolean useSynch = false;

  public Handler(Map map) {
    this.map = map;
  }

  public void setUseSynch(boolean flag) {
    this.useSynch = flag;
  } 

  public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
    try {
      if (useSynch) {
        return invokeWithSynch(method, args);
      } else {
        return method.invoke(map, args);
      }
    } catch (InvocationTargetException e) {
      throw e.getTargetException();
    }
  }

  private Object invokeWithSynch(Method method, Object[] args) throws Throwable {
    synchronized (map) {
      return method.invoke(map, args);
    }
  }
}

Then the proxy can be created like this:


Map map = new HashMap();
Handler handler = new Handler(map);
Map mapProxy = (Map)Proxy.newProxyInstance(getClass().getClassLoader(), new Class[] { Map.class }, handler);

With that my test methods can be reduced to:


private void testPut(params..., boolean useSynch) throws Exception {
  do_set_up(params)
  
  handler.setUseSynch(useSynch);
  mapProxy.put("k1", "v1");

  assertions depends on useSynch
}

Then I just need to call this method twice and accomplish the same thing.


testPut(params..., false);
testPut(params..., true);

Pretty neat huh? Java proxy is nice tool to know.

Cluster a standalone Spring app to calculate Mandelbrot set

2007-07-26T09:07:00.000-07:00

I've heard about Spring for awhile and have made little attempt to check out what it was. Most of the descriptions about Spring gave me the impressions that only big enterprise applications would use Spring. Man I was wrong :) As I dug down into Pro Spring book and read the Spring reference, I was able to convert one of my stand alone apps into Spring and reaped the benefit that Spring has to offer. Not only that, I was able to cluster my bean to share it among JVMs with little effort, thanks to Terracotta.

My application is a simple demo of calculating the Mandelbrot set. It is multithreaded, using a queue task and workers. Here is a rough sketch (I'm UML illiterate):


-------------------------------------            -----------------------------
 MandelbrotModel:                                  CalculateNode : Runnable
   rawData: int[][]                                 model: MandelbrotModel
   workLoad: BlockingQueue
 ------------------------------------  1 ----->* -----------------------------

Each CaculateNode has a reference to the model, thus the workLoad queue. It will take a task (a Segment) out of workLoad, process it, and put back the findings in "rawData". Spring comes into the picture when it allows me to wire the dependency between nodes and model, like so:


   <bean id="model" class="mandelbrot.MandelbrotModel" scope="singleton">
        <property name="length">
            <value>600</value>
        </property>    
        <property name="numNodes">
            <value>2</value>
        </property>
        <property name="numTasks">
            <value>20</value>
        </property>        
    </bean>

    <bean id="node" class="mandelbrot.CalculateNode" scope="singleton">
        <property name="model" ref="model">
        </property>
    </bean>

The above snippet describes 2 beans, a "model" and a "node". Properties are 1 on 1 matching of fields in your class. Say the field "private int length" in the model is set to 600 pixel, and so on. Spring "magic" is in the "node" bean, it says, point node's field of "model" to the model bean above, hence the dependency injection.


public class CalculateNode implements Runnable {
  private MandelbrotModel        model;

  public void setModel(MandelbrotModel model) {
    this.model = model;
  }
}

Notice I have a setModel() method in the CalculateNode but I don't have to call it explicitly in my program. Spring will call it and set it for me. My node can do all it work, pretending that it has the model reference. The main method is really simple.


  public static void main(String[] args) {
    ApplicationContext ctx = new ClassPathXmlApplicationContext(
        "mandelbrot.xml");
    
    CalculateNode node1 = (CalculateNode) ctx.getBean("node");
    new Thread(node1).start();
    
    CalculateNode node2 = (CalculateNode) ctx.getBean("node");
    new Thread(node2).start();
  }

I just need to query out 2 nodes and start them in 2 threads, no need to worry where is my model and how the nodes got a reference to it.

But wait, there's more. Spring also offers to publish application events for you for cheap. Say now that I have data of the Mandelbrot, I can display it on a Swing application. I don't want to wait for all the nodes to finish calculation and display the graphic in the end. I want to paint it in real time, as soon as a node finishes with a segment, I'll paint it. To pull that off, I need to know when a segment is finished, the right source would be to ask the model. It knows that information when a node reports back with data. The "ghetto" way would be implement a listener list, traverse that list and and invoke callback function for each listener. This was the old implementation that I had. It does the job but it aint neat. In real world, you might see how this is not desirable sometimes because it couples your components. Spring allows you to solve this problem nicely: if you want to publish events, use the application context and call publishEvent() with your message! That's all you have to do.

Let examine this function in the MandelbrotModel. It's invoked by a node to report result


  public synchronized void addSegmentData(int[][] data, Segment segment) {
    for (int row = segment.getStart(); row < segment.getEnd(); row++) {
      System.arraycopy(data[row - segment.getStart()], 0, rawData[row], 0,
          length);
    }

    ctx.publishEvent(new SegmentEvent(this, segment));
  }

"ctx" is a reference to the context of the application, automatically given to anyone if you implement a Spring interface "ApplicationContextAware".


  @Override
  public void setApplicationContext(ApplicationContext applicationcontext)
      throws BeansException {
    ctx = applicationcontext;
  }

As for someone who wants to listen to events, implements ApplicationListener interface and catch your events:


  @Override
  public void onApplicationEvent(ApplicationEvent event) {
    if (event instanceof SegmentEvent) {
      final SegmentEvent segEvent = (SegmentEvent) event;
      processSegment(segEvent.getSegment());
    }
  }

I have my ViewerFrame (extends from JFrame) to act as listener, declared in my XML bean definition file and also has a reference to the model:


      <bean id="frame" class="mandelbrot.ViewerFrame" scope="singleton">
        <property name="model" ref="model">
        </property>
    </bean>

There is no change in the main method. As Spring initializes the "frame" bean, it will automatically show up. This is pretty easy because I made itself aware, knowing that it has been created by implementing InitialzingBean interface


  @Override
  public void afterPropertiesSet() throws Exception {
    int length = model.getLength();
    this.image = new BufferedImage(length, length, BufferedImage.TYPE_INT_RGB);

    this.setSize(length, length);
    SwingUtilities.invokeLater(new Runnable() {
      public void run() {
        setVisible(true);
      }
    });
  }

The fun doesn't end here though. Now I want to have another JVM to join in the calculation, making it go faster (a little bit of drama here since the Mandelbrot set is pretty fast to calculate). The ideal scenario is the second JVM also has the same "model" in memory then its minions of "nodes" can just pound on the work. This second JVM can be on a totally separate machine even. How the heck anyone is gonna pull this off without changing the code, worrying about networking, sharing heap, etc...? This is where the power of Terracotta comes in. I just mark the "model" bean as shared and the SegmentEvent as distributed. That will enable any JVM in the cluster having reference to this model, and events published from model will be cluster-wise.

Terracotta is fully Spring aware, and the snippet below is how I wired it to my Spring app.


  <spring>
      <jee-application name="*">
        <application-contexts>
          <application-context>
            <paths>
              <path>mandelbrot.xml</path>
            </paths>
            <beans>
              <bean name="model">
              </bean>
            </beans>
            <distributed-events>
             <distributed-event>mandelbrot.SegmentEvent</distributed-event>
            </distributed-events>
          </application-context>
        </application-contexts>
      </jee-application>
</spring>

When I use Terracotta eclipse plugin, after starting a Terracotta server (used to hold the shared objects), starting two instances of my applications that shared the "model" is as easy as hitting the Run button twice. Now I have 2 JVMs, each has 2 nodes working on the same model. Neato. As my bird would say "oh wow" when something excites him.

So there you have it. I'm a Spring novice now but I'm sure there are many cools things from Spring waiting to be discovered. And with the added boost from Terracotta, the fun is multiplied.

Here is the link to the source code

Alternative to properties file: YAML beans

2007-06-27T09:52:00.000-07:00

Plain ol properties files are great for setting configurations for your apps. But if your configuration requires structured data and you prefer not use XML, there is a great alternative: YAML format and beans.

Let assume we have a config as below. One client and multiple servers.


client:
 log: /tmp/client/log
 debug: on

servers:
 - host: zeus
   port: 9510

 - host: apollo
   port: 9511

Yaml treats the whole config file as a hash map, keys are "client" and "servers". Subsequently, it treats "client" as a another hash map, and "servers" is an array of maps. Sounds complicated but it's straight forward once you're getting a hang of it.

To parse this config, JYaml makes it easier than fixing cereal:


   Map config = (Map) YAML.load(new FileReader("config.yml"));

   Map client = (Map) config.get("client");
   System.out.println("Client: " + client);
      
   List servers = (ArrayList) config.get("servers");
   System.out.println("Servers: " + servers);

And output is:


Client: {debug=true, log=/tmp/client/log}
Servers: [{host=zeus, port=9510}, {host=apollo, port=9511}]

You would access client's and server's properties like this:


  client.get("log");
  Map firstServer = (Map)servers.get(0);
  firstServer.get("host");

If you want the config being mapped to Java beans, JYaml can populate the beans for you. This is my preferred way since it's much nicer.

The beans for your config will be as simple as:


/* Top level bean for the whole config */
public class TcConfig {
  private List servers;
  private Client client;
  
  public List getServers() {
    return servers;
  }
  public void setServers(List servers) {
    this.servers = servers;
  }
  public Client getClient() {
    return client;
  }
  public void setClient(Client client) {
    this.client = client;
  }
}

/* Client */
public class Client {
  private String log;
  private boolean debug;
  
  /* getters and setters */
  
  public String toString() {
    return "log=" + log + ", debug=" + debug;
  }
}

/* Server */
public class Server {
  private String host;
  private int port;
  private String log;
  
  /* getters and setters */
  
  public String toString() {
    return "host=" + host + ", port=" + port + ", log=" + log;
  }
}

And to populate your beans, just 1 line of code:


  TcConfig config = (TcConfig)Yaml.loadType(new File("config.yml"), TcConfig.class);
      
  System.out.println("Servers : " + config.getServers());
  System.out.println("Client : " + config.getClient());

XMLBeans can accomplish the same thing with other features like autogenerated Java beans, enforce schema, etc, but for simple things, I prefer Yaml.

Share that POJO - Hibernate clustered and empowered

2007-06-14T11:11:00.000-07:00

I haven't had much experience with OR mapping in Java apps so I wanted to study Hibernate tutorial and tried to cluster it at the same time. Starting with Terracotta 2.4 (currently at stable0), Hibernate is supported.

My first question to the engineer who worked on the feature is "What is being shared in Hibernate?" It turns out, the plain old Java objects (POJOs) that Hibernate constructs from the database are the ones that we're interested in sharing, in this case, across multiple JVMs. I told myself, that's pretty handy, so we don't have to hit the database again for the same information on another node when those objects have already been loaded and shared thanks to Terracotta DSO server. He told me that's not the only cool feature though. With those shared objects from Hibernate, one JVM can just reassociate them to its Hibernate session and start to access colletions that are mapped as one-to-many, many-to-many, that don't exist yet in memory due to lazy initalization. Wow. (Of course, that Hibernate session will have to hit the database for that new info you request. You don't always get free beer!)

With that peaked interest, I worked on this cool and thorough tutorial from Hibernate.org

The schema for this tutorial as follow:


  EVENTS           PERSON_EVENT       PERSON
  ----------       -------------      ----------
  *EVENT_ID        *EVENT_ID          *PERSON_ID
   EVENT_DATE      *PERSON_ID          FIRSTNAME
   TITLE                               LASTNAME
  
    
  PERSON_EMAIL_ADDR
  ------------------
  *PERSON_ID
  *EMAIL_ADDR

For each table, there is a Javabean to represents it. Each javabean instance will be mapped to a row in the database. Well, that's Hibernate in a nutshell for me (speaking with my own ignorance and limited experience)

The xml mapping for Events table as follow:


<hibernate-mapping>

<class name="events.Event" table="EVENTS">
  <id name="id" column="EVENT_ID">
      <generator class="native"/>
  </id>
  <property name="date" type="timestamp" column="EVENT_DATE"/>
  <property name="title"/>

  <set name="participants" table="PERSON_EVENT" lazy="true" inverse="true" cascade="lock">
      <key column="EVENT_ID"/>
      <many-to-many column="PERSON_ID" class="events.Person"/>
  </set>
</class>

</hibernate-mapping>

It's a many-to-many relationship between Events and Person. Notice I set the lazy loading to true, it is needed to demonstrate the point I mentioned earlier.

Here is Person's map file.


<class name="events.Person" table="PERSON">
  <id name="id" column="PERSON_ID">
      <generator class="native"/>
  </id>
  <property name="age"/>
  <property name="firstname"/>
  <property name="lastname"/>

  <set name="events" table="PERSON_EVENT">
      <key column="PERSON_ID"/>
      <many-to-many column="EVENT_ID" class="events.Event"/>
  </set>

  <set name="emailAddresses" table="PERSON_EMAIL_ADDR">
    <key column="PERSON_ID"/>
    <element type="string" column="EMAIL_ADDR"/>
  </set>

</class>

For hibernate.cfg.xml file, I changed the config from the tutorial a little bit to use Derby database. It is run as server mode. Here is the change:


<property name='connection.driver_class'>org.apache.derby.jdbc.ClientDriver</property>
<property name='connection.url'>jdbc:derby://localhost:1527/MyDbTest</property>
<property name='connection.username'>user1</property>
<property name='connection.password'>user1</property>

Derby is shipped with jdk1.6.0_01, under java-home/db/lib. To start it,


% java -jar derbyrun.jar server start

Then I created "MyDbTest" database by running this once:


% java -jar derbyrun.jar dblook -d "jdbc:derby://localhost:1527/MyDbTest;create=true"

Now on to the EventManager.java of the tutorial. I didn't follow it fully. I created 2 events, and 3 persons. First event is "Engineer meeting", second one is a "Document Meeting" contain these participants

// engMeeting = {steve, orion, tim}
// docMeeting = {steve, orion}

Here is a snippet of EventManager.java.


public class EventManager {

// shared object - declared as a root in tc-config.xml
private static final List events = new ArrayList();

public static void main(String[] args) {
  EventManager mgr = new EventManager();

  // create 3 persons Steve, Orion, Tim
  Long steveId = mgr.createAndStorePerson("Steve", "Harris");
  mgr.addEmailToPerson(steveId, "steve@terracottatech.com");

  ...

  // create 2 events
  Long engMeetingId = mgr.createAndStoreEvent("Eng Meeting", new Date());
  mgr.addPersonToEvent(steveId, engMeetingId);
  mgr.addPersonToEvent(orionId, engMeetingId);
  mgr.addPersonToEvent(timId, engMeetingId);

  Long docMeetingId = mgr.createAndStoreEvent("Doc Meeting", new Date());

  ....

  // store our events into a shared object
  synchronized (events) {
    events.addAll(mgr.listEvents());
  }

  HibernateUtil.getSessionFactory().close();
}

private List listEvents() {

  Session session = HibernateUtil.getSessionFactory().getCurrentSession();
  session.beginTransaction();

  List result = session.createQuery("from Event").list();

  session.getTransaction().commit();
  return result;
}

Most of it is straight forward and copied from the Hibernate tutorial. Terracotta comes into the picture with the presence of the shared-object "events". It is declared as a root in Terracotta config and it will hold the list of events that Hibernate loaded from database. Here is how declare it in tc-config.xml:


<root>
<field-name>events.EventManager.events</field-name>
<root-name>tcEvents</root-name>
</root>

The synchronized block of shared object "events" is necessary. It's in a context of multithreads and multi-JVM.

So that's enough for us to start the first JVM. I turned on SQL log from log4j so we can see the SQL being generated by Hibernate. There are a lot of sql statements printed out, both "insert" and "select" as you would expect Hibernate to
generate. After it's finished, our "events" list will contain events object, and since it's set with "lazy=true", their participant set will be empty. We can confirm this by looking at Terracotta Admin console:

Under root, we have 1 root, namely "tcEvents" that has 2 objects. Expanding them, and we'll see the participant set is empty. Hibernate internal set is null.

org.hibernate.collection.PersistentSet.set=null

So that is my first app. Here comes my second app that run in a totally seperate VM. I caled it EventChecker and it will list emails of people in the first meeting.

First, it has to know about the same shared object "tcEvents" that we put onto Terracotta DSO Server. We can do this easily by specify it in tc-config.xml:


<roots>
  <root>
    <field-name>events.EventManager.events</field-name>
    <root-name>tcEvents</root-name>
  </root>
  <root>
    <field-name>events.EventChecker.events</field-name>
    <root-name>tcEvents</root-name>
  </root>
</roots>

By using the same root-name, it's mapped to the same list of EventManger. Its code is pretty short so I list all of it here:


public class EventChecker {
  // shared object
  private static final List events = new ArrayList();

  public static void main(String[] args) {

  synchronized (events) {
    // list events before even opening a Hibernate session
    System.out.println("** events: " + events);

    Session session = HibernateUtil.getSessionFactory().getCurrentSession();
    session.beginTransaction();

    // reassociate transient pojos to this session
    for (Iterator it = events.iterator(); it.hasNext(); ) {
      session.lock((Event)it.next(), LockMode.NONE);
    }

    // list people in first event
    Event event = (Event)events.get(0);
    Set people = event.getParticipants();
    System.out.println("** people: " + people);

    // list emails of people from first event
    Set emails = new HashSet();
    for (Iterator it = people.iterator(); it.hasNext(); ) {
      Person person = (Person)it.next();          
      emails.addAll(person.getEmailAddresses());
    }
    System.out.println("** emails: " + emails);

    session.getTransaction().commit();
    HibernateUtil.getSessionFactory().close();
  }
 }
}

Here it is you see the usage of synchorized block on the shared object. We're accessing it in brand new JVM. Without Terracotta, the events list will be empty. With Terracotta, the events list contains events from EventManager, faulting by DSO server transparently.
I then reassociated the pojos to my newly created Hibernate session. This is always needed when you work with Hibernate. I didn't know about this until I got an exception saying I don't have a session or session might have been closed by Hibernate.

After that, I can list the pariticipants and their emails using Hibernate. This is all Hibernate doing. I love it :)

Here is the output, it will show that Hibernate didn't generate SQL to load in the events since it's already loaded and stored by Terracotta. It only loaded from Person and Person_Email_Addr:


** events: [Eng Meeting: 2007-06-14 11:06:12.0, Doc Meeting: 2007-06-14 11:06:12.0]

Hibernate: select participan0_.EVENT_ID as EVENT1_1_, participan0_.PERSON_ID as PERSON2_1_, person1_.PERSON_ID as PERSON1_2_0_, person1_.age as age2_0_, person1_.firstname as firstname2_0_, person1_.lastname as lastname2_0_ from PERSON_EVENT participan0_ left outer join PERSON person1_ on participan0_.PERSON_ID=person1_.PERSON_ID where participan0_.EVENT_ID=?

** people: [Orion Letizi, Steve Harris, Tim Eck]

Hibernate: select emailaddre0_.PERSON_ID as PERSON1_0_, emailaddre0_.EMAIL_ADDR as EMAIL2_0_ from PERSON_EMAIL_ADDR emailaddre0_ where emailaddre0_.PERSON_ID=?
Hibernate: select emailaddre0_.PERSON_ID as PERSON1_0_, emailaddre0_.EMAIL_ADDR as EMAIL2_0_ from PERSON_EMAIL_ADDR emailaddre0_ where emailaddre0_.PERSON_ID=?
Hibernate: select emailaddre0_.PERSON_ID as PERSON1_0_, emailaddre0_.EMAIL_ADDR as EMAIL2_0_ from PERSON_EMAIL_ADDR emailaddre0_ where emailaddre0_.PERSON_ID=?

** emails: [teck@terracottatech.com, steve@terracottatech.com, orion@terracottatech.com]

Looking at the Admin console again, we see that the participant set is now loaded in by Hibernate and shared by Terracotta.

There is one gotcha that I ran into that I thought I should mention. In hibernate.cfg.xml, there is an option to clean out and create the schema every time Hibernate start.


<!-- Drop and re-create the database schema on startup -->
<property name=\"hbm2ddl.auto\">create</property>

When you run EventManger, you need it to create the schema for you. But when you run EventChecker, you don't want Hibernate to wipe out your database. You should comment out that line before running it.

I had fun working on Hibernate tutorial and playing it Terracotta. I don't have much experience with J2EE and Hibernate so I can't really comment on a practical use case. Maybe you will have better idea how to use it :)

Performance comparision between ConcurrentHashMap and synchronized HashMap in Terracotta

2007-04-13T11:22:00.000-07:00

Since the introduction of ConcurrentHashMap in Java 5, it has been the better choice over HashMap in highly threaded applications. To see how much better, Brian Goetz from his book "Java Concurrency in Practice" wrote a test to compare performance between the two maps. The test sceanario is this:

For N threads concurrently execute a loop that chooses a random key and looks up value corresponding to that key. If the value is found, it is removed with a probability of 0.02. If not, it is added to the map with a probability of 0.6.

The result is ConcurrentHashMap performs much better when number of threads increases. Below is the graph taken from the book with the author's permission.
Since Open Terracotta supports ConcurrentHashMap, I wanted so see if the performance advantage is still there (and how much) in distributed environment. My test scenario is modeled after Brian's. However, the number of threads are spread over 4 nodes (Linux RH4). Here is my test code:


package tc.qa;

import java.util.Map;
import java.util.Random;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CyclicBarrier;
import java.util.concurrent.atomic.AtomicInteger;

public class ConcurrentHashMapLoadTest extends Thread {
  private static double WRITE_PROBABILITY  = 0.6;
  private static double REMOVE_PROBABILITY = 0.02;
  private static int    THREAD_COUNT       = 4;
  private static int    VM_COUNT           = 4;
  private static int    RUNTIME            = 5 * 60 * 1000;
  private static int    KEY_RANGE          = 100000;

  // roots - in TC world, these are static finals and accessible across JVMs
  private Map           map                = new ConcurrentHashMap();
  private AtomicInteger throughput         = new AtomicInteger(0);
  private CyclicBarrier barrier;

  private Random        random;
  private int           op;

  public ConcurrentHashMapLoadTest() {
    random = new Random();
    barrier = new CyclicBarrier(THREAD_COUNT);
  }

  public void run() {

    // ready
    if (barrier() == 0) {
      System.out.println("Started...");
    }

    // go
    long start = System.currentTimeMillis();
    while (System.currentTimeMillis() - start < RUNTIME) {
      Integer key = new Integer(random.nextInt(KEY_RANGE));
      if (get(map, key) != null) {
        if (random.nextDouble() < REMOVE_PROBABILITY) {
          remove(map, key);
          op++;
        }
      } else {
        if (random.nextDouble() < WRITE_PROBABILITY) {
          put(map, key, key);
          op++;
        }
      }
      op++;
    }

    throughput.addAndGet(op);

    if (barrier() == 0) {
      System.out.println("Map type:           "
          + map.getClass().getSimpleName());
      System.out.println("Runtime:            " + RUNTIME);
      System.out.println("Number of threads:  " + THREAD_COUNT);
      System.out.println("Write probability:  " + WRITE_PROBABILITY);
      System.out.println("Remove probability: " + REMOVE_PROBABILITY);
      System.out.println("Ops per second:     "
          + (throughput.intValue() * 1000.0 / RUNTIME));
    }
  }

  private int barrier() {
    try {
      return barrier.await();
    } catch (Exception e) {
      e.printStackTrace();
    }
    return -1;
  }

  private Object get(Map map, Object key) {
    if (map instanceof ConcurrentHashMap) {
      return map.get(key);
    } else {
      synchronized (map) {
        return map.get(key);
      }
    }
  }

  private void put(Map map, Object key, Object value) {
    if (map instanceof ConcurrentHashMap) {
      map.put(key, value);
    } else {
      synchronized (map) {
        map.put(key, value);
      }
    }
  }

  private void remove(Map map, Object key) {
    if (map instanceof ConcurrentHashMap) {
      map.remove(key);
    } else {
      synchronized (map) {
        map.remove(key);
      }
    }
  }

  public static void getParams() {
    WRITE_PROBABILITY = Double.parseDouble(System.getProperty("wp", "0.6"));
    REMOVE_PROBABILITY = Double.parseDouble(System.getProperty("rmp", "0.02"));
    THREAD_COUNT = Integer.parseInt(System.getProperty("thread", "4"));
    VM_COUNT = Integer.parseInt(System.getProperty("vm", "4"));
    RUNTIME = Integer.parseInt(System.getProperty("runtime", "300000"));
  }

  public static void main(String[] args) {
    getParams();
    int threads_per_vm = THREAD_COUNT / VM_COUNT;
    for (int t = 0; t < threads_per_vm; t++) {
      ConcurrentHashMapLoadTest thread = new ConcurrentHashMapLoadTest();
      thread.start();
    }
  }

}

I wrapped the put(), get() methods to easily switch the map types. After running this test on 4 nodes with jdk1.6.0_01, I got result as below:

The Y axis is throughput, normalized to throughput of 4 threads in ConcurrentHashMap.
With network overhead and locks contention in Terracotta, ConcurrentHashMap still far outweights HashMap. Notice how performance of HashMap doesn't degrade much when number of threads increases. This all thanks to the way Terracotta replicates only the delta changes in the map to other nodes. No serialization involved.

The gap between ConcurrentHashMap and HashMap in Terracotta isn't big as it was in the case of 1 JVM like Brian's test. I'm not sure I'm comparing oranges with apples here because obviously it's not the same test, just the same idea. We're always striving for better performance overall and each release has proven that.

Check us out and let us know what you think. http://www.terracotta.org

Hung-

Using POSTGIS to find points of interest within a radius

2007-02-11T19:05:00.000-08:00

I've recently looked into using PostGIS, an extension that enables support for spatial objects in PostgresSQL database.

The problem I was trying to solve is, given a point with (longitude, latitude), how can I find all other points within a certain radius. I've solved it before using formula to calculate distances between 2 points but it's pretty slow if your table has a lot of records. PostGIS can help solve this problem faster and more elegant.

First thing first, let's get us some data to play around with. I found this excellent tutorial that teaches you how to get zipcode data from the US Census. I'll skim through it here, but you should definitely check out that article.

Assume you have a zipcode table with these columns:


state    zip     latitude        longitude
-------------------------------------------
CA 92230 33.911404 -116.768347
CA 92234 33.807761 -116.464731
CA 92236 33.679872 -116.176562

Now you need to add a column to this table to hold the geometric data represents by [longitude, latitude] pair.


SELECT AddGeometryColumn( 'public', 'zipcode', 'geom', 32661, 'POINT', 2 );

* add column 'geom' into table 'zipcode' under schema 'public'
* the column holds a point with 2 dimensions and has spherical reference id (srid) 32661

* now we populate the table with data converted from srid 4269 (longitude/latitude measurement) to srid 32661 (WGS 84 system)


UPDATE zipcode
SET geom = transform(PointFromText('POINT(' || longitude || ' ' || latitude || ')',4269),32661) ;

The transformation from one spherical reference to another is needed for calculating the surface distance. WGS84 is measured in meters by the way.

Now your table will look like this


state zip  latitude   longitude    geom
-----------------------------------------------
CA  92230  33.911404  -116.768347  0101000020957F0000B7FB4ED5F2C44EC166AFB20D1A3D5341
CA  92234  33.807761  -116.464731  0101000020957F0000AA8102ECECFD4EC18284302439245341
CA  92236  33.679872  -116.176562  0101000020957F000067A4B44D2D3B4FC1596E974B370E5341

Now we only need to write a query, say, find me all the zipcode within 10 mile radius of "92230". (or 16093 meters)

SELECT state, zip
FROM zipcode
WHERE
distance(
transform(PointFromText('POINT(-116.768347 33.911404)', 4269),32661),
geom) < 16093

Here function distance(here_geom, there_geom) returns meters. Notice how you would read in a point(long/lat) from string using PointFromText() then do the transform.

However, this query isn't optimal. It calculates distances from zip 92230 to all other zipcodes. We would rather have a bounding box of 10 miles around our interested point, filtering out the points that interact with this box then calculate the exact distances to the center. Sounds like a lot of work but in reality, it can be achieve by adding another condition:

SELECT state, zip
FROM zipcode
WHERE
geom && expand(transform(PointFromText('POINT(-116.768347 33.911404)', 4269),32661), 16093)
AND
distance(
transform(PointFromText('POINT(-116.768347 33.911404)', 4269),32661),
geom) < 16093

The new query is much much faster. My benchmark shows 200% in speed compared to the first query. Pretty sweet!

Using Java bytecode in clustering technique

2006-12-12T14:33:00.000-08:00

When I first heard of clustering Java objects, I immediately thought about object serialization. As someone just got out of school, it's an educated deduction :) I soon learned it's not always the case when speed and scalability are taken into account. Serialization of a plain ol' java object is rather slow when you have lots of transactions, and lots of objects. So how else then you would maintain the object integrity across cluster?

For Terracotta DSO, we choose to go one level down, the JVM bytecode (as opposed to the application level) There's like a whole new world of Java when you decide to look into it. As it turns out, every time you set or get values to variables, the opcodes for them are "getfield" and "putfield". DSO looks for these opcodes and record the mutation of an object in one JVM, then replay that "tape" in another JVM on the same clustered object. The final effect is that we can replicate object data in multiple JVMs on the field level.

Let take a look at a simple class: Person.java
To create the byte code, run these commands:

$> javac Person.java
$> javap -c Person > Person.bc

Person.java


public class Person {
   private String name;
  
   public Person(String aName) {
       name = aName;
   }
  
   public String getName() {
       return name;
   }
  
   public void setName(String aName) {
       name = aName;
   }
}

Person.bc


Compiled from "Person.java"
public class Person extends java.lang.Object{
public Person(java.lang.String);
 Code:
  0:    aload_0
  1:    invokespecial    #1; //Method java/lang/Object."":()V
  4:    aload_0
  5:    aload_1
  6:    putfield    #2; //Field name:Ljava/lang/String;
  9:    return

public java.lang.String getName();
 Code:
  0:    aload_0
  1:    getfield    #2; //Field name:Ljava/lang/String;
  4:    areturn

public void setName(java.lang.String);
 Code:
  0:    aload_0
  1:    aload_1
  2:    putfield    #2; //Field name:Ljava/lang/String;
  5:    return

}

Voila, the gut of our class is exposed. For DSO, this is what we work on. So as you can see, the DSO claim about only sending the deltas (the changes) of an object across network, makes sense. The magic is in the bytecode instrumentation, using ASM framework.

Similarly, locking by using "synchronized" blocks can be observed by the tell-tales "monitorenter", "monitorexit" bytecodes. Don't take my words for it, try it out yourself. I've learned a great deal since I start studying Java bytecode.

One good source I found is: Java bytecode:
Understanding bytecode makes you a better programmer

Ari Zilka talk at Google is another great source.

Have fun-

Share precious heap memory across multiple JVM's

2006-10-08T23:56:00.000-07:00

I've written some puzzle solvers for fun during college. Among them was for Boggle board. One annoying thing about it is the startup time. It took a while to load up a 4 MB English dictionary into memory. Some of my other puzzle solvers also use the same dictionary. Wouldn't it be cool if you only need to load this dictionary once and all of the apps could use it? It would stay resident in the heap and ready to be used in different applications and between runs.

One other solution is to use a database but this is not always preferable, mostly due to speed, and of course it is not as elegant. I wanted to use the dictionary object as a native Java object. The idea was shelved for awhile until I came to work with Terracotta, whose distributed shared objects (DSO) might just pull off the resident-heap-memory idea nicely.

First, let's take look at the puzzle solver. Given a 4 by 4 board, the main objective is to find as many words as possible that can be constructed from the letters of sequentially adjacent cubes. The board I’m solving in the game has the content as follows:

c  a  t  d
l  i  n  e
m  a  r  o
p  e  t  s

Some obvious words are: calm, line, mine, and dent.

The program will use the dictionary to scan for all possibilities. A recursive algorithm works great in solving this puzzle. For every possible path, it will look up the current word in dictionary. So let's see how we get the dictionary loaded into memory.

In the Terracotta world, that would mean loading the data into a root (shared object). That shared object will be transparently clustered and made available to any other JVM in the cluster.

The dictionary is implemented as a Trie for fast retrieving. The dictionary loader will read words from a file, one word per line and insert into the trie’s structure. For efficiency, we'll batch the inserts. Here I choose the batch size to be 50. Adjusting this number will affect the load time.


package dictionary;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.List;
import java.util.Vector;


public class DictionaryLoader {

private Dictionary dictionary = new Dictionary();

public void load(File dictFile) {
  int batchSize = 50;
  List<String> batchOfWords = new Vector<String>();

  BufferedReader reader = null;
  try {
      reader = new BufferedReader(new FileReader(dictFile));
      String word;
      while ((word = reader.readLine()) != null) {
          batchOfWords.add(word);

          if (batchOfWords.size() == batchSize) {
              synchronized (dictionary) {
                  dictionary.add(batchOfWords);
                  batchOfWords.clear();
              }
          }
      }

      /* add the rest of the words left in the batch */
      if (batchOfWords.size() > 0) {
          synchronized (dictionary) {
              dictionary.add(batchOfWords);
          }
      }

      System.out.println("size = " + dictionary.size());

  } catch (Exception e) {
      e.printStackTrace();
  }

  finally {
      try {
          if (reader != null) reader.close();
      } catch (IOException e) {
          e.printStackTrace();
      }
  }
}

public static void main(String[] args) {
  DictionaryLoader loader = new DictionaryLoader();
  long elapsed = System.currentTimeMillis();
  loader.load(new File("dict.txt"));
  elapsed = System.currentTimeMillis() - elapsed;
  System.out.println("Time took to load the dictionary (ms): "
                      + elapsed);
}
}

In the Terracotta configuration file, tc-config.xml, we declare the dictionary field to be a shared "root." The entire graph of objects referenceable by this root automatically becomes clustered by Terracotta. We also give this root field a cluster-wide name ("dictionary") with the root-name tag. This is very important because this is how other applications get access to this same root.

<root>
<field-name>dictionary.DictionaryLoader.dictionary</field-name>
<root-name>dictionary</root-name>
</root>

To test our little dictionary, I wrote the DictionaryLookup app. It also declares a dictionary object, but names it differently.


package dictionary;

public class DictionaryLookup {

private static final Dictionary dict = new Dictionary();


public void checkWord(String word) {
  if (dict.isWord(word)) {
      System.out.println("'" + word + "' is a word" );
  }
  else if (dict.isPrefix(word)) {
      System.out.println("'" + word + "' is a prefix");
  }
  else {
      System.out.println("'" + word + "' is NOT found");
  }
}


public static void main(String[] args) {

  DictionaryLookup dl = new DictionaryLookup();
  String[] words = {"eat", "my", "shorts", "homer", "crapola", "dict",
          "config", "configuration", "sweet", "abracadabra"};

  for (int i = 0; i < words.length; i++) {
       dl.checkWord(words[i]);
   }
}
}

The field "dict" is declared as root in tc-config.xml but that doesn’t mean the server will have 2 roots (namely two clustered versions of dictionary objects). The trick is the root-name tag. It tells the server to treat these two roots as one, interchangeably.

<root>
<field-name>dictionary.DictionaryLoader.dictionary</field-name>
<root-name>dictionary</root-name>
</root>
<root>
<field-name>dictionary.DictionaryLookup.dict</field-name>
<root-name>dictionary</root-name>
</root>

The interesting thing about the DictionaryLookup is how it assumes the dictionary content is already there and available for use. All it does is to declare a Dictionary field and Terracotta automatically assigns the same “dictionary” shared root to that field.

DictionaryLookup has proven that this approach works correctly and as I wanted. I can go ahead and attach the dictionary to our puzzle solver:

<root>
<field-name>dictionary.DictionaryLoader.dictionary</field-name>
<root-name>dictionary</root-name>
</root>
<root>
<field-name>dictionary.DictionaryLookup.dict</field-name>
<root-name>dictionary</root-name>
</root>
<root>
<field-name>games.boggle.Board.theDictionary</field-name>
<root-name>dictionary</root-name>
</root>

If you look at Board.java, you'll see how the game can focus on its main logic without having to worry about loading up the dictionary in memory. This eliminates huge amount of time if you choose to use a really big size dictionary. And not only just one application can use dictionary at a time; multiple applications from multiple networks can take advantage of this network attached memory.

Source code available here.