Li Xiao is on the way

Tips of embedding ruby gem in Java

2009-05-29T22:06:00.000-07:00

I spent 2 weeks on embedding a ruby workflow engine ruote in Java by JRuby (will publish it later).

The followings are tips I got:

The JRuby wiki page Getting Started (http://wiki.jruby.org/wiki/Getting_Started) is out of date. Read http://wiki.jruby.org/wiki/Direct_JRuby_Embedding instead.
Should not use JavaUtil.* to convert java/ruby objects, which doesn't work well; use JavaEmbedUtils instead. And you probably should read javadoc of JavaEmbedUtils.
JavaEmbedUtils#invokeMethod accepts Java objects as arguments to call a method of ruby object so that you don't have to convert every Java objects to JRuby objects.
Create adapter for ruby object and wrap the ruby object with pure Java interface. So that no more JRuby interface need to care about outside.
Here is a JRuby IRubyObject adapter, which could get embedding easier.
A org.jruby.Ruby object looks like is thread safe, no matter calling evalScriptlet / callMethod. Wrote a test for it and worked well in my case.
Call JavaEmbedUtils#terminate to tear down a ruby runtime object and release resources, which is important in tests, otherwise you probably get out-of memory error.
Be careful to use YAML to dump a ruby object to string and convert the string to Java String, then load it back later. It didn't work well in my case when there is non-string ruby object attribute inside the ruby object dumped. In my case, the object attribute inside was load as string object when load the dumped string. And I failed to write a simple case to reproduce the problem. Marshal works well for me. (I use JRuby 1.3.0RC2 in my case)
You can package ruby gem's lib directory with your Java code as jar file to publish it. (same with jruby complete jar)
When package ruby code inside jar file, should be careful with using 'File.dirname(__FILE__)' to locate file location, which may doesn't work anymore (The "File.dirname(__FILE__)" will be looked like "...xx/xxx.jar!yyyy/zzzz").
If you are depending on another JRuby gem (e.g. ActiveRecord-jdbc-adapter) which includes a jar file in lib, you should package with the class files instead of the jar file, or put the jar file separated with your jar file. Java could not load a jar inside a jar file.

Fireworks is back, a tool auto run tests in IntelliJ IDEA for you

2009-04-25T08:57:00.000-07:00

Back to Java project currently. Kent's JUnitMax makes me miss my Fireworks, so spend some time to fix it on IntelliJ IDEA 8.1. You can find it at here

When I started this project, the laptop I used was not power enough to run tests in background without bothering my development, so I got a running test progress bar popup in Fireworks.

The hardware is faster and better now. The progress bar may bother someone now, I am removing it and making all tests running background, next version will come soon.

I think I can call back my dream: don't make me test. It's about running tests bothers me, and write script or AllTests.java for running all tests bothers me. So I need a tool to help me to run tests and keep eyes on my edited and failures tests. In ruby world, there is one called ZenTest gem is doing similar thing. And JUnitMax is on eclipse.

Another thing is considering:

What's you like when a tool running test in background for you?

1) paralleling all tests with high cpu cost but fast feedback

2) one slow process but wouldn't bother you

TeamCity with IntelliJ ? I just don't have resource. So it's out.

Clean your setup methods

2008-12-19T01:07:00.000-08:00

I don't think setup methods are evil(I agree most of the points discussed in the blog), but I do think:

It's not the right place to initialize test data. It is almost impossible that all your test methods would use all the same test data in the long term.
It's not the right place to initialize expectations of mocks. Do you ever put assertions in the setup method? Expectations of mocks are similar stuff with assertions.
Preparing test environment is right stuff put into setup method. Test environment is different with test data. Test environment is always same for all test methods, because a clean test environment is important for isolated test. For examples: preparing an empty messaging broker; preparing controller, request & response objects for rails functional test.
Use helper methods to organize your test domain. Even the code of preparing your test environment should be organized as proprietary helper with meaningful name, which is not only keeping your code readable & maintainable, but also keeping your setup methods simple. Mmm... I agree sometimes it's clean and clear you just put all your need in the setup methods.
Keeping setup method simple should always be considered when you are adding new code.

I cleaned a test setup method recently, the result I got was that the time of running the test was reduced from 200+ sec to 60+ sec. I felt good, and would keep doing it.

DTR 1.0.0 released: all about test grid

2008-10-26T05:35:00.000-07:00

DTR is not a CI tool. It was supposed to be a tool only take responsibility of running test in distributed process to fit with test framework and CI tools.
But it was so hard to build a test grid that DTR only did job of running test in distributed process. The hard part is synchronizing codebase within grid, which I thought Distributed SCM could help.

DTR is also supposed to be used in an internal network that you never need to care about the network performance when you are running test. My team member got problem with working at home for he need run a long time test suite to commit code. One idea for fixing this problem is to use a CI tool to act as him in the office, and what he need to do is push his code to the codebase monitored by the CI. Yes, a pre-commit build. This actually is good for whole team.

So I get impetus to improve DTR. And here is new release which provides new features for setting up more stable and maintainable test grid.

Release 1.0.0 Changes:
* support synchronizing codebase
* lookup agents by broadcast
* group agents for different project or environment usage
* no need launch dtr rinda server anymore
* agents log would be output into master process log

You can find more info at here: https://github.com/xli/dtr/tree

-------------
Our new test grid built on CC.rb + Git + DTR is working well for a while.
You may like to get a try, here is WPC's branch of CC.rb which fixed some Git plugin problems: http://github.com/wpc/cruisecontrol.rb/tree/master

It just need:

Make your project work with CC.rb + Git
Change your test task to be dtr test task. Of course, you need launch some DTR agent within your local network.
Create cc projects for your team members, one member one project.

-------------

What's next release

On Mingle project, we have script to make our test run in different process with different database, so that DTR doesn't handle running test in multi-processes with independency database for Rails project. I think it is a worth work.
You can see the progress here: http://github.com/xli/dtr/tree/v1.1.0

-

Introduce Hickey: prepare test data simple and fast by your Rails model

2008-10-05T07:46:00.000-07:00

Spending long time on preparing test data bothers me.
Spending long time to understand test data prepared for test bathers me.

Suppose we have the following models:

% class Project < ActiveRecord::Base
%   has_many :tags
%   has_many :cards
% end
%
% class Tag < ActiveRecord::Base
%   has_many :taggings, :class_name => '::Tagging'
%   belongs_to :project
% end
%
% class Tagging < ActiveRecord::Base
%   belongs_to :tag
%   belongs_to :taggable, :polymorphic => true
% end
%
% class Card < ActiveRecord::Base
%   has_many :taggings, :as => :taggable
%   belongs_to :project
% end

And we need prepare a domain for test:

One project has 2 cards
Card one is tagged as 'first_tag'
Card two is tagged as 'ignore me'

Here is my new thoughts:

% Hickey.dump :project => {
%   :identifier => 'hickey', :cards => [
%     {
%       :name => 'first card',
%       :taggings => [{:tag => {:name => 'first_tag'}}],
%     },
%     {
%       :name => 'dont make me think',
%       :taggings => [{:tag => {:name => 'ignore me'}}]
%     }
%   ]
% }

More details or if you want to give it a shot, checkout Hickey on github.

MM: Development context console integrating with Mingle

2008-09-25T08:59:00.001-07:00

MingleMingle project was deleted because of confusing name.
It is renamed as "MM" and moved to here on Github.
It still is an example project for integrating with Mingle by REST API in ruby.

MM is redesigned as a console instead of command line tool for my daily work to remember my development context.

The following is a feature list:

transition: run on card, card transition list
view card favorite
view card
simple todo tasks tool
card/view context
remember your context history
do job by selecting item from list
integrating with terminal console: run console command directly
commit code with running transition

GEM Installation

Run the following if you haven't already: gem sources -a http://gems.github.com

Install the gem: sudo gem install xli-mm

Usage

MM Console should be following your local project repository directory, so that every project has its own context. Go to your project directory, for example, my dtr project repository is at ~/workspace/dtr, type 'mm' in the directory to launch the console.

For integrating with Mingle, you need setup a variable in the MM console named 'site', the value should be your mingle project url with your login name and password, see the following example log how to setup it.

The following is an example log, after you installed mm gem and type 'mm' in your project directory:

>>>
mm, version 0.0.3
Type 'help' for usage
> > site='https://xli:mypassword@myproject.mycompany.com/projects/mingle'
https://xli:mypassword@myproject.mycompany.com/projects/mingle
mingle> > #5079
#5079 Allow admin to update transition only property values - but preserve the use of automated transitions on the grid view
mingle> story 5079> help

0) Basic Commands
1) Context
2) Integrating With Mingle
3) Quick Start
4) Transition
5) Variables

! > Type index number to select item from list.

mingle> story 5079> 0
- Runtime Variables: show existed runtime variables
How to set a variable? example: fixed = 'complete fix'
How to unset a variable? example: fixed = ''
MM stores most of stuff as variables so that you can easily access them.
- command keys or match pattern: variables, v

- Clean Cache: clean current cached resource objects: (no object cached)
- command keys or match pattern: clean_cache

- Tabs: show tabs in your Mingle project
- command keys or match pattern: tabs, t

- Help: show this doc
- command keys or match pattern: help

- To Do: manage your todo list based on context
- command keys or match pattern: todo( (.+))?

- Open: Open card, view or your context(card or view) in your default browser. This command sends 'open 'http:/project_url' to your Terminal, which may not work in Windows.
- command keys or match pattern: open( (.+))?

mingle> story 5079> 1
Context is something MM remembered for you. There are 2 kinds of context, one is card, another is view. Context will be displayed as prompt, so that you can know what's context you have currently. For example, if your context is a card, you can simply type in 'open' to open the card in your default browser.
mingle> story 5079> name
Performance Tuning: Transition execution
mingle> story 5079> tabs

0) 2.1 help updates
1) forum bugs
2) My Work
3) Technical Debt
4) QA Bugs to Close
5) 2.1 Testing
6) Dev Bugs to Fix

! > Type index number to select item from list.

mingle> story 5079> 2

0) #5022 Performance Tuning: Transition execution

! > Type index number to select item from list.

mingle> My Work> 0
#5022 Performance Tuning: Transition execution
mingle> task 5022> properties
---
:"task status": open
:priority: low
:release: "2.1"
:owner: 24
mingle> task 5022>
mingle> task 5022> history

0) story 5079
1) My Work
2) task 5022

! > Type index number to select item from list.

mingle> task 5022> 0
#5079 Allow admin to update transition only property values - but preserve the use of automated transitions on the grid view
mingle> story 5079> owner
Suzie
mingle> story 5079> transitions

0) add story to internal release theme
1) Complete Development
2) move to 200

! > Type index number to select item from list.

mingle> story 5079>

MM TODOS

xli:mm(master)$ mm
>>>
mm, version 0.0.3
Type 'help' for usage
> > todo

0) Start Work: output system cmd io directly instead of at the end of command

1) Start Work: find cards by mql

2) Start Work: todo command: delete all deletable option

3) Start Work: todo command: delete all option

4) Start Work: history object doesn't update bug

5) Complete: should raise error when commit with invalid transition script

6) Start Work: properties of card show id of user/card obj which should be name

7) Start Work: reload context command

8) Start Work: todo command: remove completed task after one day

9) Complete: why get story from server again?

10) Start Work: daily report?

11) Delete: card&view should use business id for comparition

12) Delete: output of open command is bad

13) Delete: output of transition completed is bad

14) Delete: better output of properties

! > Type index number to select item from list.

> >
----------------

By the way, MM is only tested on Mac OS.

Stand-up, Open your mouth

2008-09-18T18:56:00.000-07:00

Got a small tool called AntiRSI for forcing you take regular breaks by freezing your screen 13 seconds(as default) after 4 mins work and freezing 8 mins after 50 mins work.

It became to be fun when I paired with WPC. We decided to stand-up as quickly as possible once the notification popup.

Growing Test

2008-09-02T04:21:00.000-07:00

On my current project(Mingle), we need 9 mins to run pre-commit tests, which is a little bit long for you to keep eyes on the test running status. When I found a rails plugin "Growl Tests", I got it was something I need for my daily work that Growl notification could call me back if I was doing something else during test running. But I don't like install a rails plugin to involve Growl notification for my project, because it is only for developer to run test and I want keep the installer script clean. And also, I need custom notifications for that my first idea was only notifying test fault. So I started another simple project Growling Test.

Later, WPC added test finished notification for getting running test status back in TextMate, which made Growling Test more fun. I don't remember when it happened first time that TextMate didn't output test framework standard output while running test_unit test framework test, but we do painful for no test framework standard output while running tests in TextMate.

Growling Test has 3 notifications registered: test finished, test fault and all test finished. I put test finished notification on my laptop LCD, and put test fault and test finished notification on working monitor. Here is a growling notifications screenshot:

To involve Growling Test in your project, you need install Growling Test gem:
  Run the following if you haven't already:
    gem sources -a http://gems.github.com
  Install the gem:
    sudo gem install xli-growling_test

And then put the following code in your test_helper.rb:
  require 'rubygems'
  require 'growling_test'

Have fun.

Build Development Build Grid by DTR 0.0.3

2008-04-28T00:02:00.000-07:00

DTR 0.0.3 was released. It is a stable version for running a build grid.

The following is an example build grid shared within developers to run unit tests before commit code. We‘ll use Mercurial to synchronize code between Runner Agent machines and developer machine.

Background

Suppose I am a developer working on a rails project pet_store checked out at ’~/pet_store’. I have a MacBook called M1 and another 2 Mac-minis called M2 and M3. My plan is distributing tests running on M1 to M2 and M3 before commit code. M3 also will be DTR Server. All machines are accessable in local network and IPs are M1_IP, M2_IP and M3_IP. Mercurial and DTR are installed on all machines. And for running dtr runner agent in daemons mode, you need install gem daemons. The test environment will be running on Mysql. All machines are installed Mysql accessed by the following database.yml configuration:

development:
 database: pet_store_development
 adapter: mysql
 username: root
 password:
 host: localhost
test:
 database: pet_store_test
 adapter: mysql
 username: root
 password:
 host: localhost

DTR Server

Start DTR Server on M3: dtr -s -a M3_IP. Home directory would be fine.

Setup Mercurial repository on M1

Init Mercurial repository in ’~/pet_store’ directory: hg init
Edit .hgignore file to ignore files don‘t need while running tests. In this example, we will ignore files: db/schema.rb, .dtr_env_pstore, dtr_runners.pid and dtr_server*; and directories: public, tmp and log. Commit .hgignore and then commit code into Mercurial repository.
Start ‘hg serve’ in ’~/pet_store’, the default port would be 8000. We‘ll use the default port in this example.
We‘ll clone the repository to M2 and M3 later.

Build Script

Add an environment variable on all machines: export DEV_BUILD_SERVER=M3_IP
Add an environment variable on development machine M1: export DTR_MASTER_ENV=http: //M1_IP:8000/pet_store. The environment variable
DTR_MASTER_ENV would be copied to all DTR Runner Agent for sharing information between DTR Master machine and DTR Runner Agents.

Create a rake file at ’~/pet_store’ directory called: dtr_tasks.rake

DEV_BUILD_SERVER = ENV['DEV_BUILD_SERVER']
DEV_REPOSITORY_URL = ENV['DTR_MASTER_ENV']

task :dtr => ['hg:check', 'dtr:uf']

namespace :hg do
 task :reinit do
   Dir.glob("*").each do |f|
     #remove all files except files start with 'dtr_' including dtr runner pid file and current file dtr_tasks.rake
     next if f =~ /^dtr_/
     FileUtils.rm_rf(f)
   end
   FileUtils.rm_rf(".hg")
   FileUtils.rm_rf(".hgignore")
   puts %x[hg init]
   puts %x[hg pull -u -y #{DEV_REPOSITORY_URL}]
   puts %x[mkdir tmp]
   puts %x[mkdir log]
 end
 task :check do
   http = Net::HTTP.new("localhost", "8000")
   begin
     http.get("/")
   rescue Exception
     raise "Mercurial server is not running!"
   end

   raise "Please commit changes into Mercurial repository" if %x[hg st] =~ /.+/
 end
end

namespace :dtr do
 task :runner_agent do
   puts %x[dtr -R;dtr -r runner -a '#{DEV_BUILD_SERVER}' -i 'rake --rakefile dtr_tasks.rake hg:reinit;rake db:migrate db:test:prepare' -D]
 end

 require 'dtr'

 DTR.broadcast_list = [DEV_BUILD_SERVER]

 DTR::MPTask.new(:uf) do |t|
   t.libs << "test"
   t.test_files = FileList['test/unit/**/*test.rb', 'test/functional/**/*test.rb']
   t.processes = ENV['P'] || 1
   t.runner_options = "-a '#{DEV_BUILD_SERVER}' -i 'rake db:migrate db:test:prepare'"
   t.start_server = false
   t.verbose = false
 end
end

The script above can re-init Mercurial repository. Runner Agent will run the ‘hg:reinit’ task before start Runner(see the ‘runner_agent’ task in the script above), so that Runner Agent can support any developer‘s codebase to run tests.
Commit dtr_tasks.rake into Mercurial repository.

Runner Agent

Clone repository from M1 to M2 in home directory: hg clone http://M1_IP:8000/pet_store
Start runner agent in the directory ~/pet_store on M2: rake —rakefile dtr_tasks.rake dtr:runner_agent
Start runner agent on M3 same with above.

Run DTR Master on M1

Commit all changes into Mercurial
rake —rakefile dtr_tasks.rake dtr
After you started the DTR Master, you can
type the following to monitor the DTR
Server status: dtr -a $DEV_BUILD_SERVER -m
For output into console, open the debug option: dtr -a $DEV_BUILD_SERVER -d
-m

Notes

Only one developer can run tests in grid at same time.
You can build different grid for every developers, but it maybe too many runner agents need to maintain.
Runner Agent can run 2 or more runners, but you need more powerful db:test:prepare script to setup databases need and let every process run in different database. Here is one solution: 1. setup one database, then use database dump to quick clone more databases we need. 2. change database configuration to use an idle database when test start.