This blog is @deprecated

All entries have been moved to my site, so this blog will be shut down somewhere in the future. If you're interested, please visit the site dhiller.de.

Thanks for reading :)

Update all git repos via cmd line in a subdir

This one is just for me so I don't forget this "one-liner". Updates all git repositories in direct subdirectories. Works on OS X :)

cwd=$(pwd); for dir in $(find . -type d -not -name '\.*' -maxdepth 1 -print); (current_dir="$cwd/${dir##./}"; echo "Updating $current_dir"; cd $current_dir; git pull --all;)

Run multiple Intellij version installations on linux

I love Intellij very much. I like the jetbrains EAP program for trying out new features of the IDE. After being screwed a couple of times replacing a working installation by a faulty one, I elaborated a way of keeping several versions of Intellij in parallel.

As a side note: I'm reporting about a Linux Mint 18 install but I guess most other linux distributions will have minor adjustments to get this approach working.

First of all I like to keep all versions and needed files in a dedicated folder in /opt/IDEA. I have them symlinked so I can more easily switch versions:

/opt/IDEA ‹›
╰─$ ls -la    
insgesamt 24
drwxr-xr-x  6 dhiller dhiller 4096 Nov 17 11:17 .
drwxr-xr-x 13 root    root    4096 Nov 16 15:59 ..
lrwxrwxrwx  1 dhiller dhiller   13 Nov 14 17:20 current -> idea-2016.2.5
lrwxrwxrwx  1 dhiller dhiller   11 Nov  9 09:29 eap -> idea-2016.3
lrwxrwxrwx  1 dhiller dhiller   19 Okt 20 08:18 idea-2016.2.5 -> idea-IU-162.2228.15
lrwxrwxrwx  1 dhiller dhiller   19 Nov 17 11:09 idea-2016.3 -> idea-IU-163.7743.37
drwxr-xr-x  9 dhiller dhiller 4096 Okt 20 08:17 idea-IU-162.2228.15
drwxr-xr-x  9 dhiller dhiller 4096 Nov  9 09:29 idea-IU-163.7342.3
drwxr-xr-x  9 dhiller dhiller 4096 Nov 14 08:50 idea-IU-163.7743.17
drwxr-xr-x  9 dhiller dhiller 4096 Nov 17 11:09 idea-IU-163.7743.37
lrwxrwxrwx  1 root    root      37 Sep  7 11:56 idea.vmoptions -> /home/dhiller/bin/IDEA/idea.vmoptions
lrwxrwxrwx  1 root    root      43 Sep  7 11:56 run_idea_with_env.sh -> /home/dhiller/bin/IDEA/run_idea_with_env.sh

First of all I have a file where my Intellij options are stored:

File: /opt/IDEA/idea.vmoptions

-Xms1G
-Xmx2G
-XX:MaxPermSize=512m
-XX:ReservedCodeCacheSize=512m
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=50
-ea
-Dsun.io.useCanonCaches=false
-Djava.net.preferIPv4Stack=true
-XX:+HeapDumpOnOutOfMemoryError
-XX:-OmitStackTraceInFastThrow
-Dawt.useSystemAAFontSettings=on
-Dswing.aatext=true
-Dsun.java2d.xrender=true

Then I have a shell script:

File: run_idea_with_env.sh

#!/bin/bash

if [ ! -d "/opt/IDEA/$1" ]; then
        echo "Usage: $0 "
        exit 1
fi

# set environment variables
export JAVA_HOME="/usr/lib/jvm/default-java/"
export IDEA_VM_OPTIONS="/opt/IDEA/idea.vmoptions"

/opt/IDEA/$1/bin/idea.sh

Last but not least I have two or three desktop definitions to be able to start the IDE from the Menu in ~/.local/share/applications:

File: jetbrains-idea.desktop

#!/usr/bin/env xdg-open
[Desktop Entry]
Version=1.0
Type=Application
Name=IntelliJ IDEA
Icon=/opt/IDEA/current/bin/idea.png
Exec="/opt/IDEA/run_idea_with_env.sh" current
Comment=The Drive to Develop
Categories=Development;IDE;
Terminal=false
StartupWMClass=jetbrains-idea

File: jetbrains-idea-eap.desktop

#!/usr/bin/env xdg-open

[Desktop Entry]

Version=1.0

Type=Application

Name=IntelliJ IDEA EAP

Icon=/opt/IDEA/eap/bin/idea.png

Exec="/opt/IDEA/run_idea_with_env.sh" eap

Comment=The Drive to Develop

Categories=Development;IDE;EAP

Terminal=false

StartupWMClass=jetbrains-idea

The key point is the line starting with Exec that points to the version to start:

Exec="/opt/IDEA/run_idea_with_env.sh" current

But you just could start the IDE from a terminal with:

$ /opt/IDEA/run_idea_with_env.sh current

for starting the latest stable or

$ /opt/IDEA/run_idea_with_env.sh eap

for starting the latest EAP version,

When installing a new version I just download it, untar it into /opt/IDEA, change the symlink to point to the new version, and that's it.

Don't forget that thenThrow returns an OngoingStubbing instance *sigh*

I've often had the problem that when checking for a working retry mechanism (btw. spring-retry works great) I wanted to first throw an exception and then return a value or whatever else is necessary.

Easy peasy, I thought. Just create an Answer and hold state within the answer i.e. a boolean that gets set to true after it has been called:

    @Test
    public void queryIsRetried() {
        final Answer

And of course there is a much easier way, stupid me:


    @Test
    public void queryIsRetried() {
        when(lexdbJdbcTemplate.query(anyString(), any(RowMapper.class)))
                .thenThrow(new RuntimeException())
                .thenReturn(null);
        underTest.query("SELECT 1;");
    }

Git aliases, the easy way

After wrestling with the global .gitconfig file for installing aliases like a shortcut for status using vi, I thought there must be an easier way, and there is (as always when using git):

Create an alias for a compressed log with decorations and graph:

git config --global alias.l 'log --format=oneline --graph --decorate'

So instead of typing 

git log --format=oneline --graph --decorate

then you can just enter 

git l

and you will get a onelined history output complete with ascii visual graph and annotations for where every branch pointer is ATM :)

> git l

* 87e02884a6d45e0d003ffcdccf41098716ca876e (HEAD -> master) Moved files to folder

* 9d1c1dee120e5a189f562c2fc5d12709ae35102d (origin/master) Latest version

* 7b34163291fda0b82bf4777d7b9c24a22df9338e Corrected special characters

* 838ccffe24c7356f52063fc0b6e6c638dcccbd5a Removed lock file

* 74fff37b8ee77113b0702baeb0f9b85f17e5f7e9 Added gitignore

* 4b87667f117412034e977f61de8839b251b5ff81 Added conversion script

* 291eed8a5f44520e8ca706bc6fc7286978fc821c Initial commit

Source: https://git-scm.com/book/tr/v2/Git-Basics-Git-Aliases

Useful git aliases when working with Maven and Jira

Put these into your .gitconfig below the alias section:

# Return all jira issue keys from commit messages

issues = !sh -c 'git log --oneline $@ | egrep -o [A-Z]+-[0-9]+ | sort -V | uniq' -

# Fetch id of latest commit from maven release plugin

release-commit = !sh -c 'git log --all --grep=maven-release-plugin.*prepare.release --format=%H| head -1'

# Return all jira issue keys since last usage of maven release plugin

issues-since-release = !sh -c 'git issues $(git release-commit)..'

Using Infinitest for real

Making your Test driven development faster in real projects

When you are practicing TDD like I do you may blindly know your keyboard shortcut to start your test runs after you've changed something. Or do you still use your mouse? :-)

Infinitest is a great tool to get rid of those shortcuts. Whenever a change occurs in a class it runs the tests that accompany it. Well, that's already awesome but fasten your seatbelt: Infinitest also runs tests of classes that are affected by your change.

How they do this magic I haven't elaborated but I can assure you of the effect: When using Infinitest you will notice net effects of your changes much faster.

But...

I always loved Infinitest for it's incredibly short feedback loop, yet disabled it in my current maven multi module project for seemingly obvious reasons: "No, I am too lazy don't have time to figure out how to properly set it up" (That of course I would not confess to myself.)

This meant I didn't figure out how to configure it so it wouldn't always mess up my own dev environment or, being quite more of an impact, the integration test environment.

As you might guess I have successfully set up Infinitest for this monster of a project.

The project

The Project is a maven multi module project with a dozen modules and specifically a module containing most of the integration tests. It is completely stored in a remote git repository. I have set up a local virtual machine containing a database for testing and a local memcached. The project is configured via a set of -D properties that specify which of these to use during integration tests, so that these can be run locally or on the dedicated test instances running on the integration server during the Jenkins CI build. The CI build runs on every push to the server repository.

And here comes the problem: Whenever a local integration test fails to run on the local database engine, but uses the integration instance instead, it may interfere with a running CI build. 

For a thing like Infinitest, which runs a couple of tests on each change, the test runs (including the integration tests) may happen quite often. So I have to make sure Infinitest propagates the properties of i.e. where to find the database to the test cases.

The easiest solution (read: solution of the lazy one) is of course: switch it off.

Beware: configuring Infinitest for such a project requires a bit of work if you don't have it enabled directly after you started. But it can be done and is well worth the effort.

There are a couple of constraints you have to consider when using Infinitest:

1. When you are working in Eclipse you can not switch it on or off on a per project basis
2. Settings are configured using files
3. You have to have a set of files for each project
4. Test cases may be filtered by
1. Using regular expressions and/or
2. Using TestNG groups
5. Arguments are passed as -D properties in a files

See the Infinitest user guide for more information.

Preparations

The best way is: Have your integration tests in a single module. If you're using TestNG, use a group attribute in the test annotation. If you can't do that, move them into a separate package. If you can't do that and are not using TestNG, use a naming convention for your integration tests so you can easily filter them out. If you can't do that, you're screwed (well, I'm open for suggestions :-).

Setting it up

At first create a file called infinitest.args that contains all properties that have to be propagated to the test cases. Then create a file infinitest.filters that contains all filters for your integration tests (i.e. TestNG group or package regular expression or regular expression for the test case name. These files I'll call the master files for reference later on.

Put these two files into the top level maven project folder. Then make soft links in each module to these files. You can't do that? Well, you are on your own now ;-)

Hint: A very nice property of git is that it stores symbolic links in the repository. So you will share the links with the other project participants.

Don't put your master files into version control, or others will share the settings, which you probably don't want. But maybe you don't care whether they're using your database, do you? So you just put them into the .gitignore file.

Remember to not ignore the links in the module directories, of course.

So you should be fine right now. Just check the "Continuously test" option and enjoy the short feedback cycle from now on :-)

Hint: A slight modification which you may consider: For each project where you don't want to execute any tests just put an infinitest.filters with content .* instead of the symbolic link into the module directory.

So I hope this short guide helps you to get faster in test driven development and getting rid of manually executing tests via the mouse or a keyboard shortcut or whatever else you use. Please let me know if you have any corrections or suggestions.

My final notice: if you want to get really fast, just start using Infinitest whenever you start a new project. It's worth it!

Thanks for reading, anyway :-)

Rules for test driven bug fixing

Just a quick note to myself for how to do bug fixing:

1. Write an integration test that reproduces the bug
   This should create a test failure which will guard against regressions later on
2. Write integration tests for the other cases
   You should write at least one test. This should prove that it works the other way so you'll feel safe you don't break behavior later on
3. Track down the problem to unit level
   Debug to the lowest level where you can find the real cause of the problem
4. Write a unit test that reproduces the bug
   This should create a test failure which will guard against regressions later on
5. Write unit tests for the other cases
   You should write at least one test. There may be a chance that other corner cases have slipped as well, so examine closely. Keep in mind that unit tests are cheap!
6. Now, fix the bug
   Change it so all unit tests are green
7. Clean up
   Tidy up, don't leave a mess.
   Remember: Someone certainly will have to come back here. If that one is you, you will spend much more time, cursing the one who left that mess than if you do it right now!
8. Run the unit tests again
   Verify your clean up didn't break anything
9. Run the integration tests
   These should be green by now
10. Integrate changes
11. Create release

Reasons For Developing Without Tests

I've come to write this short article because I had some discussion with a developer about why he does not test. This discussion made me remember that many teams either don't do testing, neither on unit nor on integration level, or do it very sparsely. So I've come to write down a list of reasons that I've encountered and some questions that anyone hiding behind those reasons should have answers for.

Before you go right on to flaming against automated tests, please bear this in mind:

A script or program

• works in no more time than required
• when repeated does exactly the same

A human

• may be distracted
• interprets, thus may not follow the test script exactly
• takes shortcuts because he or she is bored of repetitive tasks

So here they are:

I don't have time for this

But you take the time for testing every change you make by hand each time, via the debugger or through the user interface? Or you don't and hope it will work? You want to continue doing a job manually that could be automated?
Don't you have some more important work to do?

I don't know the testing framework/tools

Learn them. The basic ramp up time for creating a unit test is very small. Every major IDE or build tool has support for testing.

Later on, when you start to integrate your tests into the build process, you will find out that Ant and maven both support JUnit out of the box.

I don't know if the boss will allow it

Ask him. Maybe he's glad that you asked and takes this as a sign of you taking responsibility for what you do.

I know the boss won't allow it

So you get paid to do a job that a script could also do, in less time and with fewer errors. Do you think he will like that when he discovers that?

I fear it will take too long

Try it in small steps. Consider that if you do nothing about it, it will take much longer in the long term.

I don't know where to start

Just start where you are, right now. The worst thing you can do is not do it, because it won't get better.

You will see, it's not as hard as you thought.

Are there any other reasons you know? Please provide them, I will see if I find an answer that convinces you.

Formatting curl xml result via xmllint

So I got fed up recently when reloading a url to check an xml result using chrome because I was using HTTP basic authentication also. Chrome was annoying me because each time i switched from no auth to auth I had to manually remove all browser data and restart chrome.

So I tried to use curl for that job. The only annoyance was that the xml is not formatted properly.

So xmllint for the rescue

curl 'yoururlhere' | xmllint --format -

and that's it.

Warum ich TDD liebe

Ich liebe TDD. Nicht deswegen, weil es so ein toller Hype ist, oder weil mich die Theorie vollkommen überzeugt hat. Sondern deswegen, weil es mir schon oft den Hintern gerettet hat.

Gestern Abend hatte sich herausgestellt, das ich es mal wieder versaut hatte. Das kommt nicht mehr so häufig vor, wie früher, aber es kommt immer noch vor. Nicht, dass man mir deswegen einen Vorwurf gemacht hätte. Wir sind alle Menschen, und Menschen machen nun mal auch Fehler.

Das eigentliche Problem war, das ich die Änderung des Verhaltens nicht in allen betroffenen Problemfällen getestet hatte. Aber ich fange einfach mal anders an:

Es war Donnerstag spätnachmittags. Nachdem ich Erfolg bei der Implementierung der Erweiterung gemeldet hatte, war die neue Version auf dem Testssystem installiert worden. Nun kam es beim Testen zu Laufzeitfehlern, unter Anderem zu meinem Lieblingsfehler, der NullPointerException. (Thomas, Du erinnerst Dich? :-)

Mir fiel auf, dass ich einen Fall, der im Betrieb sehr häufig vorkommen würde, einfach vergessen hatte, zu testen. Genauer hatte ich keine Tests für die entsprechenden Konstellationen geschrieben, und deshalb waren die Probleme nicht aufgefallen. Ich hatte mich in der Sicherheit der Tests (insgesamt allein ca. 140 Integrationstests) gewogen und fiel aufgrund der Fehlermeldung aus allen Wolken.

Nun war es bereits Donnerstag spätnachmittags, kurz vor Feierabend, und das Feature sollte am Freitag demonstriert werden. Die ersten Perlen des Angstschweißes bildeten sich auf meiner Stirn. Natürlich nicht wirklich, dazu neige ich nicht, aber ich merkte, das ich ein ernsthaftes Problem hatte.

Nun tat ich, was zunächst jeder tut, ich bekam Panik, und versuchte, auf Biegen und Brechen das Verhalten zu ändern. Aber immerhin hatte ich vorher noch etwas sehr Vernünftiges getan, wie sich später herausstellen sollte: Ich hatte Testfälle erstellt, die das bestehende Problem belegten.

Das genaue Problem war ein Fehler in einem Datencontainer, der, um die Daten später schneller wiederfinden zu können, auf dem Datentyp Short als Schlüssel basierte. Durch die Erweiterung konnte aber nicht mehr Short als Schlüssel benutzt werden, sondern es musste String-basiert sein.

Wie man sich vorstellen kann, ist das Ändern eines öffentlichen Interfaces, das diverse Zugriffsmethoden per Schlüssel besitzt, keine leichte Aufgabe, da ja auch andere Klassen als Client diese Klasse benutzen und es beim Durchführen der nötigen Änderungen dadurch zu diversen Compile-Fehlern kommt.

Wie gesagt, tat ich, was ich früher in solchen Fällen getan hatte. Aufgrund meiner aufkommenden Panik versuchte ich, mit dem brutalen Ansatz „Suchen-Ersetzen“ dem Problem der Interface-Änderung beizukommen, was sich natürlich aufgrund des Maßstabs der Änderungen als unmöglich herausstellte.

Nach zwei erfolglosen Versuchen und circa eineinhalb Stunden unnütz investierter Zeit gab ich kurz auf. Der Vorteil, wenn es spät ist, ist: man ist allein im Büro und kann in Ruhe nachdenken. Das tat ich, und beschloss einen neuen Schlachtplan:

• Compile-Fehler außerhalb der zu ändernden Klasse und der Testklasse ignorieren
• Nur kleine Iterationen
• Sobald die Klasse und die Testklasse selbst wieder kompilierbar sind, sofort alle diese Tests ausführen, mit anderen Worten : Red – Green – Refactor

Nach einer halben Stunde hatte ich die Änderungen an dieser einen Klasse und ihrer Testklasse durchgeführt. Danach machte ich mich daran, zunächst die Compile-Fehler in den benutzenden Klassen zu beheben, und als alles wieder kompilierbar war, checkte ich ein.

Dann führte ich die Integrationstests aus und stellte erfreulicherweise fest, dass „nur“ circa 20 Test fehlschlugen.

Als ich dann letztendlich alle Fehler behoben hatte, und alle Tests wieder grün waren, waren insgesamt vier Stunden vergangen. Wobei ich zugeben muss, dass ungefähr die Hälfte der Zeit unnötig war, wenn ich Ruhe bewahrt hätte und wie immer vorgegangen wäre.

Nachdem ich eine Nacht über den Vorfall geschlafen habe, kann ich drei Schlüsse ziehen:

1. Wenn man unter Stress gerät, neigt man dazu, in ursprüngliche (und in diesem Fall schlechte) Vorgehensweisen zurückzukehren.
2. Ohne die vorhandenen Tests wäre ich nicht in der Lage gewesen, so vorzugehen, wie es letztendlich nötig war, um zum Erfolg zu gelangen.
3. Den Vorteil von TDD sieht man erst wirklich ein, wenn man während der Entwicklung in ernsthafte Probleme gerät.

Übrigens: Wäre dies zwei Tage eher passiert, hätte ich eine hervorragende Geschichte für den Vortrag vom 19.01. gehabt, aber dies ist halt keine konstruierte Geschichte, sondern wirklich passiert. Dann hätte ich aber wahrscheinlich auch den Vortrag nicht gehalten.

Modal JDialogs

Recently our customers experienced problems running our software after updating to JRE6U12. In detail on Windows XP the desktop application would hang directly after connecting to the database. Downgrading the RE to U11 fixed the problem so we first thought it was a bug in the JRE.

Since google investigation did not bring any results except this we were convinced it should be our fault. After several hours of checking swing calls outside of the evnt dispatch thread, checking for undisposed dialogs and hunting down modal dialogs we found the following two statements.

dialog.setVisible(true);
dialog.setModal(true);

Do you see the problem? Neither did we at first. And since the code had worked before we did not think the order of these two statements would be the problem.

In fact it does not make much sense to set a dialog to modal after making it visible, but still these two lines were the cause of the main window losing focus, not reacting to input events any more, and for the user being unable to reactivate the main window. To the user this looks like a freeze of the application.

So now, after several hours of debugging, we know that we must call setVisible(true) after setModal(true)!

NetTool, a great HTTP request handling tool

After being annoyed of the OS X font bug from http4e I searched for a free http request analysis tool, where you can freely manipulate the request. I needed this as I'm currently developing a RESTful webservice which delivers XML and json to clients. The clients had problems in some browsers to get the ajax requests done. So I wanted to check if the right content type and charset was returned if any Accept headers had beend set on the request.

I stumbled upon NetTool and this worked like a breeze. There are only two caveats:

• The control characters don't match their OS X counterparts, so I have to use CTRL+C instead of COMMAND+C to copy values from the text fields


• The tunnelling from port 80 does only work as sudo due to user access rights (of course that's the right thing to do, but coming from windows originally I stepped into the trap at first)

Using JTidy to clean up non valid html pages

A couple of days ago I tried to modify a html page using a DOM. When I tried to convert the page into a Document instance the parser threw some SAXExceptions, complaining about the structure of the document, sort of "this tag needs to be closed" and the like.
The source was html output generated from Docbook. I neither had the time nor the intent to mess around cleaning up the generated html, but could remember there was something called Tidy.
So I searched for a Java library, and there it was. JTidy, looking like an unmaintained project, but being the right tool to clean up a html page and transform it into valid xhtml.
The API is pretty straight forward.

This is the implementation for converting (non-valid) html to a Document instance:

// Create instance
final Tidy tidy = new Tidy();

// Remove presentational clutter (don't really know
// what exactly that does, but sounds great ;-)
tidy.setMakeClean( true );

// Use XHTML output
tidy.setXHTML( true );

// Make document readable by indenting the elements
tidy.setSmartIndent( true );

// The html document received by a get request
final String s = ...;

// Converting the page into a Document instance
final Document document = tidy.parseDOM( new ByteArrayInputStream( s.getBytes() ) , null );

That's it, by now you have your html as a Document instance that you can freely manipulate.
The only thing I noticed was that the method node.setTextContent() does not work. But you can use node.appendChild( document.createTextNode( ... ) ), that does what you want.

The second part is about writing your Document to a string:

// Create a stream to write the output to
final ByteArrayOutputStream outStr2 = new ByteArrayOutputStream();

// Write modified Document to an output stream
tidy.pprint( document , outStr2 );

// Create a StringBuilder
final StringBuilder builder = new StringBuilder();

// Write output stream content to string builder
builder.append( new String( outStr2.toByteArray() , "UTF-8" ) );

// Create String
final String validXHTML = builder.toString();

At the end of the block you have your valid XHTML in a String.