XML parsing in Java

JSON is much popular now, but occasionally it is still possible to come across XML API. I had such experience recently and I have to say that in 2016 it is much easier than some 10 years ago. I mostly use Jackson for JSON, so for me best way to use it is XmlMapper plugin. After that it is plain Jackson.
There is example:

XmlMapper mapper = new XmlMapper();
List rates = mapper.readValue(ratesString, List.class);

compile 'com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.6.3'
compile 'org.codehaus.woodstox:woodstox-core-asl:4.4.1'

Groovy framework for AWS lambda

I was playing with AWS lambda recently and I find it pretty exciting. It is cheap, does not require any maintenance and it is simple. The biggest problem I found so far, is lack of tooling. AWS has web UI which does what it supposed to do, but is far from pleasant. AWS command line tools and API are much better, but are AWS centric. So, I have created tool to work with AWS lambda. It uses some convention now, but can be adjusted to whatever feels right for specific situation, idea is that it is just one script with few lines of code. Basically, it does 2 things: uploads application and local testing. I have created small example that illustrates how and what it does.

Example is trivial application that increments number by clicking on button. It is SPA, with static client in S3 and two API functions add and get.

All lambda code is just single Groovy script. It consists of router function, dependencies and logic. Router function is what will be mapped to lambda. It is function that accepts map and returns map. Input map can be plain request JSON, or some additional parameters can be injected at API gateway. Even for this trivial case it was a good idea and I injected configuration parameters - AWS access and secret keys and region for DynamoDB access, so they are hidden at admin level and are environment specific. Simplest way to do routing is just by using some designated parameter, in this example, it is called "function". Doing it with real URI will need AWS configuration, and it is not very convenient and flexible.

In this simple example, router script does logic, but for more complex cases it is better to be just router and logic would be in dependencies.

During packaging Groovy script is compiled and packaged together with all dependencies into uberjar which is required by AWS. After that, it is uploaded via AWS API.

Development can be done and tested locally by using mock server. Mock server mimics AWS, parses input into JSON like lambda, injects parameters like API gateway, includes Access-Control-Allow-Origin and separates configuration via config.js, supports debug and reloading.

How to add configuration parameters to AWS lambda with API gateway

Unfortunately, there is no easy way to add configuration parameters to AWS lambdas, there is almost nothing in context that you can configure; there is description, but besides being ambiguous, it is not accessible from function itself. Fortunately, it is possible to set additional values from API gateway, even it is not straightforward.

When you have created API gateway mapping, go to the method screen in gateway console, there is screen with multiple boxes, one of which is called Integration Request, click it and there will be section called Mapping Templates. Add new mapping template for type application/x-www-form-urlencoded, save it, edit value and change from default Input passthrough to Mapping template, then you can transform your original request, by adding additional parameters, it uses velocity for transformation, but there is basic example:

{
  "my-new-configuration" : "my value",
  "body" : "$input.path('$')",
  "params" : "$input.params()",
  "ip" : "$context.identity.sourceIp",
  "user-agent" : "$context.identity.userAgent"
}

Besides possibility to add anything to lambda function it also has access to interesting request data like headers, ip, client type, user agent, country, etc. - all will be available in your map or pojo as parameter for lambda function. And do not forget to click Deploy API after making changes to make them live.

Ansible freezes during start

If Ansible stops for several minutes or so during start without any messages, check if it calls your inventory script, because it might query for additional options using --host parameter and if there are a lot of hosts, it may take some time.

It can be easily fixed by adding this to JSON root:

    "_meta" : {
       "hostvars" : {
       }
    }

More about it here.

JSON converter is not working in unit tests in Grails

If calling grails.converters.JSON from unit test throws error like:

java.lang.RuntimeException: org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: Unconvertable Object of class: my.blog.Post
	at org.codehaus.groovy.grails.web.converters.AbstractConverter.toString(AbstractConverter.java:111)
	at org.codehaus.groovy.runtime.InvokerHelper.format(InvokerHelper.java:616)
	at org.codehaus.groovy.runtime.InvokerHelper.format(InvokerHelper.java:559)

It is easy to fix this error by adding ControllerUnitTestMixin to the test definition like:

@TestMixin(ControllerUnitTestMixin)
@TestFor(BlogService)
class BlogServiceTests {
...

Disable pretty printing of JSON in Grails

It is easy to print any object as JSON in Grails

[key:'value'] as JSON

But it gives human readable, nice, multiline formatted version of JSON with indents. If that is not needed, it is easy to switch it off, like:

new JSON([key:'value']).toString(true)

Much uglier, but does the job.

Updating model elements of CanJS component scope

CanJS scope is simple map and when you need to retrieve new element from server the easiest way seems just to assign new model to property, like:

  scope.attr('todos', new Todo.List({date:val}));

This is easy and works, but not best solution, as values blink on screen. This is because initial list is empty and values are added after they are loaded, so more smooth and reactive (bazinga!) way to do it, will be by model's findAll method callback, like:

  Todo.findAll({date:val}, function(todos){ scope.attr('todos', todos) });

Additional benefit of this approach is that you can use new values for whatever you might need besides displaying them.

CanJS is not calling Mustache function when value is updated

Mustache can map not only specific values to template, but also results of the functions, which is sometimes handy. There is example.

But recently, when I was using this feature inside of CanJS template, it only rendered results for the first time and did nothing when values were changing. This was not working properly:

  can.Component.extend({
    tag: "sum",
    scope: {
      model: new can.Model({x: 2, y:3}),
      result: function() {
        return this.model.x * this.model.y;
      }
    },
    template: '<input can-value="model.x">*<input can-value="model.y">={{result}}'
  });

As I found out, problem was that it was missing proper value invocation and simply calling values via attr method fixes the problem.
This works fine:

  can.Component.extend({
    tag: "sum",
    scope: {
      model: new can.Model({x: 2, y:3}),
      result: function() {
        return this.model.attr('x') * this.model.attr('y');
      }
    },
    template: '<input can-value="model.x">*<input can-value="model.y">={{result}}'
  });

MeteorJS like application in Grails

MeteorJS is new web framework that combines server-side and client-side programming into one, leaving mostly just client-side. I am defenitely not expert with tool and just seen demo on their homepage, but WOW effect is sure impressive. It suppose to kill all Rails and server-side development in general. Thank god I am full-stack developer, but anyway I am scared.

General idea behind this framework is autogenerating all server-side code, leaving mostly just configuration. Similar feature was recently introduced in Grails, so I was wondering - is it possible to create something similar to MeteorJS in Grails. And there is my experience.

Basically, it needs autogenerated REST, Javascript ORM and server push. REST is Grails feature since version 2.3; for client-side ORM I picked CanJS as it looks most similar to Meteor; for server push in Grails I picked spring-websocket plugin (because it is first result in Google, why else?). Unfortunately, spring-websocket is M1 and it relies on Spring 4, so Grails is 2.4M1 too, so it is not production safe, but MeteorJS is 0.8.0 now too, so we are even.

Application idea is simple TODO app. You can find full source here. There is defenitely some boilerplate code as it is not plugin, but I think in general it is pretty similar to MeteorJS example. It consists of 3 meaningful components: view, controller and domain object.

View is Mustache template, and is pretty straightforward if you are familiar with CanJS:

<h1>test</h1>
<table>
{{#todos}}
  <tr>
    <td>{{description}}</td><td><input type="checkbox" can-value="done"></td>
    <td><input type="button" value="Delete" can-click="delete"></td>
    </tr>
{{/todos}}
</table>

Description
<input type="text" can-value="description">
<input type="button" can-click="add" value="Add">

Controller is CanJS component with ORM definition:

var Todo = can.Model({
  findAll: function(id)   {return $.get('/grails-meteor-example/todos.json/')},
  findOne: function(id)   {return $.get('/grails-meteor-example/todos.json/'+id)},
  create:  function(data) {return $.post('/grails-meteor-example/todos.json/', data)},
  update:  function(id)   {return $.ajax({type: 'PUT',    url: '/grails-meteor-example/todos.json/'+id })},
  destroy: function(id)   {return $.ajax({type: 'DELETE', url: '/grails-meteor-example/todos/'+id, contentType: 'application/json' })}
}, {});

can.Component.extend({
  tag: "todos",
  template: can.view("todos.mustache"),
  scope: {
    todos: new Todo.List({}),
    description: can.compute(''),
    add: function() {
      new Todo({'description':this.description(), 'done':false}).save();
      this.description('');
    },
    delete: function(todo) {
      todo.destroy();
    }
  },
  events: {
    "{todo} change": function(){
      this.scope.attr('todos', new Todo.List({}));
    }
  }
});

$(document).ready(function() {
  $("body").html( can.view.mustache("<todos></todos>"));
});

Domain is Grails domain object mapped as REST controller:

package org.grmeteor

import grails.rest.Resource

@Resource(uri='/todos')
class Todo {
  String description
  boolean done
}

View is totally similar to MeteorJS. Controller is similar, but also contains ORM definition and server push event. Both could be easily autogenerated with Grails plugin, but at least push can be also left for practical applications (probably, you will not want to have notification with millions of parallel users). Domain is pure Grails overhead, but it would probably still be needed for SQL backends in MeteorJS (and hey, those server-side devs want to eat too!).

Of course there are other files too, like server push sender and client side HTML container for CanJS, but these are mostly static and sure can be easily hidden by plugin. So only part that is missing to make Grails MeteorJS-cool is plugin itself, which even if not for practical cases could be nice to have for WOW effect and MVPs.

Grails REST redirect after DELETE with jQuery

If you do delete from AJAX request (directly or via framework), sometimes Grails REST controller redirects response to index method, which when done with DELETE method leads to 405 HTTP error.
This happens because of incorrect content type in request. To fix it, contentType needs to be explicitly set in request, like:

$.ajax({
    url: '/todos/'+id,
    type: 'DELETE',
    contentType: 'application/json'
  })

CanJS incorrectly places elements in table row

After rendering this HTML in CanJS, 'tests' elements got placed before table and not after previous cell as it is supposed to be:

<thead>
  <tr>
    <th>Test</th>
    <th><input type="button" value="Test"></th>
    {{#tests}}
      <th>{{name}}</th>
    {{/tests}}
  </tr>
</thead>

Problem was that input tag was not closed, which is usually fine with HTML, but confuses CanJS. After closing it, everything was working as expected.

Writing binary data between Java and Python

I have small project where server-side is done in Python and client is J2ME application and they have to talk to each other. At one time I had JSON to do it, it is quite easy on both sides, there are nice libs both for Java and Python. But recently, as traffic was growing and JSON performance on J2ME is not suitable for my tasks, I decided to try Java serialization with data streams.

For Java it is trivial:

DataOutputStream daos = new DataOutputStream(baos);
daos.writeLong(value1);
daos.writeInt(value2);
daos.writeBoolean(value3);
daos.writeUTF(value4);

In Python there is no such thing, but there is nice utility package called struct. It basically allows to read and write binary data and convert it into normal data. It is configurable and with small experimenting it is possible to find matching reading and writing functions for Java counterparts.

For example to parse example that is written above:

result = struct.unpack('>II', input1)
value1 = (result[0] << 32) + result[1]
result = struct.unpack('>i', input2)   // result[0] becomes value2
result = struct.unpack('>b', input3)   // result[0] is 1 for true and 0 for false

With strings it is a little bit trickier, because with method writeUTF, Java writes string length as 2 byte number and UTF-8 string. So in Python it is:

result = struct.unpack('>H', input4)
result = input5 // it is already acceptable for Python, just calculate result[0] bytes from input

Similar is also writing from Python to Java; same data formats are good for Java reading.

There is nice trick in Python, it allows to unpack and pack a lot of parameters simultaneously, for example this example above can be shortened to:

result = struct.unpack('>IIibH', input6)  // result array now contains all unpacked values except string 

JSON formatting

Sometimes it is needed to format JSON to check it's content. Intellij IDEA and Eclipse do the job pretty well, but if these are not available it is easily possible in Google Chrome. To do it:
1) Go to Control the current page menu (document icon to the left of address input) then Developer/JavaScript console.
2) Enter console.log({qwe:123}) (where {qwe:123} is just any JSON object).
3) You got nice JSON object tree as output.
There is some noise from Chrome, but it is possible to see and browse your object.
Same feature is in Firefox with Firebug plugin, even the same syntax.


UPDATED: Nice online JSON formatting tool: http://chris.photobooks.com/json/default.htm .

localStorage returns undefined value

When I have just started to work with localStoage without reading much about it I have created something like this

localStorage['key'] = 'value';
alert(localStorage['key']);

Result was great, and I got nice alert with 'value' on it.
Quite excited, i created functionality i wanted to have, created nice tests and found out that something is not great at all and after localizing error I have found out that similar piece of code is not working as good as previous.

localStorage['key'] = {test:'value'};

alert(localStorage['key'].test);

In this case, alert returned 'undefined', but not 'value', what was quite unexpected.
However this code

localStorage['key'] = {test:'value'};

alert(localStorage['key']);

says that I have nice '[object Object]'. I checked it a bit and had found that this script

localStorage['key'] = {test:'value'};

alert(typeof localStorage['key']);

says that my localStorage result is in fact 'string', which was quite unexpected, as it should be Object.
As it turned out, localStorage uses structured clone algorithm and whether because of this or for some other reason it converts my object into string, stores it as such and therefore looses all properties and values.
Finally, I have found easy way to solve this problem by converting my object into JSON string. Fortunately Firefox have native JSON coverter functions and everything can be done simply like

localStorage['key'] = JSON.stringify({test:'value'});
alert(JSON.parse(localStorage['key']).test);

In WRT there is no such nice lib, but it is possible to download it at json.org. And it can be nicely used like this:

widget.setPreferenceForKey(JSON.stringify({test:'value'}), 'key');

alert(eval(widget.preferenceForKey('key')).test);

There is small issue though. As you see JSON lib is used to convert object to string, but it was not able to convert string to object on my Nokia device. But this small eval hack still works great and I am getting my value as expected.