Tuesday, December 18, 2012

Adding Vert.x includes to Confluence pages

Recently I wanted to include small dynamic page into Confluence site.
I created all logic in Vert.x and it worked fine in browser, but when I tried to add it via HTML Include macro in Conflunce, it gave this error:


Error rendering macro 'html-include' : null

After some investigation I found that this happens because Vert.x does not provide any proper content type header in responses and it have to be done manually, browser ignores it and treats page as HTML by default, but Confluence throws error. So it was easy to fix by adding content type to response, like:


request.response.putHeader("Content-Type", "text/html")


Thursday, November 29, 2012

Running several Grails projects in parallel

If it is needed to run several Grails commands simultaneously (for example on Jenkins) there is big change to get exception like:



19:52:24    [groovyc] org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
19:52:24    [groovyc] /var/lib/jenkins/jobs/test/workspace/src/groovy/com/test/SomeUserDetails.groovy: 3: unable to resolve class org.codehaus.groovy.grails.plugins.springsecurity.GrailsUser


This happens because Grails stores plugin classes and other files in user home directory, so for example, parallel task cleans workspace while other task is running, latter is in trouble of not finding some plugin stuff.
Fortunately, this is easy to solve by specifying working directory for every specific command. There is special property called grails.project.work.dir which can be specified on command line, for example like:



grails -Dgrails.project.work.dir=/home/jenkins/job1/ clean --non-interactive



Wednesday, October 24, 2012

List in groovy-wslite request

Groovy-wslite is currently recommended way for doing client calls to web services from groovy. It has very interesting format for describing requests, unfortunately it is not very well documented, and for me it was totally unclear how to send lists of objects to service. Fortunately, source code is available and is easy to dig if you have some time.
Basically, all you need to do is just send a bunch of regular objects, and they will be combined in list on the other end. There is example:

def response = client.send(SOAPAction:'https://server/services/service') {
  body {
    create() {
      dto {
        name('name123')
        id('123456')
        child {
              name('child1')
              birthday('2011-01-01')
            }
        child {
              name('child2')
              birthday('2012-01-01')
            }
      }
    }
  }
}


NullPointerException when saving Grails object

Recently I got very strange NPE while saving updated entity in Grails application:

java.lang.NullPointerException
 at com.test.MyController$_closure5.doCall(MyController.groovy:69)
 at com.test.MyController$_closure5.doCall(MyController.groovy)
 at net.bull.javamelody.JspWrapper.invoke(JspWrapper.java:149)
 at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:259)
 at $Proxy49.forward(Unknown Source)
 at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
 at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
 at java.lang.Thread.run(Thread.java:662)

All I was doing on line 69 was just calling save() method, object obviously was not null, all fields and dependencies too. Even more - saving another object from same method was perfectly fine, but not one particular.
Finally, I got the reason - object had null in it's version field. Obviously migration was not done properly or it was manually created object in bad schema, but setting version solved my problem easily, but bad error reporting - didn't helped.

Thursday, October 4, 2012

Reusing methods from different hierarchies with Mixins

Grails application can be tested with many different tools: Geb, Selenium, Grails own integration testing framework, etc. To use them, you have to extend some basic test case class, like GebReportingSpec, GroovyTestCase, GrailsUnitTestCase.
Usually there is a lot of methods that are used from all these frameworks, for example for data initialization and manipulation outside of test scenario scope. So how it would be possible to reuse these methods without access to class hierarchy and avoid code duplication. One easy way is to use Groovy Mixins, that allow to add new methods to any class without affecting existing class hierarchy. For example:


@Mixin(GenericTestMethods)
class SpecificTests extends GroovyTestCase {
...


Monday, October 1, 2012

Groovy tricks: initializing map with array values by default

I feel a little Mr. Haki today, so will post Groovy recipe.
Very often, there is case when you want to aggregate some values by single key, but groupBy is not suitable or flexible enough. In this case, with every new key, you should initialize this new value with array, like:


Map values = [:]
[[key:'user1', val:123],[key:'user1', val:567],[key:'user2', val:999]].each {
  def var = values[it.key]
  if (!var) {
    var = []
    values[it.key] = var
  }
  var << it.val
}
println values

[user1:[123, 567], user2:[999]]

This is nice, but "if" statement looks redundant and ugly and complicates things. Fortunately, there is great Groovy method called "withDefault" which does pretty much same things, but in much groovier way: if there is no key, instead of null it assigns specified value to your key. So above example can be rewritten like:


Map values = [:].withDefault {k->[]}
[[key:'user1', val:123],[key:'user1', val:567],[key:'user2', val:999]].each {
  values[it.key] << it.val
}
println values
[user1:[123, 567], user2:[999]]

Friday, September 28, 2012

Using Spring Security with CXF in Grails

CXF plugin is super easy way to add SOAP web service to Grails application. It creates regular service class, with few specific parameters and voila - you have SOAP. But what if you want security?
REST web services are native citizens in Grails - they are just actions in controllers, so you can use Spring Security annotations to check permissions. But there is no annotations for services. There are two easy options.
First, you can restrict actions by defining static rules. This is very simple, but not very flexible as it does not allow to configure permissions on method level, you will have to create different services for different permissions. Also, old clients may not work if WSDL requires authentication.
Second method, is to check permissions manually. In practice this is similar to annotations, and can be converted to annotations easily. There is example:


  Secret secret(int id) {
    checkRights("VIEW_SECRET")
    return Secret.get(id)
  }

  void checkRights(String rights) {
    if (!SpringSecurityUtils.ifAllGranted(rights)) {
      throw new IllegalAccessException("You don't have permission")
    }
  }

Finding similar strings in Java

Finding equal strings in Java is very easy, as well as finding substrings. But what if you have strings that are almost similar (at least from human point of view). For example: "John Doe" and "Doo Jonh".
Recently, I had to do data migration for table with very poor data quality. There is a lot of theory and books on finding duplicates, but surprisingly, there is very little actual Java implementations that could help with such things. Fortunately, there is at least one good - Simmetrics, there is even example how to use it, but it is as simple as one can imagine:


import uk.ac.shef.wit.simmetrics.similaritymetrics.*;

AbstractStringMetric metric = new MongeElkan()
println ( ['John John', 'Doo Jonh', 'ddddddd', '7777777777'].max { metric.getSimilarity('John Doe', it) })

Doo Jonh

There is a bunch of different algorithms one can use, but I found MongeElkan to be best for my case.

Tuesday, September 18, 2012

Javamelody SQL monitoring with Grails plugin

Javamelody is amazing tool for live monitoring. It has great Grails plugin which makes it very easy to use. Only issue I had with it is SQL monitoring configuration.
According to documentation there are several ways to configure data sources:
  1. magic - you don't have to do anything, it just works
  2. javamelody.datasources property in GrailsMelodyConfig.groovy - you just specify datasources
  3. replacing JDBC driver with Javamelody proxy
  4. specifying java option - javamelody.datasources
Unfortunately, all internal configuration options (1, 2, 3) didn't worked. As I investigated (quickly) it looks like there is no dataSource bean available for BeanPostProcessor and it tries to proxy after all connections were already picked by application.
So only specifying java command line option worked for me. Otherwise - great tool. It is also possible to specify multiple data sources with command line like:


-Djavamelody.datasources=java:comp/env/ds1,java:comp/env/ds2

Monday, August 20, 2012

Getting object from calling method in Java

This is hardly production scenario, but at some point I wanted to find value of a variable from a caller class. Looks like this is not trivial task and is much more complicated than it should be (for some reason), however still possible.
Finding method and class name is easy via Thread class, but accessing real objects seems to be only possible with Java Debug Interface. For this purpose, application must be executed in debug mode (for grails it is with grails-debug command). There is no need for any additional library as everything is included into JDK and probably is used by JDK debugger.
General scenario is as folows:
  • connect with debugger provided with JDI via socket (there are other options, but sockets seems default for most tools)
  • spawn new thread where you will wait to do whatever you want to do with data on stack trace
  • suspend application at point where you want to do it
  • at the end of your work - continue application
There is example:


def att = com.sun.jdi.Bootstrap.virtualMachineManager().attachingConnectors().find{it instanceof com.sun.tools.jdi.SocketAttachingConnector}
Map arguments = att.defaultArguments()
com.sun.jdi.connect.Connector.Argument hostArg = arguments.get("hostname").setValue("localhost")
com.sun.jdi.connect.Connector.Argument portArg = arguments.get("port").setValue("5005")
com.sun.jdi.VirtualMachine vm = att.attach(arguments);
def thread = vm.allThreads().find {it.name()=='main'}
def runnable = new Runnable() {
  void run() {
 while (!thread.isSuspended() ) {
   Thread.sleep(100)
 }
 def frames = thread.frames().findAll {it.location().toString() =~ /grails.plugin.spock.test.listener.PerSpecRunListener:.*/}
 frames.each {frame->
   def res = frame.thisObject()
   def field = res.type().fieldByName("failureCount")
   if (field) {
       def countref = res.getValue(field).toString()
        ...
      }
   }
     thread.resume()
  }
}
new Thread(runnable).start()
thread.suspend() 



Monday, June 18, 2012

IntelliJ IDEA Select In view with specific view

Intellij IDEA has a lot of great shortcuts, but for some reason there is no special shortcut when you need to find file in specific view, for example Grails view. Normal way to do it, is to press shortcut for "Select In..." (Alt+F1) and select your view (or press number with this view). This is basically 2 shortcuts and not too convenient.
Fortunately it is possible to make it easier by recording a Macro, and assigning it new shortcut. Just go to Edit > Macros > Start Macro recording, press "Select In..." shortcut and select your view.

Thursday, May 31, 2012

Downloading URL with login and password

In javadoc there is no clear way on how to download page that requires Basic Authentication, for example like:


  def result = new URL("http://www.requireslogin.com/").getText()


Fortunately, this is still very possible and not too complicated, just add this somewhere before calling URL:


  java.net.Authenticator.setDefault(new java.net.Authenticator() {
    protected java.net.PasswordAuthentication getPasswordAuthentication() {
      return new java.net.PasswordAuthentication("user", "password".toCharArray());
    }
  });


Possibly it would be better to have login and password as parameters, but this still works.

Saturday, May 19, 2012

Adding Ant optional task to Grails

Since version 2 Grails does not include Sshexec with built-in Gant libs, so it broke some of my deployment scripts. Fortunately it is easy to fix with nothing more than few Grapes annotations added to your Gant script:


...

@Grapes([
  @Grab(group ="org.apache.ant", module = "ant-jsch", version ="1.8.1"),
  @Grab(group ="com.jcraft", module = "jsch", version ="0.1.48"),
  @GrabConfig(systemClassLoader = true)
])

...

Possibly, there are other options like adding libs to some magic folders, but I found this one best as it avoids additional configuration, is self explanatory and hard to skip or forget as it is in script itself.

Thursday, April 26, 2012

Resolve JIRA issue after Jenkins build using API

Jenkins and JIRA integration is quite poor - there are few plugins, but they are quite far from being practical. Even such simple action like issue transition after build is missing. It is generally possible by using smart comments, but they trigger on commit, not after successful build.
Fortunately, there is a lot of customization options for Jenkins, and Jira has great API.
So, to implement issue transition it is good to have Groovy plugin and Jira Issue Updater plugin (for Jira libs). Then it is needed to add new Build step - Execute system Groovy script; and add this simple Groovy script there:


import hudson.model.*
import com.atlassian.jira.rpc.soap.client.*

def thr = Thread.currentThread()
def build = thr?.executable

def rss = new XmlParser().parseText(  new File("${build.getEnvVars()['WORKSPACE']}/../builds/${build.getEnvVars()['BUILD_ID']}/changelog.xml").getText()  )
def ids = []
rss.logentry.each {
   def match =  it.msg.text() =~ /(\w+-\d+)/
   match.each {
       ids << it[0]
   }
}

JiraSoapServiceServiceLocator jiraSoapServiceLocator = new JiraSoapServiceServiceLocator();
def jiraSoapService = jiraSoapServiceLocator.getJirasoapserviceV2(new URL('http://jira.xxxxx.com/rpc/soap/jirasoapservice-v2'));
String token = jiraSoapService.login('login', 'password');
def issues = jiraSoapService.getIssuesFromJqlSearch(token, "project=PROJECT1 and issuekey in (${ids.join(', ')})", 20)
String[] resolutions = ["23"]
for (RemoteIssue issue : issues) {
    def actions = jiraSoapService.getAvailableActions(token, issue.getKey() );
    for (RemoteNamedObject action : actions) {
         if (action.getName().equalsIgnoreCase('Resolve')) {
               RemoteFieldValue[] vals = [  new RemoteFieldValue("resolution", resolutions  )  ]
               jiraSoapService.progressWorkflowAction(token, issue.getKey(), action.getId(), vals );
         }
    }
}



This basically looks for all Jira IDs in commit comments, then queries specific project for all issues with these IDs and sets status to Resolved if it is applicable.
It is easy to customize this script to add special keys, like to ignore some commits or something.

Thursday, April 19, 2012

Groovy Eval slow performance

Groovy Eval is nice feature, unfortunately it does not have great performance. Recently, I had to calculate values on a bunch of objects and got quite dramatic performance issue. After quick investigation, I pinpointed problem to be with Eval, which can be summarized to:


def t = System.currentTimeMillis()
1000.times {
  Eval.x(44, 'x>3')
}
println (System.currentTimeMillis() - t)
7281
I tried to check if there is other solution that could do the same and I found that there is similar native Java function. In similar test I got:


import javax.script.ScriptEngine
import javax.script.ScriptEngineManager
import javax.script.SimpleBindings

def t = System.currentTimeMillis()
ScriptEngine engine = new ScriptEngineManager().getEngineByName("js");
1000.times {
  engine.eval('x>3', new SimpleBindings([x:44]))
}
println (System.currentTimeMillis() - t)
250

Obviously, it will not be absolutely the same in general and I assume Eval provides much more (and looks less ugly), but ScriptEngine is good enough for my case and it performs almost 30 times faster.

Tuesday, April 3, 2012

Accessing request parameters from UserDetailsService in Spring Security

If you need to use several properties to authenticate user with Spring Security, for example, by login and domain, there is no built-in way to do it. Even when you override UserDetailsService it only calls method with one username parameter.
Fortunately, there is easy way to access request context with Grails, so it is possible to extract any parameter you need, like:


import org.springframework.web.context.request.RequestContextHolder

class UserDetailsService implements GrailsUserDetailsService  {

  UserDetails loadUserByUsername(String username) throws UsernameNotFoundException {
    def otherParameter = RequestContextHolder.requestAttributes.params.other
       ...
  }
}


Monday, April 2, 2012

Redis exception: Cannot use Jedis when in Multi.

Jedis newbie problem when adding transactions to Redis client application:


Cannot use Jedis when in Multi. Please use JedisTransaction instead.
redis.clients.jedis.exceptions.JedisDataException: Cannot use Jedis when in Multi. Please use JedisTransaction instead.
 at redis.clients.jedis.BinaryJedis.checkIsInMulti(BinaryJedis.java:1651)
 at redis.clients.jedis.Jedis.hmset(Jedis.java:724)
 at test.TestService.updateSomething(TestService.groovy:39)
 at test.TestService$_processBet_closure1.doCall(TestService.groovy:16)
 at grails.plugin.redis.RedisService.withRedis(RedisService.groovy:67)

Description of exception is absolutely correct though a little confusing. Problem is that you are trying to call methods on Redis connection object instead of Transaction object. So most probably you have something like:


  Jedis jedis = pool.getResource()
  jedis.watch('foo')
  ...
  def transaction = jedis.multi()
  jedis.set("foo", value.toString())
  def result = transaction.exec()


But instead, you should have something like:


  Jedis jedis = pool.getResource()
  jedis.watch('foo')
  ...
  def transaction = jedis.multi()
  transaction.set("foo", value.toString())
  def result = transaction.exec()


Friday, March 30, 2012

Code snippet for JMS queue reader

There is quick snippet how to read from ActiveMQ queue with Groovy:


import org.apache.activemq.pool.PooledConnectionFactory
import javax.jms.Connection
import javax.jms.Session
import org.apache.activemq.command.ActiveMQQueue
import javax.jms.MessageConsumer
import org.springframework.jms.listener.adapter.MessageListenerAdapter
import javax.jms.Message

  PooledConnectionFactory pooledConnectionFactory
  ActiveMQQueue testJmsQueue

  pooledConnectionFactory.start()
  Connection conn = pooledConnectionFactory.createConnection()
  Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE)
  MessageConsumer consumer = session.createConsumer(testJmsQueue)
  consumer.setMessageListener(new MessageListenerAdapter() {
    @Override
    void onMessage(Message message) {
      println message.getContentMap()
    }
  })
  conn.start()


Thursday, March 22, 2012

How to read IMAP email with Groovy

There is quick snippet how to read email from Google IMAP with Groovy:

@Grab(group='javax.mail', module='mail', version='1.4')

import javax.mail.*
import java.util.Properties

def session = Session.getDefaultInstance(new Properties(["mail.store.protocol":"imaps", "mail.imaps.host":"imap.gmail.com", "mail.imaps.port":"993"]),null)
def store = session.getStore("imaps")
store.connect('imap.gmail.com', 'test@gmail.com', 'pass')
def folder = store.getFolder("INBOX")
folder.open(Folder.READ_WRITE)
folder.messages.each { msg ->
  println msg.content
}


Friday, March 16, 2012

JavaScript unit testing in Grails

Recently I wanted to test some complicated JavaScript code and surprisingly there is no a lot of options to choose from in Grails. Fortunately, I was already using Geb and it is extremely easy in this case.
I decided to use QUnit testing framework and all that needs to be done in this case is to place few static files in JS, CSS and HTML folders, create Geb page for QUnit reports and Geb Spec for launching browser. For development everything can be done (and much faster) with QUnit alone. Geb is basically only needed to run tests from CI (Jenkins in my case).


There is my Geb Spec:

package my.test.specs

import spock.lang.Stepwise
import my.test.pages.QUnitPage

@Stepwise
class QUnitSpec extends GebScreenshotsSpec {

  def "run QUnit tests"() {
    when:
    to QUnitPage

    then:
    at QUnitPage
    failedCount == '0'
  }

}


Geb page:

package my.test.pages

import geb.Page

class QUnitPage extends Page {
  static url = "http://localhost:"+System.getProperty("server.port", "8080")+"/my-test/js/tests/qunit.html"

  static at = { $("#qunit-header") }

  static content = {
    failedCount(wait:true) {$('#qunit-testresult .failed').text()}
  }
}


qunit.html and all it's content is just plain QUnit framework.

Disadvantages are that reports are not persisted or summarized with other testing frameworks, but if you have screenshots in your Geb tests, it will provide some feedback. Another issue is if you are not using Geb, than this can be overkill to use it only for JS unit tests.
But in case you already have Geb this is super-simple and lightweight approach.

Tuesday, March 13, 2012

Spock tests modularization

It is not very clear from Spock documentation how to split Spock feature methods into multiple functions. But, for example, Geb test scenarios sometimes can be quite complicated (if you want to verify something across multiple pages) and worth extracting into multiple methods for clarity and reuse.
In practice, this is very simple and all you have to do is just extract method and define Java style asserts as it is in Spock documentation about helper classes. What is not in documentation, is that you can freely mix actions with validation in same method, so for example, you can do something like this:

def "do something"() {
  when:
  someLink.click()

  then:
  validateEverything()
}
def validateEverything() {
  menuLink1.click()
  assert text1 == 'some text'
  menuLink2.click()
  assert text2 == 'other text'
}

At least this works fine with Geb.

Tuesday, February 21, 2012

Grails nullable boolean and integer fields

When your GORM domain object fields are defined as Java primitives like boolean it can only be specific value like true or false. But sometimes you may want to have also third value for undefined values or NULL as in database. With primitives you will get error like:
Executing action [edit] of controller [com.test.TestController] caused exception: Null value was assigned to a property of primitive type setter of com.test.TestObject.testproperty; nested exception is org.hibernate.PropertyAccessException: Null value was assigned to a property of primitive type setter of com.test.TestObject.testproperty

To enable nulls for such fields, you just have to change type from boolean primitive to Boolean object and that is it.

Thursday, February 16, 2012

sprintf, printf bug in Groovy

Recently I found strange issue when running Grails application on Linux. Sometimes, when formatting decimal number with sprintf it does not work and simply returned original number. After some investigation I found that this is general Groovy behavior and simple script with:

println sprintf('%.2f', 1/609)

returns 0.0016420361 as result, when it suppose to return just 0.00. I was not able to reproduce this bug on Windows or in plain Java, so most probably this is GDK issue, but I didn't investigated it very carefully.
Anyway, this is very easy to fix by casting numbers into decimals explicitly, so this code:

println sprintf('%.2f', 1d/609)

returns nice and correct results.

Friday, February 10, 2012

JavaScript parseInt does not work as expected when parameter starts with 0

There is standard way to parse Strings into Numbers in JavaScript by using parseInt function. But sometimes it works unexpected, and for example, returns 0 when you expect 8. If that is the case, then most probably problem is that when parameter starts with 0 (like '08') parseInt treats it as octal number and changes radix to 8.
This is annoying and can take some time to discover. Luckily, it is easy to fix by specifying radix directly as second parameter. For example like:

parseInt('08', 10)




Wednesday, February 8, 2012

Copy method in Groovy

If you need to copy method reference between objects at runtime, it is super easy in Groovy. Just add & in front of method name and this it. For example:

new Expando(id:it.id, user:it.user, mymethod:it.&othermethod, ...)

For some reason this is not very wide known feature (was not for me), and I unsuccessfully tried to use metaClass and other complicated things before finding this.

Friday, February 3, 2012

Freezing table header with JQuery

When you have long table on page, with a lot of numbers, and you scroll down and header is not visible anymore, it is often difficult to track what number is which column. One nice solution to this problem is to lock and freeze table header when scrolling page.
Unfortunately there is no ready out of the box solution with JQuery for this problem. Fortunately, it is easy to implement with few methods. There are examples how to do it, but for some reason they didn't worked well in my situation (no support for resize and horizontal scroll), so I have adjusted them for my situation.

For example, you have your table named tabs.

First, you have to create invisible table that will hold frozen header.

<table cellpadding='0' cellspacing='0' id="header-fixed"  style="position: fixed; top: 0px; display:none;"></table>

Then just add this Javascript:

     <script type="text/javascript">
       var tableOffset;
       var header;
       var fixedHeader;

       function resize() {
         var totalwidth = $('#tabs').css('width');
         fixedHeader.css('width', totalwidth);
         var widths = [];
         $('#tabs thead th').each(function() {
           widths.push($(this).width());
         });
         var i=0;
         $('#header-fixed th').each(function() {
           this.width = widths[i];
           i++;
         });
       }
       function resizeAndShow() {
         var offset = $(this).scrollTop();
         if (offset >= tableOffset && fixedHeader.is(":hidden")) {
           fixedHeader.show();
           resize();
         } else if (offset < tableOffset) {
           fixedHeader.hide();
         }
         fixedHeader.css('left', $('#tabs').position().left - $(this).scrollLeft());
       };
        $(document).ready(function() {
          tableOffset = $("#tabs").offset().top;
          header = $("#tabs > thead").clone();
          fixedHeader = $("#header-fixed").append(header);

          $(window).bind("scroll", resizeAndShow);
          $(window).resize(resize);
        });
     </script>

This is it.

Thursday, January 5, 2012

Getting text of selected option in Geb

Geb has very nice function to get value of selected option, but I was not able to find method to get it's text. Fortunately, it is still possible using ugly Selenium API and can be nicely hidden behind Page Object. For example, this is what I have in my Page Object:

myField {$("#my") }
myFieldValue {  new Select($("#my").getElement(0)).getFirstSelectedOption().getText() }

And this is how it looks in Spec:

when:
myField.value('B')

then:
myFieldValue == 'B'