Converting byte array to string without loosing data

Recently I had to deal with poor Redis interface, that didn't allowed to store bytes and it was not possible to update it or change. Redis handles bytes perfectly for keys and data, but there was no way to pass it. Standard solution for this task is to use Base64, but for my case overhead was too big as I had to store hundreds of millions of records. UTF is not good too, as it is ruining data. Finally, I found that right encoding for this task is to use ISO-8859-1 encoding and it works just fine.

new String([10,20,-30,126] as byte[], "ISO-8859-1").getBytes("ISO-8859-1");

Quickly create groovy script in Intellij IDEA

It is sometimes frustrating when you need quickly check db, non-trivial http, or AWS with Groovy. It is fine and great if standard lib is enough for your case, but once you need new libraries, it is not that quick. Grab is amazing, but you need to specify dependency ids and recall exact API and settings. At some point I had a bunch of typical scripts with typical settings, but you have to be in same project to find them, remember not to commit them, and they tend to grow big and have their own lives at some point. Bad for hacking and it is not convenient for majority of one-off cases.

That is where I started using IntelliJ Live Templates: they are available in every project, easily configurable, you can dispose or save results as needed, they accessible from regular files, scratches, even Groovy console.

My typical template consists of everything needed for single use case: grabs, imports, initialisation and bunch of examples. It is always easier to slash unneeded stuff than recall all API specifics. For me, it is more like tldr output, than what typical default template looks like.

There is example of one template I use for AWS S3:

@Grab('com.amazonaws:aws-java-sdk:1.9.31')
import com.amazonaws.auth.BasicAWSCredentials
import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.services.s3.model.ObjectMetadata

def client = new AmazonS3Client(new BasicAWSCredentials('access', 'secret'))

println client.listObjects("bucket", "name").getObjectSummaries().collect{it.key}

def meta = new ObjectMetadata()
meta.setContentType("text/plain")
client.putObject("bucket", "name", new ByteArrayInputStream("1".bytes), meta)

client.deleteObject("bucket", "name")

println new String(client.getObject("bucket", "name").objectContent.bytes)

client.listVersions("bucket", "name").versionSummaries.each {
    println "${it.deleteMarker}   ${it.key}   ${it.size}"
}

client.deleteVersion("bucket", "name", "123")

And of course, you can have multiple templates preconfigured for different environments and use cases.

Biggest disadvantages comparing with regular files is that they are stored somewhere deeply in IntelliJ and not easily transferable, still they are easily exportable with other settings. Other disadvantage is that content is not searchable, so you have to pack all keywords as abbreviation or description, but anyway it is not part of universal search.

Adding custom hostname to docker-machine VirtualBox image

When boot2docker was changed to docker-machine, it added dubious feature of being read only container, so no changes to VirtualBox will be preserved after restart. Saving changes is still perfectly fine for Linux Docker users, but not docker-machine users anymore for some reason.

So, to add new hostname before, you could just go to VirtualBox, change /etc/hosts file, and this is it - this will not work with docker-machine, as hosts file will be erased after restart. You are luckier, if you are not using net:host in docker-compose, there is special property for that too. But, if you are using net:host, the only working option is to change VirtualBox image and add this configuration:

VBoxManage modifyvm "default" --natdnshostresolver1 on

In this case it will use host machine to resolve all dns requests, even those from /etc/hosts.

How to add configuration parameters to AWS lambda with API gateway

Unfortunately, there is no easy way to add configuration parameters to AWS lambdas, there is almost nothing in context that you can configure; there is description, but besides being ambiguous, it is not accessible from function itself. Fortunately, it is possible to set additional values from API gateway, even it is not straightforward.

When you have created API gateway mapping, go to the method screen in gateway console, there is screen with multiple boxes, one of which is called Integration Request, click it and there will be section called Mapping Templates. Add new mapping template for type application/x-www-form-urlencoded, save it, edit value and change from default Input passthrough to Mapping template, then you can transform your original request, by adding additional parameters, it uses velocity for transformation, but there is basic example:

{
  "my-new-configuration" : "my value",
  "body" : "$input.path('$')",
  "params" : "$input.params()",
  "ip" : "$context.identity.sourceIp",
  "user-agent" : "$context.identity.userAgent"
}

Besides possibility to add anything to lambda function it also has access to interesting request data like headers, ip, client type, user agent, country, etc. - all will be available in your map or pojo as parameter for lambda function. And do not forget to click Deploy API after making changes to make them live.

My recipe for IntelliJ IDEA distraction free mode

One of the top features in IntelliJ IDEA 14.1 is distraction free mode. I am totally in love with it and not looking back, working 100% of time in it. Biggest concern about it is: loosing tabs. I was happy with tabs before, but it looks like there is a bunch of better options:

• Recent Files (Cmd + E) - basically dropdown with tabs, but you can also do text search in it.
• Switcher (Ctrl + Tab) - same as Recent Files, but opens file once shortcut is released. A little faster, but you are loosing search. This is the closest to original tabs.
• Search Everywhere (Shift + Shift) - acts as Recent Files, but if you type something - also does search for Classes, Files, Symbols, Actions, Preferences. So together with tabs it basically replaces also Cmd + E, Cmd + O, Shift + Cmd + O, Alt + Cmd + O, Shift + Cmd + A.

I chose Search Everywhere as it is most feature rich and is really awesome. I also remapped it to Ctrl + E as it includes Recent Files, but as shortcut Ctrl + E works faster and is easier to reach.

Tabs had another aspect - tabs pinning, I didn't used it a lot, but for occasional use, bookmarks replace them nicely and provide more features as can pin specific line together with file. So use F3 to bookmark and Cmd + F3 to list bookmarks.

Everything else: like splitting, navigation bar, file path, tool windows, etc. is easy accessible via shortcuts or Find Action... (Shift + Cmd + A) if you forgot shortcut.

Grails requires restart after changes in domain class

Grails officially supports domain class reloading since version 2, but recently I found problems using it together with MongoDB GORM plugin, because it required application restart after every change to domain class.

So if Hibernate is primary GORM and Mongo is used only for some particular cases, disabling it can be a good option to make development easier.

CanJS execute code on component initialization

I didn't found it described anywhere, but it works for me. If you need to execute code after component initialization, you can add init method to scope and it is executed after component is created.

    scope: {
      init: function() {
        runme();
      }
    }

Chaining jQuery promises in for cycle

JQuery has nice support for promises, but some stuff is not obvious and recently I just stuck with one particular problem: I had to prepare a bunch of closures in for cycle and added them to deferred object which was resolved afterwards. All of them had to be executed strictly one after another, but there are ajax calls so I cann't do it old fashioned way, since sinchronous ajax is weird.
So I did this:

function doPromise(someVar) {
  return function() {
    return $.ajax({ type: 'POST', url: 'http://myurl', success: doStuff, error: panic});
  }
};
function start() {
  var promise = $.Deferred();
  for (var i=0;i<10;i++) {
    promise.then(doPromise(i) );
  }
  promise.resolve();
};

And I was expecting it being called one after another, which was not the case and all of them fired simultaniously. As I found out finally, all thens are equal and I had to add them, not to original promise, but dirrectly to each other, which is a little confusing to me, but works. So this is the correct code for my case:

function start() {
  var original = $.Deferred();
  var promise = original;
  for (var i=0;i<10;i++) {
    promise = promise.then(doPromise(i) );
  }
  original.resolve();
};

Launching Spring Boot in Groovy with Grab

Since launch of Spring Boot I was wondering why it needs separate CLI, especially for Groovy where there is already Grab. Of course CLI provides some additional features, like reloading and out of the box dependencies, but why no one runs Boot with vanilla Groovy? So I tried it myself, and it looks like problem is that there is conflict with default Groovy libs, because standard installation includes servlet-api-2.4.jar which does not work with current Tomcat or Jetty, and as there is no anything like fork mode in Grails, there is only one way to avoid it - not to load this jar (delete it or use custom load conf).

Otherwise this works fine for me:

@Grab("org.springframework.boot:spring-boot-starter-web:1.0.2.RELEASE")
@Grab("org.springframework.boot:spring-boot-starter-actuator:1.0.2.RELEASE")
import org.springframework.boot.SpringApplication
import org.springframework.boot.autoconfigure.EnableAutoConfiguration
import org.springframework.web.bind.annotation.*

@RestController
@EnableAutoConfiguration
class ThisWillActuallyRun {

  @RequestMapping("/")
  String home() {
    "Hello World!"
  }
}
SpringApplication.run(ThisWillActuallyRun, args)

Access to services from gsp views in Grails

Accessing services from views is considered bad practice, so there is no easy way to do it. It defenitely should not be overused, as it complicates testing and couples logic with presentation. However it is not necessary crime and can be handy for some cases.

For example, if you need access to user parameters. Traditional way to do it, is by using taglibs. What is less known is that you can use taglibs not just to produce text output, but also inside snippets of code in views. For example, you can use it to select users preferred language without need to inject it into every controller:

Taglib:

class UserSettingsTagLib {
    def springSecurityService

    def myLanguage = { attrs, body ->
      out << springSecurityService.currentUser?.myLanguage?.id
    }

}

Gsp:

<g:select name="language" optionKey="id" optionValue="name" value="${g.myLanguage()}" from="${Language.list().sort{it.name}}"/>

Adding error messages to domain object in Grails

Sometimes it is more convenient to do some validation outside of the domain object, on the other side it is still nice to use standard error reporting facilities. For these cases, you can inject errors into domain objects like:

    if (someError) {
      domainobject.errors.rejectValue('param', "hasErrors")
    }
    ...
    if (!domainobject.hasErrors() && domainobject.save(flush: true)) {
    ...

Jenkins kills Tomcat or other Java process after job ends

Recently, I was adding new application deployment to Tomcat server that was started by Jenkins. Process was starting nicely, but was stopping at the end of the job. The only clue was log record:

Service exit with a return value of 143

This issue can be easily solved by adding -Xrs to JAVA_OPTS. Option description can be found at http://docs.oracle.com/javase/6/docs/technotes/tools/windows/java.html.

Hosting Ivy or Maven repository on Github with Gradle

If you don't have access to any public repo, but want to host libraries - then Github can be a solution.
Unfortunately, it does not provide proper interfaces for the client applications, but this is still possible.

Lets say that local root folder of your git project is /test.
First, you need to create all necessary poms and jars. To do it, you have to add this to your build.gradle file:

uploadArchives {
  repositories {
    mavenDeployer {
      repository(url: "file:///test/repo")
    }
  }
}

URL property is for proper Maven repository - for now unfortunately, Github is not, so you will have to enter some local folder. After this is added, run Gradle task uploadArchives, then commit and push all output files with git.

Unfortunately, this is not all, as SHA signature is wrong, because your git URL is not same as local. So you will need to find out proper SHA, and until you want to calculate it yourself, one easy way to do it is find some talkative Maven client, for example Grape. Luckily it is part of normal Groovy installation.

Then, you need to add new repo into Grape configuration, open ${user.home}/.groovy/grapeConfig.xml file and add your new repo like

<ibiblio name="myrepo" root="https://raw.github.com/me/mymodule/master/repo" m2compatible="true"/>

And, to get dependencies, just run Grape in verbose mode, for example like:

grape -V install my.group mymodule 0.1

It will fail, but you will get one great error message like:

...-0.1.pom: invalid sha1: expected=7f1089041d63ce7eaa5d6a35ddda3aaa606042e3 computed=8a941ef40645d449b0740023624c2f8b67b84c59 (516ms)

8a941ef40645d449b0740023624c2f8b67b84c59 is what we were looking for.
Just find file whose signature was incorrect, for example for pom it can be mymodule.pom.sha1, update it with new signature and push into git.

That is all, you have new great Maven repository with root under https://raw.github.com/me/mymodule/master/repo.

Run background java process from Jenkins

Jenkins is using mechanism to detect spawned processes and kill them after finishing job, so if you will just start such ad-hoc background daemon process, Jenkins will kill it with warning:

Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information

Fortunately, there is easy hack to tell Jenkins ignore such process, just create fake BUILD_ID and start your application, for example, this can be entered into Execute shell build step:

BUILD_ID=dontKillMe nohup java -jar test.jar &

Warning is still there, but process should be fine and running. There is more about it here.

How to kill Tomcat server running from standalone plugin

I have demo application which is running from Grails standalone plugin and wanted to automate deployment from Jenkins. Starting such application is well documented and easy, but there is nothing about stopping it.
After quick digging plugin's code, I found that it has undocumented feature to do it. It creates server on application server port + 1, and as soon as there is connection - shuts server. So if your server port is 8080, this command from same computer will kill server:

telnet localhost 8081

ActiveMQ pool logging

If you are using org.apache.activemq.pool.PooledConnectionFactory for ActiveMQ connection pooling, it is maybe frustrating to detect server connection problems as it does not log anything at INFO level. DEBUG level contains reconnection records, but also a lot of unnecessary, boring stuff, not needed for production. Fortunately, there is logger that can be used for connections monitoring, which is not nicest, but working solution, and definitely it is better than nothing:

    debug 'org.apache.activemq.util.ThreadPoolUtils'

Working around grails.serverURL redirect in Grails 2

The most annoying issue with Grails 2 migration is that it requires redirection to grails.serverURL when it is specified. This property is needed if there are absolute links in application (in emails, for example).
First problem that I had with it, was after deploying application to test environment. We deploy production version, so all redirects became redirects to production, this could be solved by externalizing Grails configuration.
Second problem was more severe. We have following setup: load balancer with two nodes behind it. Our deployment script takes first node down, deploys, runs smoke tests, releases first node, takes second node down, deploys, smoke tests, releases.
So basically, production URL is load balancer's entry point, and when smoke tests run on node which is down it is never redirected to tested server (because it is down and it is behind load balancer), so every redirect (after spring security login, for example), breaks tests flow.
After some digging around, I found the best option is to abandon grails.serverURL - which is totally unusable now, and add to email link generation my own parameter from externalized Grails configuration (for each environment). This is sad as createLink with absolute parameter was great function and it didn't required dragging custom parameters.
That is how it is now.
What I would love to see in the future is grails.serverURL is used ONLY in all links generated with absolute:true and nowhere else, but all redirects work as if grails.serverURL not specified at all.

Importing plugin classes into Grails scripts

Classes from plugin src folder are not available to Grails scripts or plugin scripts itself. They can be compiled and manually loaded, but will be only available for applications or tests launched from script - not to script itself.
However it is easy to work around this issue if it is your plugin or you can rebuild it. It is only needed to compile classes into lib when packing plugin, after that all libs are loaded when running script.
For example this can be done by adding next snippet into _Events.groovy file of the plugin:

eventPackagePluginStart = {
  compile()
  ant.jar ( destfile : 'lib/pluginlibs.jar' , basedir : 'target/classes', includes:"com/myplugin/**")
}

Getting object from calling method in Java

This is hardly production scenario, but at some point I wanted to find value of a variable from a caller class. Looks like this is not trivial task and is much more complicated than it should be (for some reason), however still possible.
Finding method and class name is easy via Thread class, but accessing real objects seems to be only possible with Java Debug Interface. For this purpose, application must be executed in debug mode (for grails it is with grails-debug command). There is no need for any additional library as everything is included into JDK and probably is used by JDK debugger.
General scenario is as folows:

• connect with debugger provided with JDI via socket (there are other options, but sockets seems default for most tools)
• spawn new thread where you will wait to do whatever you want to do with data on stack trace
• suspend application at point where you want to do it
• at the end of your work - continue application

There is example:

def att = com.sun.jdi.Bootstrap.virtualMachineManager().attachingConnectors().find{it instanceof com.sun.tools.jdi.SocketAttachingConnector}
Map arguments = att.defaultArguments()
com.sun.jdi.connect.Connector.Argument hostArg = arguments.get("hostname").setValue("localhost")
com.sun.jdi.connect.Connector.Argument portArg = arguments.get("port").setValue("5005")
com.sun.jdi.VirtualMachine vm = att.attach(arguments);
def thread = vm.allThreads().find {it.name()=='main'}
def runnable = new Runnable() {
  void run() {
 while (!thread.isSuspended() ) {
   Thread.sleep(100)
 }
 def frames = thread.frames().findAll {it.location().toString() =~ /grails.plugin.spock.test.listener.PerSpecRunListener:.*/}
 frames.each {frame->
   def res = frame.thisObject()
   def field = res.type().fieldByName("failureCount")
   if (field) {
       def countref = res.getValue(field).toString()
        ...
      }
   }
     thread.resume()
  }
}
new Thread(runnable).start()
thread.suspend() 

Downloading URL with login and password

In javadoc there is no clear way on how to download page that requires Basic Authentication, for example like:

  def result = new URL("http://www.requireslogin.com/").getText()

Fortunately, this is still very possible and not too complicated, just add this somewhere before calling URL:

  java.net.Authenticator.setDefault(new java.net.Authenticator() {
    protected java.net.PasswordAuthentication getPasswordAuthentication() {
      return new java.net.PasswordAuthentication("user", "password".toCharArray());
    }
  });

Possibly it would be better to have login and password as parameters, but this still works.