Adding custom hostname to docker-machine VirtualBox image

When boot2docker was changed to docker-machine, it added dubious feature of being read only container, so no changes to VirtualBox will be preserved after restart. Saving changes is still perfectly fine for Linux Docker users, but not docker-machine users anymore for some reason.

So, to add new hostname before, you could just go to VirtualBox, change /etc/hosts file, and this is it - this will not work with docker-machine, as hosts file will be erased after restart. You are luckier, if you are not using net:host in docker-compose, there is special property for that too. But, if you are using net:host, the only working option is to change VirtualBox image and add this configuration:

VBoxManage modifyvm "default" --natdnshostresolver1 on

In this case it will use host machine to resolve all dns requests, even those from /etc/hosts.

How to add configuration parameters to AWS lambda with API gateway

Unfortunately, there is no easy way to add configuration parameters to AWS lambdas, there is almost nothing in context that you can configure; there is description, but besides being ambiguous, it is not accessible from function itself. Fortunately, it is possible to set additional values from API gateway, even it is not straightforward.

When you have created API gateway mapping, go to the method screen in gateway console, there is screen with multiple boxes, one of which is called Integration Request, click it and there will be section called Mapping Templates. Add new mapping template for type application/x-www-form-urlencoded, save it, edit value and change from default Input passthrough to Mapping template, then you can transform your original request, by adding additional parameters, it uses velocity for transformation, but there is basic example:

{
  "my-new-configuration" : "my value",
  "body" : "$input.path('$')",
  "params" : "$input.params()",
  "ip" : "$context.identity.sourceIp",
  "user-agent" : "$context.identity.userAgent"
}

Besides possibility to add anything to lambda function it also has access to interesting request data like headers, ip, client type, user agent, country, etc. - all will be available in your map or pojo as parameter for lambda function. And do not forget to click Deploy API after making changes to make them live.

How to create executable JAR from single Groovy script

Groovy scripts are nice - they allow to use dependencies and write logic in small single file, but Groovy is mostly available on developers machine, but not on servers, so deploying them is a little more tricky. You can package them in uberjar with Gradle, but it requires additional build file and compatible structure, so it is more like project than script already.

To keep things simple, I have created Groovy script that converts other Groovy scripts into executable JARs. It packages all dependencies from Grab and compiles original script into Java main class with same name as script and sets it in manifest. So it is perfectly usable from empty JVM. JAR structure is also compatible with AWS lambda, so it can be launched from there too, just put logic in specific method. To convert your script, just download it from here and run like:

./scriptjar.groovy input.groovy output.jar

Script tries to find groovy libs via GROOVY_HOME, so set it or just hack it for your location, it is just one small Groovy script.

Erlang connection from boot2docker

If you need to connect from outside of Docker into boot2docker instance on MacOS, but nothing happens: First, make sure that all ports are mapped in VirtualBox instance - at minimum it requires 4369 and 20000-20999. Then, for hidden nodes, you may need to ping them explicitly, like:

net_adm:ping('server@127.0.0.1').

Changing SSH port for dynamic inventory script in Ansible

If you have to write dynamic inventory script in Ansible, but need to use SSH port which is not 22, there is example output of the script output:

{"databases":{
    "hosts":["127.0.0.1"],
    "vars":{
        "ansible_ssh_port":2222
        }
    }
}

Ansible freezes during start

If Ansible stops for several minutes or so during start without any messages, check if it calls your inventory script, because it might query for additional options using --host parameter and if there are a lot of hosts, it may take some time.

It can be easily fixed by adding this to JSON root:

    "_meta" : {
       "hostvars" : {
       }
    }

More about it here.

VirtualBox DHCP gives same IP address to different machines

Usually happens after cloning VMs and problem is because there is the same MAC address. Check virtual machine network settings and change as needed.

Script that logs time to JIRA from Git commit messages

If you need to report work time in JIRA and native comment hashes are not suitable for you, there is simple Groovy script that does the same and can be adjusted to particular scenario or configuration:

@GrabResolver(name='test', root='https://maven.atlassian.com/content/groups/public/')
@Grab( group ='com.atlassian.jira.plugins', module = 'jira-soapclient', version = '4.4' )
@Grab(group='org.eclipse.jgit', module='org.eclipse.jgit', version='3.3.2.201404171909-r')

import com.atlassian.jira.rpc.soap.client.JiraSoapServiceServiceLocator
import com.atlassian.jira.rpc.soap.client.RemoteWorklog
import groovy.transform.TailRecursive
import org.eclipse.jgit.lib.Repository
import org.eclipse.jgit.revwalk.RevCommit
import org.eclipse.jgit.revwalk.RevWalk
import org.eclipse.jgit.storage.file.FileRepositoryBuilder
import java.text.SimpleDateFormat
import static java.util.Calendar.*

@TailRecursive
static Set<RevCommit> subcommits(RevCommit commit, int start, RevWalk walk, def all) {
  if (commit.commitTime > start) {
    all << commit
    commit.parents.collect{ walk.parseCommit(it) }.each {
      subcommits(it, start, walk, all)
    }
  }
  return all
}

Repository repository = new FileRepositoryBuilder().setGitDir(new File(args[3])).setMustExist(true).build()
RevWalk walk = new RevWalk(repository)

def formattime = new SimpleDateFormat('yyyy-MM-dd')
formattime.setTimeZone(TimeZone.getTimeZone("UTC"))
Date start = formattime.parse(args[4])
Date end = formattime.parse(args[5])

def all = subcommits(walk.parseCommit(repository.resolve("HEAD")), start.time / 1000L as int, walk, new HashSet())
def mine = all.findAll{ it.authorIdent.name == args[6] && it.fullMessage =~ /^PRJ-[0-9]+/ }

def actual = mine.min{ it.commitTime }

def tasks = (start..end).findAll{!(it[DAY_OF_WEEK] in [SUNDAY, SATURDAY])}.collect {date ->
  actual = mine.find{ date.time / 1000L <= it.commitTime && (date + 1).time / 1000L > it.commitTime } ?: actual
  [date:date, task:(actual.fullMessage =~ /(PRJ-[0-9]+)/)[0][1]]
}

if (args[0] == 'test') {
  tasks.each { println "${it.date} - ${it.task}" }
} else if (args[0] == 'prod') {
  def jiraSoapService = new JiraSoapServiceServiceLocator().getJirasoapserviceV2(new URL('http://jira.com/rpc/soap/jirasoapservice-v2'))
  String token = jiraSoapService.login(args[1], args[2])
  tasks.each { jiraSoapService.addWorklogAndRetainRemainingEstimate(token, it.task, new RemoteWorklog(timeSpent: "8h", startDate:it.date.toCalendar())) }
}

It uses a bunch of command line parameters - first is either prod or test; test just prints tasks, prod sends them to JIRA. There is an example CLI command:

groovy gittotempo.groovy prod jirauser jirapassword c:\work\project\.git 2014-08-01 2014-08-31 gituser

Grails dies in the middle of Jenkins jobs after update

If you have upgraded recently and facing problem when job suddenly dies in the middle of test execution, like:

19:20:39  | Running 277 unit tests... 156 of 277
19:20:53  Build step 'Execute shell' marked build as failure
19:20:54  Description set: 

Most probably it is because since version 2.2 test-app, run-app, run-war runs in fork mode and Jenkins has some fancy zombie process detection that kills spawned processes it does not like.
Quick solution is to disable fork in test environment or you can refer my previous post about hiding such processes.

Traffic redirection to https in Grails behind load balancer

Grails security plugin has nice feature that allows automatically redirect queries for specific pages to https. For example, you can put this code into Config.groovy to redirect all requests:

grails.plugins.springsecurity.secureChannel.definition = [
    '/**':         'REQUIRES_SECURE_CHANNEL'
]

Unfortunately this does not work when your Tomcat is behind load balancer that does all encryption. To do it requires a little more work.

First, it is needed to specify how to detect either connection is secure or not. Standard way to do it with load balancers is to use X-Forwarded-Proto header. So it needs to be specified in Config.groovy like:

grails.plugins.springsecurity.secureChannel.useHeaderCheckChannelSecurity = true
grails.plugins.springsecurity.portMapper.httpsPort = 443

Worst part, is that it requires Tomcat reconfiguration, so it is needed to update Connector element in server.xml, like:

<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443"  proxyName="yourserver.com" proxyPort="443" secure="true" scheme="https"  />

Good part is that it works nicely locally and across multiple environments, no need for fake certificates or different builds.

Setting environment variables when running command via SSH in Ubuntu

There are less environment variables available when running command directly by SSH then interactively. This is because for some reason, Ubuntu does a little less for non-interactive shells and stops executing .bashrc after this line:

[ -z "$PS1" ] && return

So if you wish to run anything, it should be done before, like:

export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_21/
[ -z "$PS1" ] && return

Jenkins kills Tomcat or other Java process after job ends

Recently, I was adding new application deployment to Tomcat server that was started by Jenkins. Process was starting nicely, but was stopping at the end of the job. The only clue was log record:

Service exit with a return value of 143

This issue can be easily solved by adding -Xrs to JAVA_OPTS. Option description can be found at http://docs.oracle.com/javase/6/docs/technotes/tools/windows/java.html.

Using JIRA API from Groovy

If you need to script some JIRA task, it is very easy to do:

@GrabResolver(name='atlassian', root='https://maven.atlassian.com/content/groups/public/')
@Grab( group ='com.atlassian.jira.plugins', module = 'jira-soapclient', version = '4.0' )

import com.atlassian.jira.rpc.soap.client.*

def jira = new JiraSoapServiceServiceLocator().getJirasoapserviceV2(new URL('http://host/rpc/soap/jirasoapservice-v2'))
String token = jira.login('user', 'pass')
println jira.getVersions(token, "PRJ").sort({ it.sequence }).find({ !it.released }).name

which takes latest unreleased version for the project PRJ.

Hosting Ivy or Maven repository on Github with Gradle

If you don't have access to any public repo, but want to host libraries - then Github can be a solution.
Unfortunately, it does not provide proper interfaces for the client applications, but this is still possible.

Lets say that local root folder of your git project is /test.
First, you need to create all necessary poms and jars. To do it, you have to add this to your build.gradle file:

uploadArchives {
  repositories {
    mavenDeployer {
      repository(url: "file:///test/repo")
    }
  }
}

URL property is for proper Maven repository - for now unfortunately, Github is not, so you will have to enter some local folder. After this is added, run Gradle task uploadArchives, then commit and push all output files with git.

Unfortunately, this is not all, as SHA signature is wrong, because your git URL is not same as local. So you will need to find out proper SHA, and until you want to calculate it yourself, one easy way to do it is find some talkative Maven client, for example Grape. Luckily it is part of normal Groovy installation.

Then, you need to add new repo into Grape configuration, open ${user.home}/.groovy/grapeConfig.xml file and add your new repo like

<ibiblio name="myrepo" root="https://raw.github.com/me/mymodule/master/repo" m2compatible="true"/>

And, to get dependencies, just run Grape in verbose mode, for example like:

grape -V install my.group mymodule 0.1

It will fail, but you will get one great error message like:

...-0.1.pom: invalid sha1: expected=7f1089041d63ce7eaa5d6a35ddda3aaa606042e3 computed=8a941ef40645d449b0740023624c2f8b67b84c59 (516ms)

8a941ef40645d449b0740023624c2f8b67b84c59 is what we were looking for.
Just find file whose signature was incorrect, for example for pom it can be mymodule.pom.sha1, update it with new signature and push into git.

That is all, you have new great Maven repository with root under https://raw.github.com/me/mymodule/master/repo.

Run background java process from Jenkins

Jenkins is using mechanism to detect spawned processes and kill them after finishing job, so if you will just start such ad-hoc background daemon process, Jenkins will kill it with warning:

Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information

Fortunately, there is easy hack to tell Jenkins ignore such process, just create fake BUILD_ID and start your application, for example, this can be entered into Execute shell build step:

BUILD_ID=dontKillMe nohup java -jar test.jar &

Warning is still there, but process should be fine and running. There is more about it here.

How to kill Tomcat server running from standalone plugin

I have demo application which is running from Grails standalone plugin and wanted to automate deployment from Jenkins. Starting such application is well documented and easy, but there is nothing about stopping it.
After quick digging plugin's code, I found that it has undocumented feature to do it. It creates server on application server port + 1, and as soon as there is connection - shuts server. So if your server port is 8080, this command from same computer will kill server:

telnet localhost 8081

ActiveMQ pool logging

If you are using org.apache.activemq.pool.PooledConnectionFactory for ActiveMQ connection pooling, it is maybe frustrating to detect server connection problems as it does not log anything at INFO level. DEBUG level contains reconnection records, but also a lot of unnecessary, boring stuff, not needed for production. Fortunately, there is logger that can be used for connections monitoring, which is not nicest, but working solution, and definitely it is better than nothing:

    debug 'org.apache.activemq.util.ThreadPoolUtils'

"Cannot find any VM in Java Home ..." error in Tomcat

During migration from JDK 6 to 7, I found strange issue: Tomcat was working fine when starting via catalina.sh, but was failing when starting via jsvc with just this error:

Cannot find any VM in Java Home /usr/lib/jvm/jdk1.7.0_10/

Jsvc recompilation or any other measure was not helping, but finally I found out that it was running flawless with specific x64 version of JDK instead of i586 version.

Running several Grails projects in parallel

If it is needed to run several Grails commands simultaneously (for example on Jenkins) there is big change to get exception like:

19:52:24    [groovyc] org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
19:52:24    [groovyc] /var/lib/jenkins/jobs/test/workspace/src/groovy/com/test/SomeUserDetails.groovy: 3: unable to resolve class org.codehaus.groovy.grails.plugins.springsecurity.GrailsUser

This happens because Grails stores plugin classes and other files in user home directory, so for example, parallel task cleans workspace while other task is running, latter is in trouble of not finding some plugin stuff.
Fortunately, this is easy to solve by specifying working directory for every specific command. There is special property called grails.project.work.dir which can be specified on command line, for example like:

grails -Dgrails.project.work.dir=/home/jenkins/job1/ clean --non-interactive

Javamelody SQL monitoring with Grails plugin

Javamelody is amazing tool for live monitoring. It has great Grails plugin which makes it very easy to use. Only issue I had with it is SQL monitoring configuration.
According to documentation there are several ways to configure data sources:

1. magic - you don't have to do anything, it just works
2. javamelody.datasources property in GrailsMelodyConfig.groovy - you just specify datasources
3. replacing JDBC driver with Javamelody proxy
4. specifying java option - javamelody.datasources

Unfortunately, all internal configuration options (1, 2, 3) didn't worked. As I investigated (quickly) it looks like there is no dataSource bean available for BeanPostProcessor and it tries to proxy after all connections were already picked by application.
So only specifying java command line option worked for me. Otherwise - great tool. It is also possible to specify multiple data sources with command line like:

-Djavamelody.datasources=java:comp/env/ds1,java:comp/env/ds2