Groovy framework for AWS lambda

I was playing with AWS lambda recently and I find it pretty exciting. It is cheap, does not require any maintenance and it is simple. The biggest problem I found so far, is lack of tooling. AWS has web UI which does what it supposed to do, but is far from pleasant. AWS command line tools and API are much better, but are AWS centric. So, I have created tool to work with AWS lambda. It uses some convention now, but can be adjusted to whatever feels right for specific situation, idea is that it is just one script with few lines of code. Basically, it does 2 things: uploads application and local testing. I have created small example that illustrates how and what it does.

Example is trivial application that increments number by clicking on button. It is SPA, with static client in S3 and two API functions add and get.

All lambda code is just single Groovy script. It consists of router function, dependencies and logic. Router function is what will be mapped to lambda. It is function that accepts map and returns map. Input map can be plain request JSON, or some additional parameters can be injected at API gateway. Even for this trivial case it was a good idea and I injected configuration parameters - AWS access and secret keys and region for DynamoDB access, so they are hidden at admin level and are environment specific. Simplest way to do routing is just by using some designated parameter, in this example, it is called "function". Doing it with real URI will need AWS configuration, and it is not very convenient and flexible.

In this simple example, router script does logic, but for more complex cases it is better to be just router and logic would be in dependencies.

During packaging Groovy script is compiled and packaged together with all dependencies into uberjar which is required by AWS. After that, it is uploaded via AWS API.

Development can be done and tested locally by using mock server. Mock server mimics AWS, parses input into JSON like lambda, injects parameters like API gateway, includes Access-Control-Allow-Origin and separates configuration via config.js, supports debug and reloading.

Using IDE scripting console in Intellij IDEA

Some time ago, Intellij added option to write simple scripts to automate it's functions, extract information and write simple plugins. It is located under Tools / IDE scripting console. It is very basic now, just scripting file that runs inside of Intellij JVM and has access to Intellij API.

 There is simple Groovy example that automatically adds non-suspending breakpoint with log expression:

import com.intellij.openapi.vfs.LocalFileSystem
import com.intellij.openapi.fileEditor.FileDocumentManager
import com.intellij.openapi.project.ProjectManager
import com.intellij.xdebugger.breakpoints.SuspendPolicy

def file = LocalFileSystem.getInstance().findFileByPath("/users/test/project/src/Test.java")
def doc = FileDocumentManager.getInstance().getDocument(file)
def bp = ProjectManager.getInstance().getOpenProjects()[0].getComponent("DebuggerManager").getBreakpointManager().addLineBreakpoint(doc, 11)
bp.getXBreakpoint().setLogExpression('"test"')
bp.getXBreakpoint().setSuspendPolicy(SuspendPolicy.NONE)

Of course, this basic task can be done through XML configuration, but IDE scripting has access to much more and can do trickier tasks. There is very cool demo. Generally, from what I have seen, API is pretty impressive, it is intuitive, powerful and easy to start with.

My biggest complains are that it is pretty verbose (as Java itself), it would be great if in future versions it would be solved at least at tooling levels - with auto imports for packages, autocompletion, and more convenience functions, like attaching to actions, getting projects or files quickly. Second problem is total lack of documentation. Basically, what I have found is only GitHub repo of Intellij community edition. It has multiple modules with API scattered across them, most generic classes are under Core API.

Script that logs time to JIRA from Git commit messages

If you need to report work time in JIRA and native comment hashes are not suitable for you, there is simple Groovy script that does the same and can be adjusted to particular scenario or configuration:

@GrabResolver(name='test', root='https://maven.atlassian.com/content/groups/public/')
@Grab( group ='com.atlassian.jira.plugins', module = 'jira-soapclient', version = '4.4' )
@Grab(group='org.eclipse.jgit', module='org.eclipse.jgit', version='3.3.2.201404171909-r')

import com.atlassian.jira.rpc.soap.client.JiraSoapServiceServiceLocator
import com.atlassian.jira.rpc.soap.client.RemoteWorklog
import groovy.transform.TailRecursive
import org.eclipse.jgit.lib.Repository
import org.eclipse.jgit.revwalk.RevCommit
import org.eclipse.jgit.revwalk.RevWalk
import org.eclipse.jgit.storage.file.FileRepositoryBuilder
import java.text.SimpleDateFormat
import static java.util.Calendar.*

@TailRecursive
static Set<RevCommit> subcommits(RevCommit commit, int start, RevWalk walk, def all) {
  if (commit.commitTime > start) {
    all << commit
    commit.parents.collect{ walk.parseCommit(it) }.each {
      subcommits(it, start, walk, all)
    }
  }
  return all
}

Repository repository = new FileRepositoryBuilder().setGitDir(new File(args[3])).setMustExist(true).build()
RevWalk walk = new RevWalk(repository)

def formattime = new SimpleDateFormat('yyyy-MM-dd')
formattime.setTimeZone(TimeZone.getTimeZone("UTC"))
Date start = formattime.parse(args[4])
Date end = formattime.parse(args[5])

def all = subcommits(walk.parseCommit(repository.resolve("HEAD")), start.time / 1000L as int, walk, new HashSet())
def mine = all.findAll{ it.authorIdent.name == args[6] && it.fullMessage =~ /^PRJ-[0-9]+/ }

def actual = mine.min{ it.commitTime }

def tasks = (start..end).findAll{!(it[DAY_OF_WEEK] in [SUNDAY, SATURDAY])}.collect {date ->
  actual = mine.find{ date.time / 1000L <= it.commitTime && (date + 1).time / 1000L > it.commitTime } ?: actual
  [date:date, task:(actual.fullMessage =~ /(PRJ-[0-9]+)/)[0][1]]
}

if (args[0] == 'test') {
  tasks.each { println "${it.date} - ${it.task}" }
} else if (args[0] == 'prod') {
  def jiraSoapService = new JiraSoapServiceServiceLocator().getJirasoapserviceV2(new URL('http://jira.com/rpc/soap/jirasoapservice-v2'))
  String token = jiraSoapService.login(args[1], args[2])
  tasks.each { jiraSoapService.addWorklogAndRetainRemainingEstimate(token, it.task, new RemoteWorklog(timeSpent: "8h", startDate:it.date.toCalendar())) }
}

It uses a bunch of command line parameters - first is either prod or test; test just prints tasks, prod sends them to JIRA. There is an example CLI command:

groovy gittotempo.groovy prod jirauser jirapassword c:\work\project\.git 2014-08-01 2014-08-31 gituser

MeteorJS like application in Grails

MeteorJS is new web framework that combines server-side and client-side programming into one, leaving mostly just client-side. I am defenitely not expert with tool and just seen demo on their homepage, but WOW effect is sure impressive. It suppose to kill all Rails and server-side development in general. Thank god I am full-stack developer, but anyway I am scared.

General idea behind this framework is autogenerating all server-side code, leaving mostly just configuration. Similar feature was recently introduced in Grails, so I was wondering - is it possible to create something similar to MeteorJS in Grails. And there is my experience.

Basically, it needs autogenerated REST, Javascript ORM and server push. REST is Grails feature since version 2.3; for client-side ORM I picked CanJS as it looks most similar to Meteor; for server push in Grails I picked spring-websocket plugin (because it is first result in Google, why else?). Unfortunately, spring-websocket is M1 and it relies on Spring 4, so Grails is 2.4M1 too, so it is not production safe, but MeteorJS is 0.8.0 now too, so we are even.

Application idea is simple TODO app. You can find full source here. There is defenitely some boilerplate code as it is not plugin, but I think in general it is pretty similar to MeteorJS example. It consists of 3 meaningful components: view, controller and domain object.

View is Mustache template, and is pretty straightforward if you are familiar with CanJS:

<h1>test</h1>
<table>
{{#todos}}
  <tr>
    <td>{{description}}</td><td><input type="checkbox" can-value="done"></td>
    <td><input type="button" value="Delete" can-click="delete"></td>
    </tr>
{{/todos}}
</table>

Description
<input type="text" can-value="description">
<input type="button" can-click="add" value="Add">

Controller is CanJS component with ORM definition:

var Todo = can.Model({
  findAll: function(id)   {return $.get('/grails-meteor-example/todos.json/')},
  findOne: function(id)   {return $.get('/grails-meteor-example/todos.json/'+id)},
  create:  function(data) {return $.post('/grails-meteor-example/todos.json/', data)},
  update:  function(id)   {return $.ajax({type: 'PUT',    url: '/grails-meteor-example/todos.json/'+id })},
  destroy: function(id)   {return $.ajax({type: 'DELETE', url: '/grails-meteor-example/todos/'+id, contentType: 'application/json' })}
}, {});

can.Component.extend({
  tag: "todos",
  template: can.view("todos.mustache"),
  scope: {
    todos: new Todo.List({}),
    description: can.compute(''),
    add: function() {
      new Todo({'description':this.description(), 'done':false}).save();
      this.description('');
    },
    delete: function(todo) {
      todo.destroy();
    }
  },
  events: {
    "{todo} change": function(){
      this.scope.attr('todos', new Todo.List({}));
    }
  }
});

$(document).ready(function() {
  $("body").html( can.view.mustache("<todos></todos>"));
});

Domain is Grails domain object mapped as REST controller:

package org.grmeteor

import grails.rest.Resource

@Resource(uri='/todos')
class Todo {
  String description
  boolean done
}

View is totally similar to MeteorJS. Controller is similar, but also contains ORM definition and server push event. Both could be easily autogenerated with Grails plugin, but at least push can be also left for practical applications (probably, you will not want to have notification with millions of parallel users). Domain is pure Grails overhead, but it would probably still be needed for SQL backends in MeteorJS (and hey, those server-side devs want to eat too!).

Of course there are other files too, like server push sender and client side HTML container for CanJS, but these are mostly static and sure can be easily hidden by plugin. So only part that is missing to make Grails MeteorJS-cool is plugin itself, which even if not for practical cases could be nice to have for WOW effect and MVPs.

Hosting Ivy or Maven repository on Github with Gradle

If you don't have access to any public repo, but want to host libraries - then Github can be a solution.
Unfortunately, it does not provide proper interfaces for the client applications, but this is still possible.

Lets say that local root folder of your git project is /test.
First, you need to create all necessary poms and jars. To do it, you have to add this to your build.gradle file:

uploadArchives {
  repositories {
    mavenDeployer {
      repository(url: "file:///test/repo")
    }
  }
}

URL property is for proper Maven repository - for now unfortunately, Github is not, so you will have to enter some local folder. After this is added, run Gradle task uploadArchives, then commit and push all output files with git.

Unfortunately, this is not all, as SHA signature is wrong, because your git URL is not same as local. So you will need to find out proper SHA, and until you want to calculate it yourself, one easy way to do it is find some talkative Maven client, for example Grape. Luckily it is part of normal Groovy installation.

Then, you need to add new repo into Grape configuration, open ${user.home}/.groovy/grapeConfig.xml file and add your new repo like

<ibiblio name="myrepo" root="https://raw.github.com/me/mymodule/master/repo" m2compatible="true"/>

And, to get dependencies, just run Grape in verbose mode, for example like:

grape -V install my.group mymodule 0.1

It will fail, but you will get one great error message like:

...-0.1.pom: invalid sha1: expected=7f1089041d63ce7eaa5d6a35ddda3aaa606042e3 computed=8a941ef40645d449b0740023624c2f8b67b84c59 (516ms)

8a941ef40645d449b0740023624c2f8b67b84c59 is what we were looking for.
Just find file whose signature was incorrect, for example for pom it can be mymodule.pom.sha1, update it with new signature and push into git.

That is all, you have new great Maven repository with root under https://raw.github.com/me/mymodule/master/repo.

grails-coffeescript-resources is not working after deploying war in production

There is great Grails plugin that allows to work with CoffeeScript very easily, compiles .coffee files on the fly and does all dirty work.
However, once I tried to deploy my application to test environment, all CoffeeScript files became available in browser as is, without compilation. After some investigation I found out that problem is that it does not supports applications deployed as packed WAR files (with unpackWARs="false" in server.xml), it was pretty easy to fix by modifying plugin script grails-app/resourceMappers/CoffeeScriptResourceMapper.groovy by changing:
 

      File input = grailsApplication.parentContext.getResource(resource.sourceUrl).file

to this

      File input
      try {
        input = grailsApplication.parentContext.getResource(resource.sourceUrl).file
      } catch (FileNotFoundException e) {
        input = new File(original.absolutePath)
      }

and it should work.
One problem is that this change is inside plugin, so you can't do change in your code, and should change plugin itself. I submitted fix with pull request on GitHub, hopefully it will be merged. Until then it is possible to compile your own version from here, or just replace this one line on build server.

Local git setup error

Recently I decided to check GIT. I am old fan of Tortoise tools and I decided to start with TortoiseGit. It is nice tool with familiar interface.

I am a little familiar with Mercurial and heard that they are quite similar in concepts. So I started by creating repository by command “Git create repository here”.
Then I created “Git Clone” of my repository in new folder. Then I started to play and created new files and commited changes. But then, when I tried to push my changes to master repository I got big ugly error:

git.exe push    "origin" master:master

remote: error: refusing to update checked out branch: refs/heads/master        
remote: error: By default, updating the current branch in a non-bare repository        
remote: error: is denied, because it will make the index and work tree inconsistent        
remote: error: with what you pushed, and will require 'git reset --hard' to match        
remote: error: the work tree to HEAD.        
remote: error:         
remote: error: You can set 'receive.denyCurrentBranch' configuration variable to        
remote: error: 'ignore' or 'warn' in the remote repository to allow pushing into        
remote: error: its current branch; however, this is not recommended unless you        
remote: error: arranged to update its work tree to match what you pushed in some        
remote: error: other way.        
remote: error:         
remote: error: To squelch this message and still keep the default behaviour, set        
remote: error: 'receive.denyCurrentBranch' configuration variable to 'refuse'.        
To C:\########
 ! [remote rejected] master -> master (branch is currently checked out)
error: failed to push some refs to 'C:\########'

This was quite confusing and error is not helping much. Also this scenario was working nicely with Mercurial, so I was really confused. But after some googling I found really nice article about git at http://cworth.org/hgbook-git/tour/, and it is explained nicely how to create repository that can be used with “push”. You just have to create it with

git --bare init --shared

It is really strange that there is no any option that can be set while creating repository and you still have to do it command line way. But issue was solved.