Showing posts with label java. Show all posts
Showing posts with label java. Show all posts

Saturday, August 20, 2016

Converting byte array to string without loosing data


Recently I had to deal with poor Redis interface, that didn't allowed to store bytes and it was not possible to update it or change. Redis handles bytes perfectly for keys and data, but there was no way to pass it. Standard solution for this task is to use Base64, but for my case overhead was too big as I had to store hundreds of millions of records. UTF is not good too, as it is ruining data. Finally, I found that right encoding for this task is to use ISO-8859-1 encoding and it works just fine.


new String([10,20,-30,126] as byte[], "ISO-8859-1").getBytes("ISO-8859-1");





Saturday, July 9, 2016

URL canonicalization and normalization in Java

Recently I had to implement integration with Google Safe Browsing in Java and one part of the task is URL normalisation, basically it is like JSoup for URL. You should remove redundant parts, decode, encode, etc. Seems trivial: even java.net.URI has normalisation, but it really was not trivial, nothing was working and result was not even remotely compliant.

After searching and trying everything suggested on Stackoverflow, I finally found working solution - URL-Detector from Linkedin. Lib itself looks raw and it is not even in public Maven as of now, but it successfully passes all Google tests after replacing port and using URL without fragment.

Saturday, June 25, 2016

Quickly create groovy script in Intellij IDEA

It is sometimes frustrating when you need quickly check db, non-trivial http, or AWS with Groovy. It is fine and great if standard lib is enough for your case, but once you need new libraries, it is not that quick. Grab is amazing, but you need to specify dependency ids and recall exact API and settings. At some point I had a bunch of typical scripts with typical settings, but you have to be in same project to find them, remember not to commit them, and they tend to grow big and have their own lives at some point. Bad for hacking and it is not convenient for majority of one-off cases.

That is where I started using IntelliJ Live Templates: they are available in every project, easily configurable, you can dispose or save results as needed, they accessible from regular files, scratches, even Groovy console.

My typical template consists of everything needed for single use case: grabs, imports, initialisation and bunch of examples. It is always easier to slash unneeded stuff than recall all API specifics. For me, it is more like tldr output, than what typical default template looks like.

There is example of one template I use for AWS S3:


@Grab('com.amazonaws:aws-java-sdk:1.9.31')
import com.amazonaws.auth.BasicAWSCredentials
import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.services.s3.model.ObjectMetadata

def client = new AmazonS3Client(new BasicAWSCredentials('access', 'secret'))

println client.listObjects("bucket", "name").getObjectSummaries().collect{it.key}

def meta = new ObjectMetadata()
meta.setContentType("text/plain")
client.putObject("bucket", "name", new ByteArrayInputStream("1".bytes), meta)

client.deleteObject("bucket", "name")

println new String(client.getObject("bucket", "name").objectContent.bytes)

client.listVersions("bucket", "name").versionSummaries.each {
    println "${it.deleteMarker}   ${it.key}   ${it.size}"
}

client.deleteVersion("bucket", "name", "123")

And of course, you can have multiple templates preconfigured for different environments and use cases.

Biggest disadvantages comparing with regular files is that they are stored somewhere deeply in IntelliJ and not easily transferable, still they are easily exportable with other settings. Other disadvantage is that content is not searchable, so you have to pack all keywords as abbreviation or description, but anyway it is not part of universal search.

Saturday, May 7, 2016

Trim is not removing all whitespaces in Java

Java trim is removing only ASCII whitespace characters, but ignores unicode whitespaces. This is backward compatibility thing, and there is big and detailed explanation of this problem It can be easily fixed by using regular expression that will remove all official unicode whitespaces:


Pattern TRIM_PATTERN = Pattern.compile("^\\s*(.*?)\\s*$", Pattern.UNICODE_CHARACTER_CLASS);
Matcher matcher = TRIM_PATTERN.matcher(input);
if (matcher.matches() && matcher.groupCount() > 0) {
    return matcher.group(1);
}
return input;

But for more extreme cases you may want to use also this pattern

"^[\\s\\u2060\\u200D\\u200C\\u200B\\u180E\\uFEFF\\u00AD]*(.*?)[\\s\\u2060\\u200D\\u200C\\u200B\\u180E\\uFEFF\\u00AD]*$"

Saturday, March 5, 2016

XML parsing in Java

JSON is much popular now, but occasionally it is still possible to come across XML API. I had such experience recently and I have to say that in 2016 it is much easier than some 10 years ago. I mostly use Jackson for JSON, so for me best way to use it is XmlMapper plugin. After that it is plain Jackson.
There is example:


XmlMapper mapper = new XmlMapper();
List rates = mapper.readValue(ratesString, List.class);




compile 'com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.6.3'
compile 'org.codehaus.woodstox:woodstox-core-asl:4.4.1'





Friday, February 12, 2016

Using property files with map values in Spring configuration beans

Spring and Spring Boot can map property files as configuration beans automatically. What is less known is that it can easily wire Map objects too:


@Configuration
@Component
@PropertySource("classpath:config.properties")
public class Config {
    @Value("#{${my.map}}")
    private Map map;
...





my.map={\
'key1' : 'val1', \
'key2' : 'val2', \
...




Saturday, December 5, 2015

Returning first successful result from CompletableFuture

Recently I wanted to get first non empty result from bunch of CompletableFutures. For my case, allOf - is bad because it runs all futures till end and returns nothing. AnyOf - is bad because it returns first result, even if it is empty. So there is no such function in Java API, but it is still very easy to implement, and maybe it is even more logical, because actually there are 2 cases: found or not found; and it can be easily solved with two CompletableFutures. There is example:

import java.util.concurrent.CompletableFuture

success = new CompletableFuture()

def futures = (1..10).collect { CompletableFuture.supplyAsync {
                                    if (val == 6) {
                                        success.complete(val)
                                    } else {
                                        Thread.sleep(1000)
                                    }
                                }
                          }

println CompletableFuture.allOf(futures as CompletableFuture[])
        .thenApply({ throw new RuntimeException("not found") })
        .applyToEither(success, {it}).get()


This way it will return result if there is any, or throw exception if all done and there is no result, and it can be easily adjusted to return default value, for example.

Saturday, October 17, 2015

Trim is not removing all whitespaces in Java

Java trim is removing only ASCII whitespace characters, but ignores unicode whitespaces. This is backward compatibility thing, and there is big and detailed explanation of this problem. It can be easily fixed by using regular expression that will remove all official unicode whitespaces:
Pattern TRIM_PATTERN = Pattern.compile("^\\s*(.*?)\\s*$", Pattern.UNICODE_CHARACTER_CLASS);
Matcher matcher = TRIM_PATTERN.matcher(input);
if (matcher.matches() && matcher.groupCount() > 0) {
    return matcher.group(1);
}
return input;

But for more extreme cases you may want to use also this pattern
"^[\\s\\u2060\\u200D\\u200C\\u200B\\u180E\\uFEFF\\u00AD]*(.*?)[\\s\\u2060\\u200D\\u200C\\u200B\\u180E\\uFEFF\\u00AD]*$"

besides whitespaces it will also remove other invisible symbols.

Wednesday, September 16, 2015

Using IDE scripting console in Intellij IDEA

Some time ago, Intellij added option to write simple scripts to automate it's functions, extract information and write simple plugins. It is located under Tools / IDE scripting console. It is very basic now, just scripting file that runs inside of Intellij JVM and has access to Intellij API.

 There is simple Groovy example that automatically adds non-suspending breakpoint with log expression:

import com.intellij.openapi.vfs.LocalFileSystem
import com.intellij.openapi.fileEditor.FileDocumentManager
import com.intellij.openapi.project.ProjectManager
import com.intellij.xdebugger.breakpoints.SuspendPolicy

def file = LocalFileSystem.getInstance().findFileByPath("/users/test/project/src/Test.java")
def doc = FileDocumentManager.getInstance().getDocument(file)
def bp = ProjectManager.getInstance().getOpenProjects()[0].getComponent("DebuggerManager").getBreakpointManager().addLineBreakpoint(doc, 11)
bp.getXBreakpoint().setLogExpression('"test"')
bp.getXBreakpoint().setSuspendPolicy(SuspendPolicy.NONE)


Of course, this basic task can be done through XML configuration, but IDE scripting has access to much more and can do trickier tasks. There is very cool demo. Generally, from what I have seen, API is pretty impressive, it is intuitive, powerful and easy to start with.

My biggest complains are that it is pretty verbose (as Java itself), it would be great if in future versions it would be solved at least at tooling levels - with auto imports for packages, autocompletion, and more convenience functions, like attaching to actions, getting projects or files quickly. Second problem is total lack of documentation. Basically, what I have found is only GitHub repo of Intellij community edition. It has multiple modules with API scattered across them, most generic classes are under Core API.

Thursday, May 29, 2014

Grabing JavaScript libraries via WebJars in Spring Boot

WebJars is nice and familiar way to work with JavaScript dependencies from Java projects. Spring Boot supports it, and it is pretty easy to add them for Groovy projects, just with good old Grab annotation, like:

@Grab("org.webjars:canjs:2.0.2")

Then all CanJS files become available under /webjars/canjs/2.0.2/. If not sure, list of files can be looked up at WebJars website.

Thursday, April 3, 2014

MeteorJS like application in Grails

MeteorJS is new web framework that combines server-side and client-side programming into one, leaving mostly just client-side. I am defenitely not expert with tool and just seen demo on their homepage, but WOW effect is sure impressive. It suppose to kill all Rails and server-side development in general. Thank god I am full-stack developer, but anyway I am scared.

General idea behind this framework is autogenerating all server-side code, leaving mostly just configuration. Similar feature was recently introduced in Grails, so I was wondering - is it possible to create something similar to MeteorJS in Grails. And there is my experience.

Basically, it needs autogenerated REST, Javascript ORM and server push. REST is Grails feature since version 2.3; for client-side ORM I picked CanJS as it looks most similar to Meteor; for server push in Grails I picked spring-websocket plugin (because it is first result in Google, why else?). Unfortunately, spring-websocket is M1 and it relies on Spring 4, so Grails is 2.4M1 too, so it is not production safe, but MeteorJS is 0.8.0 now too, so we are even.

Application idea is simple TODO app. You can find full source here. There is defenitely some boilerplate code as it is not plugin, but I think in general it is pretty similar to MeteorJS example. It consists of 3 meaningful components: view, controller and domain object.

View is Mustache template, and is pretty straightforward if you are familiar with CanJS:

<h1>test</h1>
<table>
{{#todos}}
  <tr>
    <td>{{description}}</td><td><input type="checkbox" can-value="done"></td>
    <td><input type="button" value="Delete" can-click="delete"></td>
    </tr>
{{/todos}}
</table>

Description
<input type="text" can-value="description">
<input type="button" can-click="add" value="Add">

Controller is CanJS component with ORM definition:

var Todo = can.Model({
  findAll: function(id)   {return $.get('/grails-meteor-example/todos.json/')},
  findOne: function(id)   {return $.get('/grails-meteor-example/todos.json/'+id)},
  create:  function(data) {return $.post('/grails-meteor-example/todos.json/', data)},
  update:  function(id)   {return $.ajax({type: 'PUT',    url: '/grails-meteor-example/todos.json/'+id })},
  destroy: function(id)   {return $.ajax({type: 'DELETE', url: '/grails-meteor-example/todos/'+id, contentType: 'application/json' })}
}, {});

can.Component.extend({
  tag: "todos",
  template: can.view("todos.mustache"),
  scope: {
    todos: new Todo.List({}),
    description: can.compute(''),
    add: function() {
      new Todo({'description':this.description(), 'done':false}).save();
      this.description('');
    },
    delete: function(todo) {
      todo.destroy();
    }
  },
  events: {
    "{todo} change": function(){
      this.scope.attr('todos', new Todo.List({}));
    }
  }
});

$(document).ready(function() {
  $("body").html( can.view.mustache("<todos></todos>"));
});


Domain is Grails domain object mapped as REST controller:

package org.grmeteor

import grails.rest.Resource

@Resource(uri='/todos')
class Todo {
  String description
  boolean done
}


View is totally similar to MeteorJS. Controller is similar, but also contains ORM definition and server push event. Both could be easily autogenerated with Grails plugin, but at least push can be also left for practical applications (probably, you will not want to have notification with millions of parallel users). Domain is pure Grails overhead, but it would probably still be needed for SQL backends in MeteorJS (and hey, those server-side devs want to eat too!).

Of course there are other files too, like server push sender and client side HTML container for CanJS, but these are mostly static and sure can be easily hidden by plugin. So only part that is missing to make Grails MeteorJS-cool is plugin itself, which even if not for practical cases could be nice to have for WOW effect and MVPs.

Monday, February 24, 2014

Setting different JAVA_HOME for Gradle in IntelliJ IDEA

There are a lot of different options to set JDK folder in Gradle, but in IntelliJ the only option that worked for me was adding this line:

gradle.java.home=C:/Program Files/Java/jdk1.8.0/

into idea.properties file.

Thursday, February 13, 2014

Storing sessions in Redis with Spring Boot

Tomcat has nice support to use Redis for session replication with this awesome library. However Spring Boot launches embedded Tomcat, so there is no traditional XML configuration, it is still super easy to change default manager to use Redis for session replication, just define containerCustomizer bean, like:

@Configuration
@EnableAutoConfiguration
@ComponentScan
public class App {
  public App() {
  }

  @Bean
  public EmbeddedServletContainerCustomizer containerCustomizer(){
    return factory -> {
      TomcatEmbeddedServletContainerFactory containerFactory = (TomcatEmbeddedServletContainerFactory) factory;
      containerFactory.addContextValves(new RedisSessionHandlerValve());
      containerFactory.setTomcatContextCustomizers(Arrays.asList(context -> {
        context.setSessionTimeout(30);
        context.setManager(new RedisSessionManager(){{
          setHost("redis.server.com");
        }});
      }));
    };
  }

  public static void main(String[] args) throws Exception {
    SpringApplication.run(App.class, args);
  }
}

(this is syntax with new shiny Java 8 lambdas, but with few additional boring types it should compile in old Javas too).

Thursday, November 21, 2013

Script to parse groovy source code

Recently, I had to parse Groovy class to extract some information. Reflection was not good as it was not easy to get all dependencies and I had to preserve comments too. This is quite easy to do by utilizing Groovydoc internals which are part of the default libraries:


import antlr.collections.AST
import org.codehaus.groovy.antlr.AntlrASTProcessor
import org.codehaus.groovy.antlr.SourceBuffer
import org.codehaus.groovy.antlr.UnicodeEscapingReader
import org.codehaus.groovy.antlr.parser.GroovyLexer
import org.codehaus.groovy.antlr.parser.GroovyRecognizer
import org.codehaus.groovy.antlr.treewalker.SourceCodeTraversal
import org.codehaus.groovy.tools.groovydoc.SimpleGroovyClassDoc
import org.codehaus.groovy.tools.groovydoc.SimpleGroovyClassDocAssembler

def reader = new File("/path/to/package/org/Groovy.groovy").newReader()
SourceBuffer sourceBuffer = new SourceBuffer()
UnicodeEscapingReader unicodeReader = new UnicodeEscapingReader(reader, sourceBuffer)
GroovyLexer lexer = new GroovyLexer(unicodeReader)
unicodeReader.setLexer(lexer)
GroovyRecognizer parser = GroovyRecognizer.make(lexer)
parser.setSourceBuffer(sourceBuffer)
parser.compilationUnit()
AST ast = parser.getAST()

def visitor = new SimpleGroovyClassDocAssembler("/path/to/package", "org/Groovy.groovy", sourceBuffer, [], new Properties(), true)
AntlrASTProcessor traverser = new SourceCodeTraversal(visitor)
traverser.process(ast)
SimpleGroovyClassDoc doc = (visitor.getGroovyClassDocs().values() as List)[0]
doc.methods().each {
  println it.name()
  println it.commentText()
  println it.annotations()
}


This is definitely not nicest Groovy code, and looks more like Java, but gets the job done. By the way, it should be possible to parse Java too, though I didn't tried to.

Thursday, September 26, 2013

Convert color console output into HTML with Java

Some command line tools can print colored text, which looks nice in console itself (if it is supported), but not so nice when you need to show it in HTML:

[47;32mhello world

There is small console tool "aha" that can pipe such input and transform it directly in command line, but sometimes you have to do it from code. For such cases there is Java library Jansi which prints ANSI escape sequences into Windows consoles. Their less advertised feature is library for direct conversion between ANSI and HTML, which it does just fine:

@Grapes(
    @Grab(group='org.fusesource.jansi', module='jansi', version='1.11')
)

import org.fusesource.jansi.*

private String colorize(String text) throws IOException {
  new ByteArrayOutputStream().with {
    new HtmlAnsiOutputStream(it).with {
      write(text.getBytes("UTF-8"))
      close()
    }
    return new String(it.toByteArray(), "UTF-8");
  }
}

println colorize(" [47;32mhello world")

Wednesday, August 28, 2013

Jenkins kills Tomcat or other Java process after job ends

Recently, I was adding new application deployment to Tomcat server that was started by Jenkins. Process was starting nicely, but was stopping at the end of the job. The only clue was log record:

Service exit with a return value of 143

This issue can be easily solved by adding -Xrs to JAVA_OPTS. Option description can be found at http://docs.oracle.com/javase/6/docs/technotes/tools/windows/java.html.


Thursday, June 20, 2013

Multipart file upload in Groovy HTTPBuilder

HttpClient from HttpComponents is very powerful, but super ugly library. Groovy has nice wrapper around it, called HTTPBuilder, which looks better and has DSLish API. It is just as powerful as HttpClient, as it is using it underneath. Unfortunately, it has far less documentation and bad googleability.
For example I tried to use it for file upload, and it is not in original documentation, and nothing sane on StackOverflow or likes. Fortunately, it is still possible and easy (despite HttpClient ugliness creeping in).

    def http = new HTTPBuilder( 'http://localhost:8080/' )
    http.auth.basic 'test', '123456'
    http.request( POST, JSON ) { req ->
      uri.path = '/project/addFile'
      uri.query = [ id:projectId ]
      requestContentType = 'multipart/form-data'
      MultipartEntity entity = new MultipartEntity()
      entity.addPart("file", new ByteArrayBody("123456".getBytes("UTF-8"), 'filename.jpg'))
      req.entity = entity
    }



Wednesday, April 24, 2013

ChannelResolutionException in Spring Integration with Groovy

Recently, after launching Spring Integration beans I got weird exception:
ChannelResolutionException: no output-channel or replyChannel header available
After some googling I found out that this error means that Spring Integration tries to configure reply channel, which in my case meant that I am replying into channel-adapter, which is not possible. So basically, problem was that my service method was defined as:
def myServiceMethod()

which Spring understands as it returns something, even if it does not. So this should be told explicitly, like:
void myServiceMethod()

and this is it.

Tuesday, April 2, 2013

Run background java process from Jenkins

Jenkins is using mechanism to detect spawned processes and kill them after finishing job, so if you will just start such ad-hoc background daemon process, Jenkins will kill it with warning:


Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information

Fortunately, there is easy hack to tell Jenkins ignore such process, just create fake BUILD_ID and start your application, for example, this can be entered into Execute shell build step:

BUILD_ID=dontKillMe nohup java -jar test.jar &

Warning is still there, but process should be fine and running. There is more about it here.

Thursday, January 17, 2013

"Cannot find any VM in Java Home ..." error in Tomcat

During migration from JDK 6 to 7, I found strange issue: Tomcat was working fine when starting via catalina.sh, but was failing when starting via jsvc with just this error:

Cannot find any VM in Java Home /usr/lib/jvm/jdk1.7.0_10/

Jsvc recompilation or any other measure was not helping, but finally I found out that it was running flawless with specific x64 version of JDK instead of i586 version.