Skip to content
BranchesTags

Repository files navigation

jackson-dataformat-msgpack

This Jackson 3.x extension library is a component to easily read and write MessagePack encoded data through jackson-databind API.

It extends standard Jackson streaming API (JsonFactory, JsonParser, JsonGenerator), and as such works seamlessly with all the higher level data abstractions (data binding, tree model, and pluggable extensions).

Requirements: Java 17+ and Jackson 3.x. For Jackson 2.x, use org.msgpack:jackson-dataformat-msgpack from msgpack-java instead.

Why this project was resumed

This repository is where jackson-dataformat-msgpack started before it moved into msgpack-java as msgpack-jackson. It was picked up again for Jackson 3, as a rewrite that encodes and decodes MessagePack on Jackson's own buffers instead of through msgpack-core:

  • Jackson 3 is a new major version with a new API, and a standalone module can follow its release cadence.
  • msgpack-core has its own buffer layer, so every value crossed two buffer layers. Working directly on Jackson's buffers, as Jackson's own CBOR and Smile modules do, removed that overhead (writes +66%, reads +17% on the same POJO, see jmh/results/) and brought Jackson's read constraints, name canonicalization and buffer pooling to MessagePack.

How it works inside is described in docs/DESIGN.md.

Install

Maven

<dependency>
  <groupId>org.komamitsu</groupId>
  <artifactId>jackson-dataformat-msgpack</artifactId>
  <version>3.0.0</version>
</dependency>

Sbt

libraryDependencies += "org.komamitsu" % "jackson-dataformat-msgpack" % "3.0.0"

Gradle

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.komamitsu:jackson-dataformat-msgpack:3.0.0'
}

Basic usage

Serialization/Deserialization of POJO

All you need is a MessagePackMapper, which is a tools.jackson.databind.ObjectMapper for MessagePack. You can then use it for MessagePack data in the same way as jackson-databind.

// Instantiate ObjectMapper for MessagePack
ObjectMapper objectMapper = new MessagePackMapper();

// Serialize a Java object to byte array
ExamplePojo pojo = new ExamplePojo("komamitsu");
byte[] bytes = objectMapper.writeValueAsBytes(pojo);

// Deserialize the byte array to a Java object
ExamplePojo deserialized = objectMapper.readValue(bytes, ExamplePojo.class);
System.out.println(deserialized.getName()); // => komamitsu

We strongly recommend calling MessagePackMapper.Builder#handleBigIntegerAndBigDecimalAsString() if you serialize and/or deserialize BigInteger/BigDecimal values. See Serialize and deserialize BigDecimal as str type internally in MessagePack format for details.

ObjectMapper objectMapper = MessagePackMapper.builder().handleBigIntegerAndBigDecimalAsString().build();

Serialization/Deserialization of List

// Instantiate ObjectMapper for MessagePack
ObjectMapper objectMapper = new MessagePackMapper();

// Serialize a List to byte array
List<Object> list = new ArrayList<>();
list.add("Foo");
list.add("Bar");
list.add(42);
byte[] bytes = objectMapper.writeValueAsBytes(list);

// Deserialize the byte array to a List
List<Object> deserialized = objectMapper.readValue(bytes, new TypeReference<List<Object>>() {});
System.out.println(deserialized); // => [Foo, Bar, 42]

Serialization/Deserialization of Map

// Instantiate ObjectMapper for MessagePack
ObjectMapper objectMapper = new MessagePackMapper();

// Serialize a Map to byte array
Map<String, Object> map = new HashMap<>();
map.put("name", "komamitsu");
map.put("age", 42);
byte[] bytes = objectMapper.writeValueAsBytes(map);

// Deserialize the byte array to a Map
Map<String, Object> deserialized = objectMapper.readValue(bytes, new TypeReference<Map<String, Object>>() {});
System.out.println(deserialized); // => {name=komamitsu, age=42}

Example of Serialization/Deserialization over multiple languages

Java

// Serialize
Map<String, Object> obj = new HashMap<String, Object>();
obj.put("foo", "hello");
obj.put("bar", "world");
byte[] bs = objectMapper.writeValueAsBytes(obj);
// bs => [-126, -93, 102, 111, 111, -91, 104, 101, 108, 108, 111,
//        -93, 98, 97, 114, -91, 119, 111, 114, 108, 100]

Ruby

require 'msgpack'

# Deserialize
xs = [-126, -93, 102, 111, 111, -91, 104, 101, 108, 108, 111,
      -93, 98, 97, 114, -91, 119, 111, 114, 108, 100]
MessagePack.unpack(xs.pack("C*"))
# => {"foo"=>"hello", "bar"=>"world"}

# Serialize
["zero", 1, 2.0, nil].to_msgpack.unpack('C*')
# => [148, 164, 122, 101, 114, 111, 1, 203, 64, 0, 0, 0, 0, 0, 0, 0, 192]

Java

// Deserialize
bs = new byte[] {(byte) 148, (byte) 164, 122, 101, 114, 111, 1,
                 (byte) 203, 64, 0, 0, 0, 0, 0, 0, 0, (byte) 192};
TypeReference<List<Object>> typeReference = new TypeReference<List<Object>>(){};
List<Object> xs = objectMapper.readValue(bs, typeReference);
// xs => [zero, 1, 2.0, null]

Advanced usage

Serialize multiple values without closing an output stream

tools.jackson.databind.ObjectMapper closes an output stream by default after it writes a value. If you want to serialize multiple values in a row without closing an output stream, disable StreamWriteFeature.AUTO_CLOSE_TARGET.

OutputStream out = new FileOutputStream(tempFile);
ObjectMapper objectMapper = MessagePackMapper.builder()
        .disable(StreamWriteFeature.AUTO_CLOSE_TARGET)
        .build();

objectMapper.writeValue(out, 1);
objectMapper.writeValue(out, "two");
objectMapper.writeValue(out, 3.14);
out.close();

The file now holds three MessagePack values back to back. The next section shows how to read them.

Deserialize multiple values without closing an input stream

tools.jackson.databind.ObjectMapper closes an input stream by default after it reads a value. If you want to deserialize multiple values in a row without closing an input stream, disable StreamReadFeature.AUTO_CLOSE_SOURCE.

// tempFile holds the three values written in the previous section
FileInputStream in = new FileInputStream(tempFile);
ObjectMapper objectMapper = MessagePackMapper.builder()
        .disable(StreamReadFeature.AUTO_CLOSE_SOURCE)
        .build();
System.out.println(objectMapper.readValue(in, Integer.class));   // => 1
System.out.println(objectMapper.readValue(in, String.class));    // => two
System.out.println(objectMapper.readValue(in, Double.class));    // => 3.14
in.close();

In this mode the parser reads exactly the bytes of the value it returns, so the stream is left positioned at the next value.

Serialize not using str8 type

Old msgpack-java (e.g 0.6.7) doesn't support MessagePack str8 type. When your application needs to communicate with such an old MessagePack library, you can disable the data type like this:

ObjectMapper objectMapper = new MessagePackMapper(new MessagePackFactory().setStr8FormatSupport(false));
// A string of 32 to 255 bytes is serialized as str16 instead of str8
byte[] resultWithoutStr8Format = objectMapper.writeValueAsBytes(str8LengthString);

Serialize integer map keys as MessagePack integers

Map keys are written as strings, the same text Jackson writes for JSON, so every key reads back through its usual KeyDeserializer. To write Integer, Long, Short and Byte keys as MessagePack integers instead, enable integer keys:

ObjectMapper objectMapper = new MessagePackMapper(
        new MessagePackFactoryBuilder().supportIntegerKeys(true).build());

byte[] bytes = objectMapper.writeValueAsBytes(Collections.singletonMap(42, "Hello"));   // {42: "Hello"}

Map<Integer, String> deserialized = objectMapper.readValue(bytes, new TypeReference<Map<Integer, String>>() {});
System.out.println(deserialized);   // => {42=Hello}

Other key types stay strings either way.

Use an object as a map key

A key of any other type is written as the same string Jackson writes for JSON: the value of its @JsonKey or @JsonValue accessor, otherwise its toString(). The mapper must be able to build the type back from that string: through a single-String constructor, a static valueOf(String) or fromString(String), a @JsonCreator factory taking one String, a @JsonDeserialize(keyUsing = ...) on the class or on the map property, or a KeyDeserializer registered on the mapper. Otherwise writing the key fails with an InvalidDefinitionException, because it could not be read back.

public class UserId {
    private final String value;

    @JsonCreator
    public UserId(String value) { this.value = value; }

    @JsonKey
    public String value() { return value; }

    // equals() and hashCode() on value
}

ObjectMapper objectMapper = new MessagePackMapper();

byte[] bytes = objectMapper.writeValueAsBytes(Collections.singletonMap(new UserId("u-1"), "Alice"));   // {"u-1": "Alice"}

Map<UserId, String> deserialized = objectMapper.readValue(bytes, new TypeReference<Map<UserId, String>>() {});

For the same reason a map, a collection or an array (other than byte[]) cannot be a key. Jackson's JSON output writes such keys as their toString() and fails only when reading them; this library refuses them when writing. A key serializer you register yourself is not checked, so pair it with a matching KeyDeserializer.

This check, like the integer handling of Short and Byte keys, is part of MessagePackMapper; a plain ObjectMapper built on MessagePackFactory does not have it.

UUID values

MessagePackMapper writes a UUID value as its 36-character string, which reads back as the same UUID. Jackson would otherwise write it as 16 bytes of binary, which this library does not read back as a UUID.

Serialize and deserialize BigDecimal as str type internally in MessagePack format

By default, for backward compatibility, a BigDecimal is written as a MessagePack integer if it has no fractional part, and otherwise as a float64 if that represents it exactly. A value that fits neither (too many digits for a double, or a magnitude beyond 64-bit integers) fails with IllegalArgumentException. So we strongly recommend calling MessagePackMapper.Builder#handleBigIntegerAndBigDecimalAsString() to internally handle BigDecimal values as String.

ObjectMapper objectMapper = MessagePackMapper.builder().handleBigIntegerAndBigDecimalAsString().build();

Pojo obj = new Pojo();
// This value is too large to be serialized as double
obj.value = new BigDecimal("1234567890.98765432100");

byte[] converted = objectMapper.writeValueAsBytes(obj);

System.out.println(objectMapper.readValue(converted, Pojo.class));   // => Pojo{value=1234567890.98765432100}

MessagePackMapper.Builder#handleBigIntegerAndBigDecimalAsString() is equivalent to the following configuration.

ObjectMapper objectMapper = MessagePackMapper.builder()
        .withConfigOverride(BigInteger.class,
                o -> o.setFormat(JsonFormat.Value.forShape(JsonFormat.Shape.STRING)))
        .withConfigOverride(BigDecimal.class,
                o -> o.setFormat(JsonFormat.Value.forShape(JsonFormat.Shape.STRING)))
        .build();

Serialize and deserialize Instant instances as MessagePack extension type

timestamp extension type is defined in MessagePack as type:-1. Registering TimestampExtensionModule.INSTANCE module enables automatic serialization and deserialization of java.time.Instant to/from the MessagePack extension type.

ObjectMapper objectMapper = MessagePackMapper.builder()
        .addModule(TimestampExtensionModule.INSTANCE)
        .build();
Pojo pojo = new Pojo();
// The type of `timestamp` variable is Instant
pojo.timestamp = Instant.now();
byte[] bytes = objectMapper.writeValueAsBytes(pojo);

// The Instant instance is serialized as MessagePack extension type (type: -1)

Pojo deserialized = objectMapper.readValue(bytes, Pojo.class);
System.out.println(deserialized);   // "2022-09-14T08:47:24.922Z"

Deserialize extension types with ExtensionTypeCustomDeserializers

ExtensionTypeCustomDeserializers helps you to deserialize your own custom extension types easily.

Deserialize extension type value directly

// In this application, extension type 59 is used for byte[]
byte[] bytes = new MessagePackMapper().writeValueAsBytes(new MessagePackExtensionType((byte) 59, hexspeak));

// Register the type and a deserializer to ExtensionTypeCustomDeserializers
ExtensionTypeCustomDeserializers extTypeCustomDesers = new ExtensionTypeCustomDeserializers();
extTypeCustomDesers.addCustomDeser((byte) 59, data -> {
    if (Arrays.equals(data,
              new byte[] {(byte) 0xCA, (byte) 0xFE, (byte) 0xBA, (byte) 0xBE})) {
        return "Java";
    }
    return "Not Java";
});

ObjectMapper objectMapper = new MessagePackMapper(
        new MessagePackFactory().setExtTypeCustomDesers(extTypeCustomDesers));

System.out.println(objectMapper.readValue(bytes, Object.class));
  // => Java

Use extension type as Map key

static class TripleBytesPojo
{
  public byte first;
  public byte second;
  public byte third;

  public TripleBytesPojo(byte first, byte second, byte third)
  {
    this.first = first;
    this.second = second;
    this.third = third;
  }

  @Override
  public boolean equals(Object o)
  {
    :
  }

  @Override
  public int hashCode()
  {
    :
  }

  @Override
  public String toString()
  {
    // This key format is used when serialized as map key
    return String.format("%d-%d-%d", first, second, third);
  }

  static class KeyDeserializer
      extends tools.jackson.databind.KeyDeserializer
  {
    @Override
    public Object deserializeKey(String key, DeserializationContext ctxt)
    {
      String[] values = key.split("-");
      return new TripleBytesPojo(Byte.parseByte(values[0]), Byte.parseByte(values[1]), Byte.parseByte(values[2]));
    }
  }

  static TripleBytesPojo deserialize(byte[] bytes)
  {
    return new TripleBytesPojo(bytes[0], bytes[1], bytes[2]);
  }
}

:

byte extTypeCode = 42;

ExtensionTypeCustomDeserializers extTypeCustomDesers = new ExtensionTypeCustomDeserializers();
extTypeCustomDesers.addCustomDeser(extTypeCode, new ExtensionTypeCustomDeserializers.Deser()
{
  @Override
  public Object deserialize(byte[] value)
        throws IOException
  {
    return TripleBytesPojo.deserialize(value);
  }
});

SimpleModule module = new SimpleModule();
module.addKeyDeserializer(TripleBytesPojo.class, new TripleBytesPojo.KeyDeserializer());
ObjectMapper objectMapper = MessagePackMapper.builder(
        new MessagePackFactory().setExtTypeCustomDesers(extTypeCustomDesers))
        .addModule(module)
        .build();

Map<TripleBytesPojo, Integer> deserializedMap =
        objectMapper.readValue(serializedData,
            new TypeReference<Map<TripleBytesPojo, Integer>>() {});

Use extension type as Map value

static class TripleBytesPojo
{
  public byte first;
  public byte second;
  public byte third;

  public TripleBytesPojo(byte first, byte second, byte third)
  {
    this.first = first;
    this.second = second;
    this.third = third;
  }

  static class Deserializer
      extends StdDeserializer<TripleBytesPojo>
  {
    protected Deserializer()
    {
      super(TripleBytesPojo.class);
    }

    @Override
    public TripleBytesPojo deserialize(JsonParser p, DeserializationContext ctxt)
    {
      return TripleBytesPojo.deserialize(p.getBinaryValue());
    }
  }

  static TripleBytesPojo deserialize(byte[] bytes)
  {
    return new TripleBytesPojo(bytes[0], bytes[1], bytes[2]);
  }
}

:

byte extTypeCode = 42;

ExtensionTypeCustomDeserializers extTypeCustomDesers = new ExtensionTypeCustomDeserializers();
extTypeCustomDesers.addCustomDeser(extTypeCode, new ExtensionTypeCustomDeserializers.Deser()
{
  @Override
  public Object deserialize(byte[] value)
      throws IOException
  {
    return TripleBytesPojo.deserialize(value);
  }
});

SimpleModule module = new SimpleModule();
module.addDeserializer(TripleBytesPojo.class, new TripleBytesPojo.Deserializer());
ObjectMapper objectMapper = MessagePackMapper.builder(
        new MessagePackFactory().setExtTypeCustomDesers(extTypeCustomDesers))
        .addModule(module)
        .build();

Map<String, TripleBytesPojo> deserializedMap =
        objectMapper.readValue(serializedData,
            new TypeReference<Map<String, TripleBytesPojo>>() {});

About

Jackson extension that adds support for MessagePack / msgpack.org[Jackson-dataformat] http://msgpack.org/

Resources

Readme
Apache-2.0 license
Activity

Stars

30 stars

Watchers

1 watching

Forks

6 forks
Report repository

Releases

Packages

Used by

Contributors

Languages