Avro's Reflect Datum Writer

March 6, 2011 Mike Barinek

As some of you may know, I’ve been writing a bit of Java in Boulder recently. Overall, it’s pretty exciting and a nice change from Ruby.

That being said, I’ve somewhat isolated our Java development to server-side components. I still consider Ruby/Rails to be the best solution for web application development, although recently had a need for Java.

Anyway, I’ve been playing around with Avro and I thought I’d share a small example that marshals a domain object to bytes. The bytes are later passed around several queues within application.


package com.barinek.devourer.avro;

import com.barinek.devourer.rest.resources.Activity;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.reflect.ReflectDatumWriter;

import java.io.ByteArrayOutputStream;
import java.io.IOException;

public class ReflectMarshaller {

    private final Schema schema = ReflectData.get().getSchema(Activity.class);

    public byte[] marshal(Activity activity) throws IOException {
        ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
        ReflectDatumWriter< Activity > reflectDatumWriter = new ReflectDatumWriter< Activity >(schema);
        DataFileWriter< Activity > writer = new DataFileWriter< Activity >(reflectDatumWriter).create(schema, outputStream);
        writer.append(activity);
        writer.close();
        return outputStream.toByteArray();
    }
}

The interesting bit is that I don’t have a .proto file or .json representation of the type. The Activity class is a Plain Old Java Object.

The above snippet is a candidate to replace some proto files, assuming the MicroBenchmark test suite passes.

Look for more Java posts in the near future.

Here’s an update with unmarshal and T params.


package com.barinek.devourer.avro;

import org.apache.avro.Schema;
import org.apache.avro.file.DataFileStream;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.reflect.ReflectDatumReader;
import org.apache.avro.reflect.ReflectDatumWriter;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

public class ReflectMarshaller {

    public byte[] marshal(Object activity) throws IOException {
        Schema schema = ReflectData.get().getSchema(activity.getClass());
        ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
        ReflectDatumWriter< Object > reflectDatumWriter = new ReflectDatumWriter< Object >(schema);
        DataFileWriter< Object > writer = new DataFileWriter< Object >(reflectDatumWriter).create(schema, outputStream);
        writer.append(activity);
        writer.close();
        return outputStream.toByteArray();
    }

    public < T > T unmarshal(Class< T > returnType, byte[] bytes) throws IOException {
        Schema schema = ReflectData.get().getSchema(returnType);
        ByteArrayInputStream inputStream = new ByteArrayInputStream(bytes);
        ReflectDatumReader< T > reflectDatumReader = new ReflectDatumReader< T >(schema);
        DataFileStream< T > reader = new DataFileStream< T >(inputStream, reflectDatumReader);
        Object activity = reader.next();
        reader.close();
        inputStream.close();
        return ( T ) activity;
    }
}

About the Author

Biography

More Content by Mike Barinek
Previous
RPX Seeks Rails Developers in San Francisco
RPX Seeks Rails Developers in San Francisco

At Pivotal Labs, one of the services we provide our clients is helping them interview and hire. Pivotal Lab...

Next
New in Pivotal Tracker: Reply to emails, delete comments, browse public projects!
New in Pivotal Tracker: Reply to emails, delete comments, browse public projects!

Team collaboration with Pivotal Tracker just got better - you can now post comments to stories right from y...

Enter curious. Exit smarter.

Learn More