Date: Fri, 28 Mar 2008 11:18:04 -0400

From: rodrick.brown@GMAIL­.COM

Subject: [ADVANCED-JAVA] java serialization alternative

To: ADVANCED-JAVA@DISCU­SS.DEVELOP.COM

Do any of you guys have experience with Java serialization and what

kind of overheads it causes? As far as I know, it produces very large

messages and is not particularly fast, so I am concerned about that

given that it is pervasively used in our system core for sending

messages between daemons over our messaging API.

One possible alternative is http://xstream.code­haus.org/ which can be

tuned to produce fairly small messages (by aliasing class names and

types for example). However, I have no sense of the speed. I will

do some performance tests.

Comments, questions, etc?

--

[ Rodrick R. Brown ]

http://www.rodrickb­rown.com

http://www.linkedin­.com/in/rodrickbrown­

===================­================

This list is hosted by DevelopMentor® http://www.develop.­com

View archives and manage your subscription(s) at http://discuss.deve­lop.com

We are building a large scale system which relies on messaging

heavily. We tried all kinds of serialization formats: Java, JAXB, XML

and JSON with XStream. All of them were useless for high throughput.

So we switched to Externalizable style serialization and were able to

get much better performance. Which was evident upfront but we didn't

have to write that code.

...

We are building a large scale system which relies on messaging

heavily. We tried all kinds of serialization formats: Java, JAXB,

XML and JSON with XStream. All of them were useless for high

throughput. So we switched to Externalizable style serialization and

were able to get much better performance. Which was evident upfront

but we didn't have to write that code. Also now it behooves us to

come with some kind of versioning scheme so that we push rolling

upgrades to the cluster w/o bringing the whole cluster down. I can

dig up the numbers if you would like.

A

Yes it does. But it seemed to worked much better in terms of time taken to

do so. I don't seem to remember the exact numbers but increased our

throughput by atleast 400%.

Avinash> Date: Fri, 28 Mar 2008 18:54:25 -0700> From: rschulz@SONIC.NET>

Subject: Re: [ADVANCED-JAVA] java serialization alternative> To:

ADVANCED-JAVA@DISCU­SS.DEVELOP.COM> > On Friday 28 March 2008 18:41,

Avinash Lakshman wrote:> > We are building a large scale system which relies

on messaging> > heavily. We tried all kinds of serialization formats: Java,

JAXB, XML> > and JSON with XStream. All of them were useless for high

throughput.> > So we switched to Externalizable style serialization and were

able to> > get much better performance. Which was evident upfront but we

didn't> > have to write that code.> > Doesn't Externalizable require you to

implement the actual conversion to> and from the external format?> > > >

...> > > Randall Schulz> > ===================­================> This list is

hosted by DevelopMentor(R) http://www.develop.­com> > View archives and

manage your subscription(s) at http://discuss.deve­lop.com

___________________­____________________­____________________­______

Windows Live Hotmail is giving away Zunes.

http://www.windowsl­ive-hotmail.com/Zune­ADay/?locale=en-US&o­cid=TXT_TAGLM_Mobile­_Zune_V3

===================­================

This list is hosted by DevelopMentor(R) http://www.develop.­com

View archives and manage your subscription(s) at

http://discuss.deve­lop.com

Do any of you guys have experience with Java serialization and what

kind of overheads it causes? As far as I know, it produces very large

messages and is not particularly fast, so I am concerned about that

given that it is pervasively used in our system core for sending

messages between daemons over our messaging API.

One possible alternative is http://xstream.code­haus.org/ which can be

tuned to produce fairly small messages (by aliasing class names and

types for example). However, I have no sense of the speed. I will

do some performance tests.

Comments, questions, etc?

--

[ Rodrick R. Brown ]

http://www.rodrickb­rown.com

http://www.linkedin­.com/in/rodrickbrown­

===================­================

This list is hosted by DevelopMentor(R) http://www.develop.­com

View archives and manage your subscription(s) at

http://discuss.deve­lop.com

Yes it does. But it seemed to worked much better in terms of time taken to do so.

I don't seem to remember the exact numbers but increased our throughput by atleast 400%.

Avinash> Date: Fri, 28 Mar 2008 18:54:25 -0700> From: rschulz@SONIC.NET> Subject: Re: [ADVANCED-JAVA] java serialization alternative> To: ADVANCED-JAVA@DISCU­SS.DEVELOP.COM> > On Friday 28 March 2008 18:41, Avinash Lakshman wrote:> > We are building a large scale system which relies on messaging> > heavily. We tried all kinds of serialization formats: Java, JAXB, XML> > and JSON with XStream. All of them were useless for high throughput.> > So we switched to Externalizable style serialization and were able to> > get much better performance. Which was evident upfront but we didn't> > have to write that code.> > Doesn't Externalizable require you to implement the actual conversion to> and from the external format?> > > > ...> > > Randall Schulz> > ===================­================> This list is hosted by DevelopMentor® http://www.develop.­com> > View archives and manage your subscription(s) at http://discuss.deve­lop.com

___________________­____________________­____________________­______

Windows Live Hotmail is giving away Zunes.

http://www.windowsl­ive-hotmail.com/Zune­ADay/?locale=en-US&o­cid=TXT_TAGLM_Mobile­_Zune_V3

===================­================

This list is hosted by DevelopMentor® http://www.develop.­com

View archives and manage your subscription(s) at http://discuss.deve­lop.com

Avinash Lakshman wrote:

Likely because you were making conscious or unconscious assumptions

about your code that allowed you to do the coding more efficiently.

If you go to making every communicated object "externalizable" then

its morally equivalent really to building your own packet protocol.

About the only time I can imagine this would be useful is if you

wanted to keep RMI while ditching the Serialization code. Myself i

havent run into a case where default Serialization was inappropriate

but RMI still was....

===================­================

This list is hosted by DevelopMentor® http://www.develop.­com

View archives and manage your subscription(s) at

http://discuss.deve­lop.com

(I would argue

that reliably maintaining compatibility is almost impossible with

serialization without a robust test suite which easily counteracts the

simplicity of serialization).

Bob

Interesting statement - care to elaborate? I've found that evolving

the class design is easy with serialization but then, I've been mostly

adding fields and paid attention that the default value of the field

is okay for existing serialized object representations. On a single

object level, it is basically an object-to-(name-val­ue-pair) codec.

What pitfalls specifically do you have in mind?

Making compatible changes isn't hard. Identifying incompatible

changes, for example if a programmer makes a change to a class but

forgets to consider serialization, is very tricky.

You simply can't

verify compatibility statically with serialization.

The only way to

verify that two classes are compatible is to run them and test them.

You can either keep serialized data around in a file or something, or

you can use a trick like this:

http://crazybob.org­/2006/01/unit-testin­g-serialization-evol­ution_13.html

.

Either approach is a pain in the butt.

If you use a tool like Thrift, you can check two definitions for

compatibility statically, so for example, you could add a submit check

to your source control system or test suite to detect incompatible

changes.

On Sat, Mar 29, 2008 at 3:10 PM, Attila Szegedi <szegedia@gmail.com­> wrote:

Making compatible changes isn't hard. Identifying incompatible

changes, for example if a programmer makes a change to a class but

forgets to consider serialization, is very tricky.

You simply can't

verify compatibility statically with serialization. The only way to

verify that two classes are compatible is to run them and test them.

You can either keep serialized data around in a file or something, or

you can use a trick like this:

http://crazybob.org­/2006/01/unit-testin­g-serialization-evol­ution_13.html.

Either approach is a pain in the butt.

If you use a tool like Thrift, you can check two definitions for

compatibility statically, so for example, you could add a submit check

to your source control system or test suite to detect incompatible

changes.

Bob

===================­================

This list is hosted by DevelopMentor® http://www.develop.­com

View archives and manage your subscription(s) at http://discuss.deve­lop.com

On 2008.03.29., at 19:01, Bob Lee wrote:

Interesting statement - care to elaborate? I've found that evolving

the class design is easy with serialization but then, I've been mostly

adding fields and paid attention that the default value of the field

is okay for existing serialized object representations. On a single

object level, it is basically an object-to-(name-val­ue-pair) codec.

What pitfalls specifically do you have in mind?

Attila.

===================­================

This list is hosted by DevelopMentor® http://www.develop.­com

View archives and manage your subscription(s) at

http://discuss.deve­lop.com

Bob Lee wrote:

This is an issue we will be having to deal with down the line with

darkstar based apps. I've been toying mentally with the idea of a tool

that would take as input a serialized graph, an old code base, and a new

code base, and allow you to work interactively to solve an

incompatibilities, hopefully resultign in what is effectively a

translation script.

Right - the real pain (which your blog entry doesn't show) is when

both classes are actually two versions of the same class, so you must

ensure you have both .class files available somewhere, have them

loaded through different class loaders, and then persuade the

ObjectInputStream to deserialize to the right class (as it's

notoriously ignorant of thread context class loader...). PITA indeed.

Nice - thanks for the pointer. I wonder how easy is it to extract this

functionality from Thrift, as otherwise it seems to be geared toward

RPC, and I'm more on the async messaging side of architecture...

I assume you could just use the IDL functionality on its own, but I

haven't used Thrift before. I've used a few similar proprietary

frameworks though.

On Sun, Mar 30, 2008 at 11:27 AM, Bob Lee <crazybob@crazybob.­org> wrote:

I mean, I assume you could just use the flattenning/seriali­zation

functionality on its own.

Bob

Serialization sounds good in theory, but you really want to decouple

your code from its persistent state.

Tools like Thrift force you to do

it. Serialization makes it easy not to do it--i.e. you really should

use the DTO pattern, but many people don't. If you couple your code

and its persistent state, you might be able to add a field here and

there, but what happens when you want to perform big refactorings and

rewrites? Your runtime classes may not match the serialized forms

anymore; serialization doesn't handle this well. To make matters

worse, the serialization format is very opaque and difficult to tool

with anything besides the original class. I think this is one of many

reasons why relational databases are still so much more popular for

long term persistence.

Bob

On Sun, Mar 30, 2008 at 11:28 AM, Bob Lee <crazybob@crazybob.­org> wrote:

===================­================

This list is hosted by DevelopMentor® http://www.develop.­com

View archives and manage your subscription(s) at http://discuss.deve­lop.com

Well, not in our case.

On Sun, Mar 30, 2008 at 4:30 PM, Jeff Kesselman

<Jeffrey.Kesselman@­sun.com> wrote:

You say that until one of your users wants to rewrite something

without losing all their data.

There are plenty of things that can be re-written without losing data.

See "serialization compatible changes."

Beyond that, as i mentioned

before, I'm already looking at ways to build a merge-type tool.

The fact of the matter is I need transparent POJO data management for my

client base and I know of no other way to achieve that.

At the end of the day there are always advantages and limits to every

tchhnique. A good craftsman does not so fall in love with his hammer he

forgets how to turn a screw, or vice versa.

And if he hist himself in the thumb with the hammer a few times, he

asks "what am i doing wrong with this tool." He doesn't throw it away.

I'm not expert by JRuby might be an option.

The fact of the matter is I need transparent POJO data management for my

client base and I know of no other way to achieve that.

Serialization sounds good in theory, but you really want to decouple

your code from its persistent state. [...] I think this is one of many

reasons why relational databases are still so much more popular for

long term persistence.