Java Profiling, continued

Posted in Wayne Adams on June 29th, 2009 by admin
I've been building a Java profiler lately, partly to address what I think are shortcomings in the current set of free/shareware profilers, and also to enjoy the flexibility of obtaining only the profiling information I need. If you've spent much time with profilers, you already know they can have quite an impact on the application they're profiling! My goal with my own profiler is to obtain a narrow set of information, having more control over the size and frequency of the data collection.

One feature I really like about hprof (see http://java.sun.com/developer/technicalArticles/Programming/HPROF.html) is the heap=sites option. If you're not familiar with this option, it is described in the above link, but in short, hprof can provide a profile of the allocation of objects per site, where a site is a unique Java stack trace of a fixed depth. This can be very handy information when you are investigating the heap usage of an allocation, but it is extremely costly to get with hprof (in one recent case I was investigating, starting an application with the hprof agent slowed the application down by several orders of magnitue).

Today I'll discuss how I get this same information without the overhead of hprof. I'll build on my last blog entry about the java.lang.instrument package and combine bytecode injection with the ThreadMXBean to get this same information.

The basic outline is:
  1. Create a java.lang.instrument profiling agent.
  2. Intercept all constructors as they are loaded, instrumenting them on constructor entry and constructor exit. Information passed back to the profiler application will include the thread ID and the size of the object created.
  3. In the profiling agent, keep track of the total memory allocated for each object type, as well as a list of stack trace signatures at object instantiation, with frequency of each. The stack trace is obtained via the ThreadMXBean of the JVM.
  4. Rank objects by total amount of memory allocated, the set of unique stack trace signatures for each constructor, and the number of times the object was instantiated via each stack trace.
This information will approximate the heap=sites output of hprof, and can be obtained more economically for a number of reasons:
  • We can restrict the classes (constructors) instrumented. Note that if we do not instrument every class, we may not see every instance of a particular object's instantiation, but in practice what you usually care about are objects instantiated by your application. Learning in hprof that 20 MB of java.lang.Objects were created as a side effects of a java.* method call usually doesn't help diagnose performance problems.
  • The custom profiler can get this information in a very small fraction of the same time that hprof would require to get it, as it is a much more targeted application.
  • With the custom profiler, we have a lot more control over the interval during which the data is collected. For example, we can clear our profiler's data, initiate an operation in the profiled application, and see just the allocation history for the time period of interest. I actually do not know how to obtain this information with hprof; the heap=sites option dumps its summary at application termination (as a snapshot of the heap at that point in time), which may not be what you want.
As I mentioned in my last entry, I use Javassist (see http://sourceforge.net/project/showfiles.php?group_id=22866&package_id=80766). Javassist will create a Java class representation from the bytecode array, and from there it is very easy to obtain the constructors, using CtClass.getConstructors(). The CtConstructor.insertBefore() and insertAfter() methods round out all you will need from Javassist to instrument your code.

Object size is a pesky issue in Java, especially if you have experienced the convenience of the sizeof() operator. I use the java.lang.instrument.Instrumentation instance, supplied by the JVM in the call to the profiler's premain() method, to get the size of the objects (using getObjectSize(Object)). It isn't exactly clear what size this method returns; the javadocs are a little cagey and it's safe to safe this value isn't a deep copy. While it is possible to drill down through object references in a class and attempt to accumulate a deep-copy value, this operation itself can be very time-consuming, incomplete (due to inaccessible attributes) and slightly misleading (static references, anyone?). Not to mention you can easily get a stack overflow error in the process. It may not be perfect, but I have chosen to simply use the value returned by Instrumentation.getObjectSize(Object), and to keep in mind that what I'm seeing is useful in a rough, rather than a precise, sense.

Since the profiler runs in the same JVM as the profiled application, it is fairly simple to get a reference to the JVM ThreadMXBean, using java.lang.management.ManagementFactory.getThreadMXBean(). Given a thread ID (which you can arrange for your instrumented constructor to pass back to your profiler), you can get the stack trace for the thread with ThreadMXBean.getThreadInfo(long,int). Note that there are several signatures for this method call; remember to pick the one where you specify the maximum depth of the stack trace, as the default value is zero!

The value of collecting information like this in a profiler should be self-evident; however, there are greater opportunities available for this data. For example, a variety of profiling information can be used to generate events which can be used to monitor the health of the profiled system. These don't all need to be just events generated by the profiler; they could include SCM and system-maintenance events, with the result being that maybe you could see that your ASN.1 parser suddenly takes three times as long to parse a switch file since the last change was checked in and rolled out to production. In a later posting, I'll look at how such a system could be implemented, looking at the open-source tools available to analyze this kind of data and make associations between events that can provide much faster resolution of operational and performance issues.


Leave A Comment »

Kayaks in Golden

Posted in Eric Daugherty on June 26th, 2009 by admin
I went for a walk with the family tonight in Golden, CO and came across some kayakers doing jumps in the river. My shutter speeds are a bit slow, but still some cool moves...

Leave A Comment »

Pruning

Posted in Jerry Andrews on June 24th, 2009 by admin
I spent the past few hours pruning the trees along my property line--mostly across the line on the neighbor's side, but they've grown in and through my fences, and developed into quite a thicket along the property line. The bushes acted as a lower story for the trees which are planted along the line, and looked quite nice; I didn't realize what a mess they'd become. Once I started, I realized they were choking out the bases of the big trees, and had become infested with a mess of thorny vines which had bound the whole thing together into a green mass of dead limbs and vine. Now that they're trimmed back, the trees look nicer, and the edges of my lawn are free to grow; in 2 weeks, I now realize, it will look a lot nicer than it has in the past couple of years.

In my project work, I regularly come across thickets of choked code which is hard to read and even harder to update. I fixed a bug yesterday which should've take an hour to fix, but took almost 12 hours, because I had to write a 2 new unit test suites, making sure to cover all the old code. That took about 10 hours; then the last 2 were spent brutally pruning and finally, fixing the bug. What's left is small and tight, and will be a lot easier to maintain, since it has a nice suite of unit tests.

It's easy to let the thicket grow. It's also easy to write tests while the classes are still small, and keep them up to date as the code evolves. Then, when a class does turn into a thicket, it can be pruned quickly and with authority, because the tests are already there.

Leave A Comment »

Edge Cases

Posted in Eric Daugherty on June 24th, 2009 by admin
In tools like Outlook, there are an impossible number of ways users can interact with the tool. This creates lots of 'edge cases' that don't get tested or developed for.

Here is one:
  1. Use the default setting that does not have the BCC line shown.
  2. Send an email with BCCs by clicking on the To: button and adding entries to the BCC line in the dialog box.
  3. Send the message
  4. Realize you want to reuse the body in a new message.
  5. Open the sent message and click resend.
  6. Type in a new To: address
  7. Send it to everyone you previously BCC'ed because Outlook doesn't show you the BCC line, even though it has entries.
Oops.

This is an edge case where adding a user to the BCC line causes it to be displayed, but opening/resending an email with users in the BCC line DOES NOT cause it to be displayed. The trigger logic is flawed, but you don't realize it is flawed unless you are aware of other ways that a specific state can occur. In this case, the logic to display the BCC line should be triggered by the 'state' of having entries in the BCC line, not the event of adding an address to the BCC line.

Leave A Comment »

The Perfect Flex Application Framework (Part 2 of 2)

Posted in Jon Rose on June 21st, 2009 by jonr

In part one, I introduced some of the things that I believe convolute the Flex third party framework space  – primarily the influence of MVC.  The history around this pattern, particularly in the Java world, has allowed frameworks to gain mindshare that do little but help enforce the pattern.  This post is about bringing the focus back to features that make life easier on developers by allowing them to build the things that users care about faster and/or easier. That’s really what third party frameworks should aim to do.

So, moving forward by pretending that no one will disagree with that assertion, here is the overview of what would be in my perfect Flex application development framework:

  • Separation of Concerns (SoC):

    In Flex development, there is a need for something outside of the core framework to help developers adhere to a separation in the application code.  In reality it is not the most difficult or interesting part of the problem, but would likely be a part of any complete third-party framework.   I personally prefer the Swiz dependency injection approach for this.  Also, there are likely a number of interesting things one could add on top of a dependency injection foundation, as we have seen with the SpringFramework in Java.

    Looking at the Swiz implementation, one can also conclude that the underlying platform needs to grow so that this can be supported in a more native way and increase the possibility that greater leverage can be gained from introducing dependency injection to Flex applications.

  • Client Side Data Management (think client side Hibernate):

    RIA introduced (or re-introduced) the notion of client side data / state.  In traditional page based applications, the developer never had to consider this possibility, as all state is kept and tracked on the server.  With advanced RIA features, state is often changed on the client, but not immediately sent back to the server.  This introduces a number of challenges in managing data state between the client and the server.  Thus, there is a need for infrastructure to support data synchronization between the client and server.

    Both LiveCycle Data Service (LCDS) and Clear Toolkit include features that help the developer with this to a certain extent, but they rely on the specific server implementations.  In my perfect imaginary world, this would all be done client-side agnostic of what server is providing the data so that one could use it regardless of what technology is providing the data.

    This may even be coming in the next version of Flash Builder:

    “Of course, data-centric applications are about more than just binding operations to components. Flash Builder lets you enable client side data management for your services, allowing on-demand fetching of data for easy and efficient scrolling through large collections of data; change tracking, allowing users to undo actions, and automating the common CRUD (create, read, update, & delete) functions that usually need to be hand-coded. The results are applications that deliver users a richer experience than is possible with ordinary HTML front ends.”

  • Security:

    In any enterprise application, there are security considerations at multiple levels. There are a number of methods and strategies for dealing with authentication, but not many Flex tools for addressing authorization. In the simplest form, one can break out authorization into the following categories: service/method level, data level, and filtering what each user can view on the screen. Securing services and data pretty much have to be handled on the server, but something is necessary to control what the user sees on the client.

    There are not many libraries I know of for implementing this common requirement. My current preference is my Chimp component for doing this declaratively through Flex metadata. Many just deal with this manually in each component, others monkey patch to add filtering logic on UIComponent or within View States. Ideally, features for dealing with this challenge would make it into any complete third-party framework.

  • Global Event Bus:

    In Mate, basically every event is handled through the Mate event bus. Having a global event bus can invaluable in implementing a number of RIA features. Yet, handling every single event this way seems a bit odd to me. I think Flex events can be broken into four categories:

    Component Events: These are events that are only omitted and handled inside the component. Think of a component that has children and they need to communicate between one another.
    Local Events: These events are omitted by components that can be handled by within the context that the component is used.
    Global & Data Change Events: Events that any arbitrary part of the system may want to be notified of when they happen, such as the user profile has changed.

    So, in my mind, the bus is a necessary part of the perfect Flex framework, but shouldn’t be used for handling all events in the system.

Those are my top requirements for the perfect third-party Flex framework. I am sure there are a number of others that it would make sense to add to the list.

In working through this post, I concluded that there is too much focus put into third-party frameworks with Flex. The items I outlined are missing gaps that need to be met with third-party frameworks, or hopefully by the underlying platform someday, but they are not the most challenging things one will encounter when building enterprise Flex applications.

The biggest challenge for Flex applications of any scale is the quality Flash Player virtual machine. The Flash Player has to get faster, better at dealing with memory consumption, and garbage collection. Flex is a good platform for building rich Internet applications, but like any emerging technology it will have to mature to continue to gain adoptions and keep the current developers.

Leave A Comment »

FitNesse for a Particular Purpose

Posted in Jerry Andrews on June 21st, 2009 by admin
I've been aware of Ward Cunningham's "Fit" framework (http://fit.c2.com/) for some time. I've never actually used it, because it seemed "hard" and my business analysts had their own way of doing things.

My current project introduced me to "FitNesse" (http://fitnesse.org/), a wiki which integrates the Fit framework with wiki pages, and forced me to learn to use it for real enterprise-scale projects, both interactively and as an automated test suite to be run as part of our continuous build process.

Boy, have I been missing out; this thing rocks!

At the core, FitNesse is a wiki. It's important to remember that, because as it's presented in the documentation, it's primarly a test tool. But here's the thing: if you just type your requirements into the wiki, it more-or-less demands that you write tests, too. That is, that you express your requirements as something that the developers must test. That has two huge benefits: 1) it makes sure developers speak the same language as the requirements, and 2) it forces requirements to be testable. Because it's a wiki, as developers and analysts come to a shared understanding of the requirements, either can easily tweak the pages to say what needs saying, and the tests to express those requirements succinctly.

As requirements and associated tests are developed in FitNesse pages, you can check them in to source control, run them regularly, and there's no need to back-trace test to code to requirements--if the FitNesse tests run, you've met all the requirements they embody. It's about the shortest distance between requirements and a quality deliverable I have ever seen, and one of the lowest overhead. Rather than checking our FitNesse pages in "at the end of an edit", we have a scheduled job which checks in any changes at the end of every day.

The examples on the Fit and FitNesse web sites are simple; I encourage you to go there now and have a look. These are great for learning, but they don't express the real power of the tool.

Here's an example from my current project in all its enterprise-messy glory. This is what real enterprise application testing is about. The image below is about 2/3 of the testing page for one scenario (click the image for a full-size rendering. It's large (about 100k, and about 3x my screen)). Most of the page sets up the test conditions. Along the way, there are tests to verify the (very complex) input data are correct; "Read Tax Rate Area", for example, generates numbers which aren't in any specific location--they're summed over dozens of inputs already in the database. Any heading with a question mark is an invitation to invoke the application code to see what the results are.

Binding this page to the application code requires a "fixture", which each developer writes. Fixtures are typically quite small. The set of fixtures which executes the tests on the page shown total 596 lines of code; it took me about a day to write, including figuring out exactly how to perform the required setup. I had to write a fixture base class which took another day, but now that I have that written, writing new complex tests will be as simple as writing this one was. This scenario, and 11 others just like it, run using the same fixture and exercises the same code as runs in production. Those 11 scenarios represent most of the edge cases in one kind of billing. There are many more; a total of around 400 scenarios and thousands of individual tests.

To run the test, you simply punch the test button in the upper right-hand corner of the page. I'll walk through some of the results. First, the top of the page notes that 2 tests succeeded, 2 failed, 2 were ignored completely (usually as a result of failure), and 2 exceptions were thrown. Each row typically represents a test, and that's not anything like the full population of the page, but exceptions tend to end things abruptly. Secondly, note the "Read Tax Rate Area" block, where two tests ran, and both failed. Thus, the assumptions which are built into results further down the page are proven wrong--the testers need to know this so they can revise their expected answers. One important thing to note is that the framework handled the display; all the fixture code does is respond to "getLipTaxRate()" and "getLiTaxRate()"--names derived from the associated headings--with a text string. The framework colored the cells and placed the "expected" vs. "actual" answers on the page.


Scrolling down, there's an exception. Yes--I made a mistake in the fixture which, under the right conditions, throws an NPE. I'll fix that in the next release. On down is a successful test.

That's what it looks like. The fixture code is easy to write, the requirements are expressed in a way the business people understand, and there's a man-day or less (usually far less) between describing the behavior in a FitNesse test and writing the fixture which connects the test to the functional code.

In our environment, FitNesse tests are also run automatically as part of our automated build cycle. The tests span hundreds of wiki pages. The results are consolidated into a nice punchy list with counts of successes and failures, much like the top of the first "post-run" page above. For failures, more detail is provided. The XML parsing is done by fixures already available in CruiseControl. I found excellent help for configuring our ant build (and thus our CruiseControl build) at http://www.jroller.com/njain/entry/integrating_fitnesse_with_cruisecontrol. Thanks, Naresh!

What more could you want in a requirements/code/test/document system? Go forth and experiment. You'll be glad you did.

Leave A Comment »

The evil of Flash (according to HTML 5)

Posted in Jon Rose on June 21st, 2009 by jonr

I just finished up a post on HTML 5 and Flash for InfoQ.com. The news post just covers the conversation happening in the community(s) (not my opinion). I am glad browser standards are finally evolving, and think all developers will benefit from the advancements. Being a Flex developer, I have obviously embraced the idea that third-party plug-ins are not a bad thing when trying to build real applications. It is too early on a Sunday morning to get to far into this debate, but I wanted to throw out a couple of points:

  1. Since the language around browser standards is very feel good, with terms like “open web,” I think it is often lost how much big companies, such as Google and Mozilla, stand to gain if developers validate their assertions around browser standards. In all reality, the open web vendors are no different than the vendors providing “third party runtimes,” they want / need people to use their platforms.

  2. The “open web” never existed. Let’s take a look at a quote from Ian Hickson, HTML 5 specification co-editor and Google employee:

    “It would be a terrible step backward if humanity’s major development platform [the Web] was controlled by a single vendor the way that previous platforms such as Windows have been.”

    Not only does the language seem a bit dramatic to me, but I think there is little risk of this happening (outside of the standards). Historically browser standards have been the main catalyst in bringing us down to the lowest common denominator in web development. We have all worked on projects where the management decided to just build the application to work in IE because the browser standards are unreliably implemented across browsers.

The reality is that the RIA space is currently experiencing wonderful competition across vendors (Flash, Silverlight, JavaFX) with applications being built and deployed on a number of different platforms. Frankly, none of the current platform advancements that are making developers lives easier have come from standards based implementations.

Leave A Comment »

Model Driven Development with Adobe Flex

Posted in Jon Rose on June 21st, 2009 by jonr

Here at Gorilla Logic, we are pretty excited about Adobe’s announcement that LiveCycle Data Services 3 beta is now available. The next version of LCDS includes advanced modeling features that look poised to bring true model driven development to Flex developers. Since Gorilla Logic started with the mission of build a model driven engine (the GXE), we are understandably excited that Adobe is bringing model driven development to the Flex development world in LCDS.

If you are interested in learning more about the release, check out my InfoQ.com news post: http://www.infoq.com/news/2009/06/model-driven-dev-with-flex.

Leave A Comment »

Netflix Saturday Processing

Posted in Eric Daugherty on June 20th, 2009 by admin
I was suprised to get an email from Netflix today notifying me that a new disc shipped out. Netflix does not normally process movies on Saturday, so mailing a movie back on Friday means you have to wait until Tuesday to get the replacment.

I did a quick search, and indeed, it looks like Netflix is introducting Saturday processing across the US.

A great service just got better.

Leave A Comment »

Measuring the Unmeasurable

Posted in Jerry Andrews on June 17th, 2009 by admin
In a previous post, I wondered how you measure yourself as a developer. The available metrics are pretty poor, overall. Consider all the ways a developer can be "good"--fast, good at translating requirements into functionality, good at the key algorithm, good at clear, concise code, good at predicting when she'll be done with a given bit of code... on and on. How do you measure that? The short answer seems to be: you don't. I haven't found a single metric which addresses developer quality which is in any way objective. Then I cast back to my earlier life as an engineer--and surprise! Exactly the same situation exists there.

At the best large companies I've worked for -- Babcock & Wilcox, GPU Nuclear, Oracle -- jobs I look back on with fondness -- individual performance was evaluated by senior people, and job descriptions were carefully thought-out lists of subjective performance metrics. When I was promoted from Engineer 2 to Engineer 3 at GPU Nuclear, my boss and I went over the metrics for that job to see which I hadn't quite mastered yet, and that became my development plan. When I mastered Engineer 3, I would start on the list for Senior Engineer 1, and when my superiors felt I had mastered enough of that list, I'd be promoted again.

No such list has governed my development as a software developer, designer, and architect, except at Oracle. I've been promoted several times, but there were never any objective criteria against which I could say "see? I have done this", or "boy, here's an area I should work on". More than once, I've been involved in projects where the "senior" architect, designer, or developers were among the least-accomplished on the team. This can only happen in a situation where measures of capability--even if they are evaluated subjectively--are not present.

The chief architect at Sabre Airline Solutions and I spent some time a year or two back discussing job descriptions for various programmer levels and career paths. While we didn't discuss specifics, I now wish we had. In my career as a developer, I have seen only one set of specifics for different developer levels: the job descriptions in Oracle Consulting Services. I wish I had those descriptions now; my recollection is that they were concise, complete, and compelling. I'll try to collect some job descriptions for developers over the next couple of months. Perhaps there's something available that could be used to address the whole "what's a good developer" question.

Leave A Comment »

« Previous Entries
Next Page »