Flex, Adobe, and alternatives to LCDS and Flash Builder

Posted in Eric Daugherty on January 21st, 2010 by admin

admin originally posted this on EricDaugherty.com.

Yesterday I came across two interesting articles, both relating to alternatives to existing Adobe commercial tools in the Flex space.

Flex/Flash Builder vs. IntelliJ IDEA

The first article was about a favorite of mine, IntelliJ IDEA. I've been an IDEA user (some may say fanatic) for a long time, and while I've used Eclipse based editors when necessary, I always feel more at home with IDEA. The article is an overview of the Flex/ActionScript support available in IDEA 9. My conclusion is: while the depth of support is impressive (refactoring, code analysis), it does suffer from what I think of as 'not my primary function' issues. Jesse's (the author) complaints are mostly around speed, and non-intuitive configuration issues. I'm also not sure how it stacks up against some of the code generation and BlazeDS/LCDS integration features of Flash Builder 4 (which is still in beta).

The 'not my primary function' issue is one of my strongest frustrations with Eclipse. I've always found IDEA more intuitive to use because the basic functions feel 'more built in' and less like a set of cobbled together plug-ins. In fact, one of my concerns with IDEA becoming a platform like eclipse (part of their new Open Source strategy) is the erosion of this advantage. I think these types of issues will crop up more and more as IDEA supports a wider set of platforms.

LCDS vs. GraniteDS


The second article was about Granite DS, an alternative to Adobe's commercial LCDS. The article is actually a 'response' to a pair of articles about the new features in LCDS 3 by fellow Gorilla Justin Shacklette. The article covered quite a bit of ground, so I'll touch on a few areas of interest.

Code Generation

Justin discusses the ability for LCDS to generate server functionality based on your Flex code.
"In LCDS 3, you write the UI and the model, and LCDS does everything else. Yep, LCDS handles everything below the UI using just the model. Not only that, but LCDS is constructs a performant, enterprise-class backend using just the model. No boilerplate code, no plumbing code, no serialization/deserialization, no Hibernate config, just a point-and-click Modeler and your UI code. Game changing."
If this works, I mean REALLY WORKS, it can be game changing. But we've all seen 'push this button for magic' before. More importantly, even if it works, is it 'how work gets done'. I think the SOFEA concept is more reasonable. In this concept Flex could be just one of many possible rich clients sharing a single set of services with other clients that could include iPhone, Android, Silverlight, JavaFX, GWT, JavaScript/Ajax, etc. clients. Obviously no application would use all of these, but having a Flex client an HTML/JavaScript/AJAX client and an iPhone client can be reasonable, especially in today's world of mobile optimized apps. And if this is the case, do you want your service layer generated by LCDS based on your Flex code?

GraniteDS takes the approach that your server code is the foundation of your application, and GraniteDS helps expose and integrate that into a Flex application. All of your modeling and service layer is written in traditional Java code, and then GraniteDS can expose the methods (via annotation) and generate ActionScript classes to map your model objects.

In reality, I think these approaches are targeted for different uses, and different audiences. Java shops that want to build a rich client will be attracted to the GraniteDS approach, while Flex applications that need small amounts of server data storage and persistence can benefit from the LCDS approach. As a long time Java developer, you can guess where I'll feel more comfortable.

Asynchronous Processing

One of the 'joys' of working with Flex is the interaction between its single application thread and asynchronous I/O. LCDS and GraniteDS approach these in separate ways. I won't rehash the comparison as I think the GraniteDS article does a good job. My view is that the LCDS way feels more like Flex, and the GraniteDS way feels more like Java. It isn't surprising giving the origination of the two projects. Both get the job done and are using the same underlying protocol, so it just boils down to a different API.

BlazeDS

Where does BlazeDS fit into all of this? As Adobe's Open Source alternative to LCDS, you might assume there was no point in the GraniteDS project. For simple RPC and messaging, BlazeDS will handle all your basic needs. With the latest release (3.2.0) well over a year old, it isn't a rapidly evolving project. However, it still provides a solid bridge between Java and Flex, so it doesn't necessarily need regular releases either.

In the end, GraniteDS is an alternative to LCDS, not BlazeDS. All three bring different feature sets to the table, so you will need to evaluate them in light of your specific needs.

What this means for Adobe

So what does this mean for Adobe? I think it is a classic good news/bad news situation. As far as I can tell, Adobe is a packages software vendor. While they have been making efforts to gain traction in the enterprise software space (ie LCDS) they are still very focused on selling packaged software: Flash Builder, Photoshop, Lightroom, Premier, etc. And in the Flex space, that means Flash Builder and LCDS licenses. If these licenses don't offset their efforts on the open source Flex SDK, why should they continue to champion the platform. (Well, there are other reasons, but the point remains).

Having strong competitors in the market for both products shows both a growing maturity and adoption of the Flex platform. It also means more competition for revenue. I think these CAN both be positive if they result in Adobe raising the bar on its efforts.

The Real Reason Outsourcing Continues To Fail

Posted in Dave Rodenbaugh on January 21st, 2010 by admin

admin originally posted this on Lessons of Failure.

Outsourcing:  The word American developers love to hate.  There are lots of stories out there about failed efforts that involve offshore development (“offshoring”).  I even have a few myself.  But this post is not about bashing outsourcing countries, the cheapskates that hire them, or the project managers who can’t control the resulting chaos.  This is about understanding why we have such a difficult time making offshore development work before any of those folks get involved.

Airline Disasters and PDI

What got me thinking about this subject was the book Outliers:  The Story of Success, by Malcolm Gladwell.  In it, Gladwell discusses the reason for a series of catastrophic airline failures.  Without repeating most of his excellent dialogue in the book, here’s the Cliff Notes version:

  • There was a study by Geert Hofstede, where he tracked the Power Distance Index (PDI) among selected world countries.
  • The Power Distance Index is an incredibly important measure of how a person in a country would generally react to an authoritarian figure.
  • Countries with high PDI would have more people willing to accept an authoritarian power figure in a paternalistic sort of way.  Like how you might defer to your father’s decisions, or to a king’s requests, for example.  People in higher PDI countries are less likely to question authority and more willing to accept instructions from those in higher positions of society.
  • Lower PDI countries are characterized by subordinates being more willing to question the orders of a superior.  Low PDI countries have people that tend to view themselves more like those in power than not.  In other words, you might judge yourself to be much like your boss in a low PDI country.

Crashes Caused By Power Differences

The NTSB regularly investigates airplane crashes to determine causes, but investigators were unnerved in the late 1990s to find a pattern of fatal crashes were found to be very airline (and also country) specific.  These two airlines with frightening records were Korean Airlines and Colombian Airlines.  The fatal crashes all had common attributes:

  • The pilots didn’t make any single fatal error.  They all started with several smaller errors that gradually built up to a catastrophic failure.
  • For any airline, the First Officers are always trained to double-check the Captain’s orders.  This is a safety protocol that prevents single-person failures from happening.  It is the First Officer’s duty to question and even override the Captain if the order he gives is unsound or improper.
  • In these particular cases, the black-box recordings indicated that the First Officers were hinting at problems, but were not strongly identifying them in a way that would make it very obvious.  Nor did they explicitly tell the Captain that he was acting against normal procedures.

Power Means Fear of Communication

In the aftermath, an astute researcher matched up the Hofstede PDI data with these two countries and found that both Korea and Colombia are high on the PDI scale.  What were the implications of this?

  • Clearly, a power difference would have existed in the cockpit between the First Officer and Captain.
  • The manner of communication between superiors and subordinates is very different if you are from a high PDI country than a lower one.
  • In a high PDI country, a subordinate must use the proper language, body posture, facial expressions and tone when communicating with a person of higher status or power.  In Korean, for example, there are no less than six distinct ways to address someone from the most formal to the least.  In the United States (a lower PDI country), we have a much looser style (“Sir”, which could apply to a General as much as a Lieutenant in the military, or even your father in some situations).
  • This disparity of power created hesitance on the part of the First Officer to embarrass his Captain when the Captain made mistakes (and in each of these cases, at least 3 mistakes were made).  So instead of using a direct method of communication (“This weather is really bad, we should turn back immediately or regroup for another airport”), he used a more subtle, formal and proper one (“Sir, look how it is raining outside.”).
  • The implication of a high PDI country is that there is a rich subtext going on between the two communicators.  But that subtext assumes that both sides are alert, paying attention and can clearly understand the implicit signals.
  • In the case of the airlines, the Captain was almost always sleep deprived and exhausted by the time the situation arose, making that communication impossible.

The researcher concluded that the pilots were inside a cultural framework that dictated how they should behave at a time when those behaviors turned out to be detrimental and outright dangerous to the safety and welfare of the passengers.  In other words, the fact they came from high PDI countries made it impossible for the proper communication to take place when it was most necessary to be plain and step outside the traditional power roles of these cultures.

Past Outsourcing Blames

What does all this have to do with outsourcing?  First, we need to understand why outsourcing has traditionally “failed”, according to both buyers (those who purchase outsourced services) and providers (those who perform the service in their local country, or send people to other countries to perform services).  Here’s a graph of combined data about a survey regarded failed projects from 2004.  The data are still relevant to today’s discussion.  (Source:  The Outsourcing Center)

Reasons of Failure for Outsourced Projects

Reasons of Failure for Outsourced Projects

Notice the even distribution of reasons once the provider & buyer survey data is combined.  This is interesting because neither party can clearly point to a single, differentiating causation factor in the failure of outsourced projects.  But I believe that’s because they asked the wrong questions in the survey.

The Real Issue with Outsourcing is Power Difference

If you have a buyer from a lower PDI country and a provider from a higher PDI country, there are already implicit consequences to your interaction that neither party will know about without prior outsourcing experience or natural cultural awareness(1).  And even with that experience, it’s not a given that they will understand the reasons behind the challenges of outsourcing.  Let me create an example from my own personal experience:

Suppose you had an American company (Buyer) and an Indian company (Provider).  The American company contracts with the Indian one to provide offshore outsourced software development at a fixed price per developer.  Certain key performance indicators are agreed upon by both parties and the game is afoot.  Let’s also assume the Indians agree to a six month project to write a content management system for the Americans.

A typical scenario of engagement might follow like this:(2)
  • The first month, everyone hammers out the requirements documents and in a great ball of fury, declares them sound and ready for implementation.  The American company at this point would typically reduce the daily oversight on the project to something more reasonable, like weekly updates.
  • The second, third and maybe even fourth months pass with little fanfare.  The Indian developers are quietly building the specified software and the Americans are receiving updates about it that are all positive and sound great.
  • At some point, the American company asks for a demo of the progress to date.  The Indians put together something after a bit of negotiation (since the Americans neglected to mention the demo as a deliverable before the end).  The Americans see the actual software and fly off the handle.  Performance is awful, the screens don’t look anything like what they want, and the software appears to be behind schedule.
  • Further code reviews by American developers indicate that the code quality is fairly poor, lacking in comments, unit tests, and filled with copy-paste blocks of duplicate code.  The Americans immediately demand the project be put under different management.
  • The project falls off of the rails somewhere after this.  It will either be canceled, brought back in house, or will be delivered extremely late after extensive modification to the original requirements.

There’s lots to pick on here on both sides of the table.  I would like to point out that the fact that I picked on Americans and Indians is actually irrelevant here. You could easily substitute “British” for Americans (3), and “Filipinos” for Indians with the same results.  But why are they so interchangeable in this fashion?  It’s because of PDI and the inherent cultural communication issues that come with it.

Dilbert

Dilbert says it best

Here is a list of the top 10 Outsource Providing countries in 2009, and their PDI scores.

  1. India (77)
  2. Thailand (64)
  3. Mexico (81)
  4. China (80)
  5. Indonesia (78)
  6. Malaysia (104)
  7. Philippines (94)
  8. Jordan (no data)
  9. Egypt (80)
  10. Bulgaria (no data)

For reference, the United States is 40 on the scale.  Western countries can run the gamut as high as Belgium (60) to as low as Austria (11).  The scale is from 1-120, where 120 is extremely high PDI.  You can see all the countries’ measurements in the original study on this colorful world map of PDI indices.  The gray countries are ones that weren’t measured.  India would be considered moderately high PDI at 77 (in the 61-80 range).

So what happens when you bring a low PDI buyer together with a high PDI provider?

In a word:  Disaster.

Cultural Context Matters In Communication

Each side expects a certain subtext to go on during a conversation because of their own cultural context.  Like this:

Low PDI Manager: So, is the new website ready for launch by Friday?

Low PDI Developer: No, and it’s going to be another 2 weeks because we need the new servers to arrive, for QA to finish with testing after they do, and then release the code.

Pretty straight question, pretty straight answer if you’re from a low PDI country like the United States.  There is little assumption about the subtext because a low PDI communicator is used to “speaking his/her mind” about it.  The information is supposed to be in the conversation as spoken words.  If it’s not there, it’s ignored.

But what if we change that a bit?  Assume the High PDI and Low PDI Developers BOTH have access to the same information and are equally competent:

Low PDI Manager: So, is the new website ready for launch by Friday?

High PDI Developer: Yes, it may be ready by then.  We are looking into it.

That seems like a bad answer if you’re from a low PDI country (mostly because we know the context from the first scenario), but it may be taken at face value because the Low PDI Manager expects straight conversation.  If there was a problem, the Low PDI Manager expected the developer would say something specific about it.  When they didn’t, the Low PDI Manager assumes that Friday will be the date.

And what about the High PDI developer?  He didn’t want to offend the Low PDI Manager, because that’s what you are careful to do in a high PDI country.  The High PDI Developer assumed that the Low PDI Manager would understand his subtext “may be ready” and either ask further questions, or understand that Friday wasn’t necessarily a realistic date.

This is just the tip of the iceberg.  If this happens on a simple conversation about a deadline, what about really big stuff like:

  • Requirements
  • Deliverables
  • Quality control testing
  • Development standards
  • Documentation

The implications are literally staggering.  In fact, I’d go so far as to say the fact that every outsourced project hasn’t failed is something of a miracle.  It’s a testament to having the right people who naturally and instinctually bridge these gaps through extra communication.

The Survey, In A New Light

Getting back to the survey questions, if you look at all of them and how they would be viewed relative to PDI, it’s arguable that PDI differential is the one, single, leading cause that relates to how providers and buyers have a hard time seeing eye-to-eye during the outsourcing process.  Of the eight named factors, I can see 6 of them that directly relate to PDI differential:

  • Buyer’s unclear expectations up front (buyer assumes he is understood when the provider stops asking questions, but that’s a typical low-high PDI interaction)
  • Poor governance (see my deadline example above)
  • Poor communication (again, the deadline example)
  • Poor cultural fit (again, the deadline example)
  • Interests become misaligned over time (you don’t understand each other’s communication needs and are frustrated)
  • Not mutually beneficial (you can’t work together because you don’t understand how to interact…)

Adding those 6 factors up, 72% of project failure reasons can be connected to PDI differential. If both sides understood that single factor going in to the process, everyone would be better served in the end.  Think I’m just making this all up?  It’s not just the buyers that complainHigh PDI country providers say the same things.

So what’s so hard about outsourcing?  It’s hard because of the cultural baggage we bring to the table on both sides, and neither side necessarily realizes it because of assumed interactions.  We need to be more aware of the cultural assumptions going in to projects like this, or we’re doomed to repeat them ad absurdum.

(1) I think it’s fair to say that most other countries would NOT say most Americans are blessed with “natural cultural awareness”.  :)

(2) Before I get lots of angry comments from Indian readers about the interaction above, yes, there are other potential outcomes and perhaps you’ve been on projects where they are all successful.  I have a mixed bag of experience on this, and it’s not about bashing Indian developers.  Like American ones, they run the gamut–good, mediocre, and what-the-hell-are-you-doing-coding.  I’ve run into all three in about the same proportions as American developers, more or less.

(3) I’m sure one or more British readers are horrified at thinking they are interchangeable with Americans at this point.

Outsource Rank Country PDI
1 India 77
2 Thailand 64
3 Mexico 81
4 China 80
5 Indonesia 78
6 Malaysia 104
7 Philippines 94
8 Jordan
9 Egypt 80
10 Bulgaria
11 Hungary 46
12 Ghana 77
13 Pakistan 55
14 Chile 63
15 Poland 68
16 Czech Republic 57
17 Argentina 49
18 Romania
19 Ukraine
20 South Africa 49
21 Russia
22 Vietnam
23 United States 46
24 Israel 13
25 Canada

No related posts.

Flex, Unicode, and Regular Expressions (RegEx)

Posted in Eric Daugherty on January 20th, 2010 by admin

admin originally posted this on EricDaugherty.com.

The internet really helps make the world smaller, and through my open source applications I've enjoyed interacting with a user base that includes many people from outside the United States. iTunes Export has seen adoption in many countries and has challenged me to continually think about internationalization (i18n) issues, from providing the application in multiple languages (English and French today) to handling character sets not traditionally seen in American music. Which brings me to the current challenge.

The problem is that iTunes allows users to create playlist names with characters that are illegal in the file system. This means that I cannot simply reuse the playlist names as the file names for the output playlist files without some processing. My earlier versions attempted to simply replace common illegal characters (such as \, /, ', ", etc.) with an _ character. However, this was rather haphazard, and required constant updates to the filter as I found new characters to filter out. I decided to be reverse the process, so I wrote a regex that replaced all the characters that were not explicitly legal. I started with \s\w which included alphanumeric characters and whitespace. I added additional characters that were legal in most file systems - . [ ] = $ & {}. My resulting regex was: /[^\s\w\-\.\[\]=$&{}]/g. This worked really well for my many of my United States users, but many users from other countries (and some from the US) found it too limiting. Common characters such as é, à, è, ä, ö, ü, Ä, Ö,Ü, and ß were filtered out to an _ even though they were legal characters for file names. So I went back and took another look at the regex.

(Note: \w matches alphanumeric characters but not accented characters like é. However, this is not a bug but matches the ECMA 262 spec)

To aid my efforts, I wrote a simple little utility to determine the Unicode character codes for certain strings, and then test out regex patterns. Here is the utility:

Flash is required. Get it here!

You can type in a string and it will convert it to the Unicode character codes. It also converts it to the regex pattern representation (simply a \x instead of \u). For example, the character é is \ue9 or \xe9. If you enter that as the regex pattern, and then enter the string Tést, you will see the é is matched.

Based on what I've seen, I needed to include the many of the characters from the Latin-1 Supplement Unicode block. Instead of including every character, I limited it to \uc0-\uff. I think that should cover all the normal use cases. The resulting regex patter in: /[^\s\w\-\.\[\]=$&{}\xc0-\xff]/g. This new regex will be included in the next release of iTunes Export (both Console and GUI). The same regex works in Java, although you will need to escape the \ characters, or in Scala use """[^\s\w\-\.\[\]=$&{}\xc0-\xff]""".

Give the above utility a try and feel free to suggest improvements.

For more information on Unicode and Regex: http://www.regular-expressions.info/unicode.html.

iTunes Export 2.2.0 Released

Posted in Eric Daugherty on January 19th, 2010 by admin

admin originally posted this on EricDaugherty.com.

The 2.2.0 release features updates and bug fixes to the console and GUI versions

In both versions:
  • Added option to exclude 'Built In' playlists, including Music, Library, Genius, Movies, etc.
For the GUI version:
  • Added auto-update functionality. Checks for a new version every 7 days. Can be disabled in the 'About' window.
  • Resolved issue where Finish button causes error on 'Select Playlist' screen. Finish button is now disabled on this screen.
  • Added option to 'About' screen to change between languages. English and French currently supported.
  • Improved French translation. Thanks to MrDNA (mrdna@laposte.net) for doing the translation.
For the Console version:
  • Fixed issue with XML validation. This version no longer attempts to validate the XML file (required downloading DTD from apple).
  • Fixed issue where spaces were being converted to underscores in playlist names.
The new version can be downloaded from the project homepage: http://www.ericdaugherty.com/dev/itunesexport/

I have received several feature requests that I am still working on, and will try to get a new release out soon. However, there is enough here to justify a release now.

If you find any issues or have questions please email me (eric@ericdaugherty.com). Please include which application and version (GUI or Console, 2.2.0 etc.) of iTunes Export and the operating system you are using.

FoneMonkey: The Update

Posted in Stu Stern on January 16th, 2010 by Stu Stern

Stu Stern originally posted this on Big Gorilla - Stu Stern's Blog.

This release of FoneMonkey, the world's first and only free and open source testing automation tool for the iPhone, fixes various bugs, adds support for additional Cocoa Touch components, and can be run on an actual iPhone device in addition to the simulator.

The latest FoneMonkey download and release instructions can be found here.

We expect to have the official FoneMonkey project site online later this month!

Java profiling with DTrace

Posted in Wayne Adams on January 15th, 2010 by admin

admin originally posted this on Wayne Adams' Blog.

DTrace, a dynamic tracing framework on Solaris (see dtrace(1m)) is a valuable and extremely easy-to-use tool if you find yourself analyzing Java performance issues on Solaris. While there is a lot of information on DTrace available (see the DTrace Wiki, which includes an extensive user guide), less information is available on using DTrace specifically with Java applications. My motivation with this post is to focus on profiling Java applications with DTrace.

Quick Tour

If you're not at all familiar with DTrace, look at the Getting Started section of the user guide. In short, DTrace allows you to specify a set of actions you would like performed at a location of interest (in the kernel or an application). A location of interest is referred to as a probe. In addition to a numerical ID, each probe is identified by the following four pieces of information:
  • The probe Provider name
  • The probe Module
  • The name of the Function where the probe is located
  • A Name for the probe, designed for human readability
(By the way, if you don't have a Solaris environment but would still like to follow along, you can install VirtualBox, then retrieve the ISO image for OpenSolaris, then follow the VirtualBox guide to get up and running on OpenSolaris).

What probes are available? Well, a lot. On my OpenSolaris (2009.06) environment, a count (using pfexec dtrace -l | wc -l), yields 60068 probes! (Note: this includes some Java-6-specific probes; more on that later). Most of these are fairly low-level probes from the point of view of a Java profiler, but let's take one and see what we can do with it.

Using DTrace, we can perform actions when the read system call is entered, using the following:
     pfexec dtrace -n syscall::read:entry
If you enter this, you'll begin to see output such as the following:
  0  57922                       read:entry 
0 57922 read:entry
0 57922 read:entry
0 57922 read:entry
0 57922 read:entry
Note this doesn't tell you a lot; you don't know the PIDs of the processes making the calls, and so on. DTrace has a number of built-in functions and variables that you can use to provide meaningful output at these locations. I won't go into this in any detail because there is a large body of information on these topics at the links I mentioned above. This is just the very quick tour to get to, and focus on, Java-specific probes.

At this point, before going any farther, you would begin placing your declarations and logic in a file. DTrace understands the D programming language, a language that will be familiar to most developers (especially C developers). The above syscall::read:entry example, placed in a file and enhanced to 1) apply only to a specified PID, and 2) to print out the name of the process, follows:

syscall::read:entry
/pid == $1/
{
printf("'%s' (PID: %d): entered syscall:read", execname, pid);
}
Place the following text in a file called blogExample.d, then find a process to monitor (I picked Firefox) and enter the following, substituting the PID of the process of your choice:
     pfexec dtrace -s blogExample.d 697
and you should see output similar to the following:
0  57922                       read:entry 'firefox-bin' (PID: 697): entered syscall:read
0 57922 read:entry 'firefox-bin' (PID: 697): entered syscall:read
0 57922 read:entry 'firefox-bin' (PID: 697): entered syscall:read
0 57922 read:entry 'firefox-bin' (PID: 697): entered syscall:read
Note the predicate (/pid == $1/) in the example which restricts actions to be performed only when pid (a built-in DTrace variable) is equal to the first argument supplied on the command line ($1). To suppress all output except for that specified in your printf() call, add the -q ("quiet") switch (you'll need to add a newline to the end of your printf() specification, also, as "quiet" mode does not provide one):
     pfexec dtrace -q -s blogExample.d 697
and you should see output similar to the following:
'firefox-bin' (PID: 697): entered syscall:read
'firefox-bin' (PID: 697): entered syscall:read
'firefox-bin' (PID: 697): entered syscall:read
'firefox-bin' (PID: 697): entered syscall:read
'firefox-bin' (PID: 697): entered syscall:read
That wraps up the quick tour of DTrace. Now, on to our main topic!

DTrace Support in Java

Java SE 6 now supports DTrace, in the form of a couple of providers (specific to the HotSpot JVM):
  • hotspot
  • hotspot_jni
For more information, see DTrace Probes in HotSpot VM. We'll be focusing on the hotspot provider in this post. Normally, you would expect to find a provider with the dtrace -l command, looking for "hotspot". This approach gets you most of the way there; actually, providers are on a per-JVM basis, and they are identified by appending the PID to the name "hotspot." So, for example, if you have a Java process running and its PID is 1234, then there will be a DTrace provider called "hotspot1234". This naming convention is the reason that you will see references to "hotspot$1" in example D-language scripts -- the author is expecting a PID on the command line and is using it to create the provider name. This convention also spares us from having to restrict our probe actions to a specific PID, as the restriction is built in to the choice of provider.

Back to the Java-specific probes. Enter the following command in a shell:

pfexec dtrace -l | grep hotspot

and you should see a list of all the hotspot and hotspot_jni probes. On my system, I got 499 probes with one Java process running; note there will be a complete set for each JVM. I have 26 "hotspot" probes, and 473 "hotspot_jni" probes. I will only be looking at "hotspot" probes in this post. You can also get a list of these probes, along with their arguments, in the Reference section of DTrace Probes in HotSpot VM.

Some Examples

Thread Monitoring

Note the following two probes related to thread activity:
  probe thread-start(char* thread_name, uintptr_t thread_name_length,
uintptr_t java_thread_id, uintptr_t native_thread_id, bool is_daemon);
probe thread-stop(char* thread_name, uintptr_t thread_name_length,
uintptr_t java_thread_id, uintptr_t native_thread_id, bool is_daemon);
One interesting detail to note with string arguments in DTrace is that they come in pairs, where the first item is a pointer to the un-terminated string and the second item is the length of the string. Because of this convention, in example code you'll frequently see the following construct, which builds a null-terminated string from the provided information:

self->str_ptr = (char*) copyin(arg0, arg1+1);
self->str_ptr[arg1] = '\0';
self->stringVariable = (string) self->str_ptr;


Note that the above fragment could be used to construct the name of the thread from the above thread-start and thread-stop probes.

To see how easy it is to use DTrace to create a simple thread profiler on Java 6, open a file in a text editor and enter the following:
     hotspot$1:::thread-start
{
self->ptr = (char*) copyin(arg0, arg1+1);
self->ptr[arg1] = '\0';
self->threadname = (string) self->ptr;

printf("thread-start: '%s' (thread ID = %d; native thread ID = %d; is_daemon? %d)\n",
self->threadname, arg2, arg3, arg4);
}

hotspot$1:::thread-stop
{
self->ptr = (char*) copyin(arg0, arg1+1);
self->ptr[arg1] = '\0';
self->threadname = (string) self->ptr;

printf("thread-stop: '%s' (thread ID = %d; native thread ID = %d; is_daemon? %d)\n",
self->threadname, arg2, arg3, arg4);
}

Note as mentioned earlier the $1 macro variable, representing the JVM process ID you will be passing in on the command line. In this script, we output everything we get from these two probes, which is the thread name, ID, native ID, and a boolean "is_daemon". Save this file as threadWatcher.d and enter the following:

pfexec dtrace -q -s threadWatcher.d pid

substituting the process ID of a running JVM. In my test Java application, I then opened a file chooser and got the following output from DTrace:

thread-start: 'Basic L&F File Loading Thread' (thread ID = 42; native thread ID = 42; is_daemon? 0)
thread-stop: 'Basic L&F File Loading Thread' (thread ID = 42; native thread ID = 42; is_daemon? 0)
thread-start: 'Basic L&F File Loading Thread' (thread ID = 43; native thread ID = 43; is_daemon? 0)
thread-start: 'Basic L&F File Loading Thread' (thread ID = 41; native thread ID = 41; is_daemon? 0)
thread-stop: 'Basic L&F File Loading Thread' (thread ID = 41; native thread ID = 41; is_daemon? 0)
thread-stop: 'Basic L&F File Loading Thread' (thread ID = 43; native thread ID = 43; is_daemon? 0)

Garbage Collection

Note the following probes relevant to garbage collection:

probe gc-begin(bool is_full);
probe gc-end();
probe mem-pool-gc-begin(
char* mgr_name, uintptr_t mgr_name_len, char* pool_name, uintptr_t pool_name_len,
uintptr_t initial_size, uintptr_t used, uintptr_t committed, uintptr_t max_size);
probe mem-pool-gc-end(
char* mgr_name, uintptr_t mgr_name_len, char* pool_name, uintptr_t pool_name_len,
uintptr_t initial_size, uintptr_t used, uintptr_t committed, uintptr_t max_size);

The first two probes provide generic information about the start and stop of a garbage collection, while the latter two provide more detailed information. We'll output information from both to compare. Note that the manager name and pool name are provided with the same pointer/length data pairs as we saw earlier.

Open a file called gcWatcher.d in a text editor and enter the following:


hotspot$1:::gc-begin
{
printf("gc-begin: is_full = %d\n", arg1);
}

hotspot$1:::gc-end
{
printf("gc-end:\n");
}

hotspot$1:::mem-pool-gc-begin
{
self->ptr1 = (char*) copyin(arg0, arg1+1);
self->ptr1[arg1] = '\0';
self->mgr_name = (string) self->ptr1;

self->ptr2 = (char*) copyin(arg2, arg3+1);
self->ptr2[arg3] = '\0';
self->pool_name = (string) self->ptr2;

printf("mem-pool-gc-begin: mgr: '%s'; pool: '%s'; initial: %d; used: %d; committed: %d; max: %d\n",
self->mgr_name, self->pool_name, arg4, arg5, arg6, arg7);
}

hotspot$1:::mem-pool-gc-end
{
self->ptr1 = (char*) copyin(arg0, arg1+1);
self->ptr1[arg1] = '\0';
self->mgr_name = (string) self->ptr1;

self->ptr2 = (char*) copyin(arg2, arg3+1);
self->ptr2[arg3] = '\0';
self->pool_name = (string) self->ptr2;

printf("mem-pool-gc-end: mgr: '%s'; pool: '%s'; initial: %d; used: %d; committed: %d; max: %d\n",
self->mgr_name, self->pool_name, arg4, arg5, arg6, arg7);
}
Save the file, then trace your Java application with the following:
     pfexec dtrace -q -s gcWatcher.d pid

For my example, after triggering a little activity in my application, I saw a number of minor collections; here is the output from one of them:

gc-begin: is_full = 0
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Code Cache'; initial: 163840; used: 1751360; committed: 1769472; max: 33554432
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Eden Space'; initial: 917504; used: 917504; committed: 917504; max: 4194304
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Survivor Space'; initial: 65536; used: 58904; committed: 65536; max: 458752
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Tenured Gen'; initial: 4194304; used: 1406096; committed: 4194304; max: 61997056
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Perm Gen'; initial: 12582912; used: 1144824; committed: 12582912; max: 67108864
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Perm Gen [shared-ro]'; initial: 8388608; used: 6733248; committed: 8388608; max: 8388608
mem-pool-gc-begin: mgr: 'Copy'; pool: 'Perm Gen [shared-rw]'; initial: 12582912; used: 7676960; committed: 12582912; max: 12582912
mem-pool-gc-end: mgr: 'Copy'; pool: 'Code Cache'; initial: 163840; used: 1751360; committed: 1769472; max: 33554432
mem-pool-gc-end: mgr: 'Copy'; pool: 'Eden Space'; initial: 917504; used: 0; committed: 917504; max: 4194304
mem-pool-gc-end: mgr: 'Copy'; pool: 'Survivor Space'; initial: 65536; used: 29648; committed: 65536; max: 458752
mem-pool-gc-end: mgr: 'Copy'; pool: 'Tenured Gen'; initial: 4194304; used: 1458336; committed: 4194304; max: 61997056
mem-pool-gc-end: mgr: 'Copy'; pool: 'Perm Gen'; initial: 12582912; used: 1144824; committed: 12582912; max: 67108864
mem-pool-gc-end: mgr: 'Copy'; pool: 'Perm Gen [shared-ro]'; initial: 8388608; used: 6733248; committed: 8388608; max: 8388608
mem-pool-gc-end: mgr: 'Copy'; pool: 'Perm Gen [shared-rw]'; initial: 12582912; used: 7676960; committed: 12582912; max: 12582912
gc-end:

Note that not a great deal of information is available from the generic garbage-collection begin and end probes, while much more information is available from the mem-pool probes.

You can do much more than simply print the values of probe arguments, of course; see the reference documents for more information. As an example, you can count and output the number of threads created in a time period, gather statistics on the average lifetime of a thread, output the average memory recovered during a garbage collection, and so on.

Other Considerations

One question you might have is "How expensive are these operations?" It depends on the operation. The probes we have looked at above are relatively inexpensive. Other probes, such as method-entry and method-exit probes, or those monitoring object allocation, are much more expensive and indeed have a significant cost when enabled, even when they are not being used.

For this reason, there is a JVM option --
-XX:+ExtendedDTraceProbes -- which when present enables these and some other expensive probes. For more information, see the "Effect of using DTrace with Java Programs" section of the Observability Using Java Platform, Standard Edition 6, and Solaris OS article. If this option is not enabled, DTrace will ignore any of the above-mentioned probes in your profiling application, sparing you from inadvertently slowing down someone's production server.

That's the end of this quick tour. As I mentioned earlier, there are a lot of resources available on DTrace, specifically the user guide on the DTrace wiki (see the "Documentation" link, or go directly to this link).

Special Characters in Flex 4

Posted in Justin Shacklette on January 12th, 2010 by admin

admin originally posted this on Saturnboy.

Recently, I was caught by a special characters vs. html entities issue in Flex 4. For reference, you can read more about special characters in the text property of a text component in the official docs. And also here on Flex Examples. Unfortunately, neither of these was exactly what I was looking for.

The Problem

I had an array of names in ActionScript that potentially contained special characters, and I wanted to output them in a spark List component. Nothing magical required, just get them on the screen.

Solution #1

One option, which I’ve NEVER seen in anyone’s code ever, is to move the definition of the array into its own Script tag without the CDATA block. For example:

<fx:Script>
    [Bindable] private var nuggets:Array = [
        'Carmelo Anthony',
        'Chauncey Billups',
        'Nen&#234;',
        'Kenyon Martin'];
</fx:Script>

Now, it doesn’t matter if we use single quotes (') or double quotes ("), because outside of the CDATA block all numeric html entities are processed. The fact that Flash Builder 4 automatically inserts the CDATA block when you open a Script tag probably means that almost no one has ever even heard of this possible solution. It’s so weird, I can’t recommend this solution.

Digging Deeper: The official docs will tell you that only a few named html entities work (&lt;, &gt;, &amp;, &quot;, &apos;), and after that you must use numeric html entities (&#NNN;).

Solution #2

Another option, that I actually thought of first, is to process the numeric html entities via a regular expression to output the correct special character. For example:

private function makeSpecialChars(item:Object):String {
    return item.toString().replace(/&#\d+;/g, replaceFunc);
}
 
private function replaceFunc():String {
    var s:String = arguments[0];
    s = s.substring(2, s.length - 1);
    s = String.fromCharCode(parseInt(s));
    return s;
}

We use a simple regular expression to match any numeric html entity, and then call a replacement function to do the work of converting the entity into a special character. The static method String.fromCharCode does the actual conversion.

Putting the converter code together with a List’s labelFunction property and we get this:

<fx:Script>
<![CDATA[
    [Bindable] private var nuggets2:Array = [
        'Carmelo Anthony',
        'Chauncey Billups',
        'Nen&#234;',
        'Kenyon Martin'];
 
    private function makeSpecialChars(item:Object):String {
        ...same as above...
    }
]]>
</fx:Script>
 
<s:List dataProvider="{new ArrayList(nuggets2)}"
    labelFunction="makeSpecialChars" />

For each element in the array, the labelFunction gets called and any numeric html entity is converted into the corresponding special character. No magic.

The Result

Here is the final result with both Solution #1 and Solution #2 together (view source enabled):

Flash is required. Get it here!

If you view the source, you can see that I’m using two separate Script tags, one with a CDATA block and one without. Who does that?

Files

Hiring Ren and Stimpy

Posted in Dave Rodenbaugh on January 11th, 2010 by admin

admin originally posted this on Lessons of Failure.

Since we’re on the topic of hiring and interviewing, let me regale you with a story about two developers.  This is an entirely fictional tale, but drawn from real-world observation and experience over 17 years of employment.

Suppose you had two new junior developers at your company:  Ren and Stimpy.  Both have similar backgrounds, education and experience.  Both passed the interview process and were inducted into your team.  Both are bright, ambitious and self-starting.  Any resemblance to the long-running popular cartoon is purely coincidental.  Stay on target, Red Five.

Your New Star Developers: Ren & Stimpy

Ren started out on the first day setting up his computer, checking out the code and understanding what his first assignment was.  Within a week, he mastered it and went to his managers for more work.  After a few more assignments, he started to get bored and whine about how menial his tasks were, even though he was a junior developer.  He started to sulk to his manager about how he could “do so much more” than just writing unit tests for some “barely used modules.”

Ren would often interrupt have impromptu meetings with other senior developers and managers hoping to get more interesting and challenging tasks.  Often times, he’d get more of the same kind of work from them.  Instead of taking lunch and wasting valuable time, he would often work through lunch and stay late to get these tasks done.

Stimpy started much the same way.  He setup his machine, checked out the code and worked on his first assignments.  He too, asked for more work when it was finished, but he started recognizing a pattern to the work he was given.  The billing system testing he was doing involved some similar functions and classes that could easily be scripted.  He worked on this side project and then let his manager know that all future tests were now written.  This made the manager’s job even easier, since the senior developers could now use this script to auto-test things as they came out with new requirements and designs without further delay.  He spent his lunch hours socializing with members from other teams to get to know them better.

Stimpy also started looking around at other things going on with the team.  He found the SVN repository was a complete mess.  He started asking other developers how it ended up that way and got a broad perspective on how the company organized code over time.  He spent extra hours creating a new repository that completely reorganized the code in the current taxonomy which made everyone’s lives better.

Stimpy noticed that the company’s development document files were just sitting around on a shared directory.  He decided to create a wiki (after first talking with the network admins to see if one already existed, and finding the ideal machine on which to install it) and upload all the docs by project.  He published this to his manager, who in turn shared it for the very first time with Marketing and Sales, so they could see the progress on engineering projects.

Stimpy was asked to sit in on several design meetings to get his input because he was able to grasp the historical significance of past development based on his repository migration (with the benefit of hindsight, he had a unique perspective that even the original developers lacked), so they wanted to know what could be salvaged from old projects and what needed to be rewritten.  Ren was still working on his unit tests and complaining to his manager about it.

It’s pretty obvious which of these developers is going to make a positive, lasting contribution to the team and which might be more of a drag on productivity.  Five things really separate these two individuals:

  1. Understanding how things work in an organization. Stimpy took it upon himself to learn about the “other infrastructure” besides his tiny module of code.  He got the bigger picture, which made it easier to understand the importance of little things and which things are more critical than others.
  2. Knowing the roles of others and how/why they do their jobs. By interacting with others (like the network admin and the other developers outside of his immediate scope), Stimpy understood how all the parts fit together to make a unified machine.  And he saw where things in that machine needed maintenance.
  3. Finding where gaps exist, and taking the initiative to fix them. How many times have you heard, “Well, we don’t have time to do that right now.”  (Sounds a little like Habit 5:  Fix It Later, don’t you think?) Doesn’t matter what the question is, you’ll get that answer a lot from most people in any company.  If you’re the one person that is willing to do extra stuff (with a smile, of course) and still gets your own job done, you’ve already risen above the pack.
  4. Have no fear of repetitive, monotonous, or boring “dirty work”-be willing to get it done. Of course no one likes this kind of work.  It would be called something different if that were the case.  But honestly, what do you remember more vividly?  The guy who fixes the annoying SVN problems the entire team suffered with for 3 months, or the guy that created an awesome rainbow table for his code from scratch in a weekend?
  5. Stepping up and taking responsibility for things outside of your job description. Arguably, they both did this, but Ren was looking for the sexy, resume-filling work, where Stimpy was just getting things done, no matter what they were.  Never did he say, “It’s not my responsibility…”  If you want to get ahead, actively find things outside of your job description that you can do and that would have a beneficial impact on the company and do them.  Don’t whine about not being challenged by your managers (or teachers).  There’s only one person in control of your life:  youDon’t give that power up, ever.

No matter whether you’re starting your first job, or your 10th, always ask yourself:  What kind of  developer do I want to be? Ren or Stimpy?  And if you can sing the Happy Happy Joy Joy song to the delight of your team, so much the better.

Related posts:

  1. Stop Dumbing Down The World with Bad Interview Questions

Coupling Design and Implementation

Posted in Jerry Andrews on January 10th, 2010 by admin

Six weeks ago or so, our development team reviewed a small design change in the way status is managed by an object. Basically, we broke one state variable into four, and thought more carefully about the state transitions allowed and expected for the class in question.

Yesterday, one of our analysts, at the prompting of one of our developers and two of our data designers, reviewed a completely new take on the same design change… none of the guys in question knew about the previous design change (they’d missed the design review). We ended up going with the previously-reviewed design.

The whole meeting and the thought that led up to it could have been avoided if there had been some mechanism for ensuring the approved design was implemented. This bit of design, like all design, is pretty much wasted if it remains in design documents.

One way to handle this would have been to somehow automatically compare the existing design artifacts to the implementation to see if they matched, and complain if they didn’t. Even if we didn’t implement the approved design right away, then, there’d be a mechanism for reminding developers that they planned to do something one way, and haven’t made it happen yet.

Three snippets of “interesting” code

Posted in Jerry Andrews on January 7th, 2010 by admin

Here’s a bit of code I use in interviews:

try {  if (foo())    return 1;} catch (Exception e) {  throw e;} finally {  return 3;}

The questions I start with are:

  1. what does this do if foo()==true? If foo()==false? If foo() throws an exception?
  2. how could you recode this more simply?
  3. the original spec was:
    • if foo is true, return 1,
    • if foo is false, return 3,
    • propagate exceptions.

Provide code to implement the spec.

It’s sobering to note that over 2/3 of interviewees for senior Java programmer positions fail all three questions. How would you answer them?

I firmly believe that even mid-tier developers should have no problem describing what they mean in code, and in understanding what others mean, even in code which is poorly written. There always seems to be a snarl somewhere that nobody wants to touch (I’ve written one or two of those myself). For the most part, though, the bad code I see is the result of “I’m not sure how to do this, but this seems to work”.

Here’s one such snippet:

static final BigDecimal ONE_HUNDRED_PERCENT = new BigDecimal("1.00")    .setScale(MathUtils.DOLLAR_SCALE, MathUtils.STANDARD_ROUNDING_MODE);

This is just bad code, unless you’re letting BigDecimal manage all the results’ scales itself (we aren’t, and you shouldn’t; see my previous article on BigDecimal rounding). It’s completely replaceable with “BigDecimal.ONE”. It leaves the resulting code less clear, and performs no useful function.

I found the following code (paraphrased) in a Java application I’m working on:

foo(vc.getNewValue(), ppCorrectedValue=vc.getNewValue());

It’s perfectly correct and exactly equivalent to:

ppCorrectedValue = vc.getNewValue();foo(vc.getNewValue(), ppCorrectedValue);

That sort of expression is common in C; most C developers are aware that the equals operator evaluates to the left-hand value of the expression. Many Java developers aren’t aware of this fact–so I was surprised to see it. I’m not sure it improves the readability of the code, though, so I’m refactoring it into the second form above.

Where I think the equals operator behavior really helps in Java is when setting up initial values:

i = j = 0;

Good code is hard enough to write; have pity on the next guy, and be as straightforward and clear as possible. Arabesques like assignment inside a function call parameter list just make your code harder to read and maintain.