Quick!  Answer the following question without thinking about it:

How would you rate your programming skills?  (Below Average, Average, or Above Average)

Based on psychological studies across many different groups, about 90% of all programmers will answer “Above Average”.

Of course, that can’t possible be true.  In a group of 100 people, 50 are above average, 50 are below average.  This effect is known as Illusory Superiority.  It’s been documented in many domains and even if you’re aware of it, you’ll probably still answer the question above as “Above Average”.

For even more fun, try asking that question to every programmer you know.  Don’t ask them in a group, ask them one-on-one to get more ‘honest’ answers.  You’ll get similar results, even from people who you know can’t program their way out of a wet paper bag (this is the Dunning-Kruger effect, but it’s related).  It’s an epidemic in our profession.

Now, let’s suppose for a second that you’re right–you are actually above average.  You are da man.  A rock star.  God like capacities amongst mere mortals.  Keyboards bow in reverence to you as you approach.  Trumpets sound on high when you commit on GitHub.

If you’re above average, then chances are you’re an expert at what you do.  Calling yourself an expert sounds quite compelling–you get respect, deference, and prestige by being one.

Being an expert means you know it all about your subject. Unfortunately, it also means you’re going to get lazy.  It means you’re going to eventually rest on your laurels and sit around thinking you’re better than everyone else instead of actually working to get there.  Your expertise will become a liability because you stop trying to learn.  Maybe not today, but soon enough.

Instead, why not consider the more likely possibility?  You’re average, or heaven forbid, below average. Aside from the personal stigma you might suffer here, think for a second about the real benefits of doing this:

• By assuming you’re not at the top of the pack, you now have incentive to get there
• By assuming you’re not the smartest person in the group, you now have the opportunity to learn something
• By assuming you’re not the best at what you do, you’re going to work harder to improve yourself

Perhaps you’ve heard of beginner’s mind?  Summed up by a zen master in classic koan-brevity:

In the beginner’s mind there are many possibilities, in the expert’s mind there are few.

The trap of calling yourself ‘expert’ at software development means that you pigeonhole yourself into some language (Java, Ruby, PHP), some industry (medical devices, social networking, games), or some specialty (embedded programming, enterprise software).  Once you establish that expertise, fear of failure arises when you consider going outside that comfort sphere.  With your golden hammer of experience, everything appears to you a nail.  You stop thinking about screwdrivers and all other possible relevant tools because they’re no longer inside your ‘expertise’.

This is why starting out in software you wonder why “experienced programmers” can’t get X, when you just learned X in a matter of days.  X could be anything:  closures, object-oriented programming, Ruby on Rails framework, Haskell programming.  It doesn’t matter in the end, the expert’s mind is cluttered with old knowledge.  The beginner’s mind is open, free of hindrances.

It’s harder to learn when you’re an expert.  And this is why being the ‘expert programmer’ is dangerous.

So what’s the number one thing you can do to be the best programmer out there?  Start by considering yourself below average. Step out of your comfort zone.  Be the averagest.

A master never stops learning, and neither should you.

No related posts.

Jakob Nielsen recently published his report on iPad’s usability and application interface consistency.  To no one’s surprise, he discovered a few issues.

What was the main problem Jakob uncovered?  Many iPad applications aren’t obvious to use:  non standard controls, confusing graphics, and counter-intuitive metaphors.  In psychology parlance, it’s because these apps lack constraints and affordances.

Constraints and affordances are nothing new, even in the real world.  You encounter them every day, but you’re only barely aware of them most of the time.  Joseph Hallinan has a great example in “Why We Make Mistakes”:

“One way to reduce errors is by introducing constraints.  Constraints are essentially simple mental aids that keep us on the right track by limiting our alternatives.  Try repeating the Star Spangled Banner if you’re American without singing it.  How much can you remember?  Now let yourself sing it.  I’ll bet you get most, if not all, of the song.  That’s a constraint against forgetting the song because that’s the only way you’ve ever learned it.

Another way to reduce errors is to use affordances.  If constraints tell us what we can’t do, affordances indicate what something can do.  Affordances may appear in many forms:  texture, shape, or size may indicate usage.  For example, a ball’s shape affords bouncing or throwing.  A knob affords turning.  Slots afford the insertion of things.  When we encounter some basic object, affordances help us answer basic questions like, “How does this work?” and “What is this used for”

An everyday affordance you’re familiar with is the pull-handle on a door.  Just by looking at it, you can immediately tell that you’re supposed to grasp on to it and pull the door open.  The opposite affordance would be the push-pad on a door, where there’s nothing to grab, only a metal surface against which to exert force as you pass through it.  Sometimes constraints are added on top of this, such as “Exit” signs on the door you use to go out as well.  These are all the clues we get on how to use a door, but you’re rarely thinking to yourself, “Oh, there’s a PULL handle, I should PULL on it.”  Affordances are subtle, important cues on how to interact with things.

Those cues can go awry.  For example, building architects foul it up occasionally by putting pull handles on the push side of doors.  Or put the “Exit” sign on the “Enter” side with an arrow pointing to the other door.  These confuse us, make us hesitate, and often force us to enjoy the embarrassment of pulling on the door you’re supposed to push (a personal favorite of mine).  Sometimes they’re done on purpose, like at a famous bar I went to in college that has two doors side by side each a large arrow pointing to the other door and a gender.

They are intentionally mislabeled so women walk into the men’s restroom and vice-versa, much to the delight of the inebriated patrons.  While that sort of bad affordance was humorous in college, it’s patently frustrating to encounter when I’m struggling with a new app on the iPhone.

This lack of affordances is why the iPad and iPhone can be so bloody hard to figure out sometimes.  We have no hints–no obvious clues what to do with things mostly because the metaphors are new and the controls are often non-standard.  Consider this interface below, heavily laden with nothing but custom graphics in the UI.  Can you tell which items allow you to interact with them, and which are merely display artifacts?

Yes, I’m confused too.  And to add to the confusion, these devices allow actions we couldn’t possibly take on the desktop, like the accelerometer.  How do you give an affordance for that?  Or the shaking action?  We aren’t used to these metaphors yet, so the affordance isn’t quite obvious.

Contrast that with this one from the iPad, where the actions you can take are fairly clear from the interface.  (Courtesy of LandingPad.org).  Without knowing what the app is all about, you can already tell (if you’ve used touch interfaces, particularly iPhone) what you can touch, what will likely result in an action, and what items are just eye candy.

There are even more great examples on that site, I encourage you to check them out.

User interface design is hard.  Punting to a graphic designer sounds like a good idea, but consider that the difference between adequate and excellent on a user interface isn’t subtle or small.  It’s the difference between usable and frustrating.  The difference between good and beautiful.  And maybe even the difference between viral and doomed to obscurity in the App Store’s bargain bin.  Make sure you don’t just create pretty graphics, put the time in to make a pretty AND usable UI.

Don’t let your users walk into the wrong gender bathroom in your applications:  use affordances to make your UI obvious and intuitive.

No related posts.

So, think you’ve got what it takes to be a consultant?  Feeling the itch because your current job isn’t motivating you like it used to?

The independence, prospect of better money and the potential for starting your own business make this idea very seductive. A million others have tread this path before you so you’d think it would be easy, right?

Nope. Not even close.

The path to success is littered with the rotting carcasses of those who went before you and failed. Sort of like the great gold rush of 1849 to San Francisco. If you really want to get into it, even after this stern warning, you have my pity. This is anything but an easy path to success, wealth or fame.

Still not daunted? That’s a decent start. Persistence is part of the key to success here.  So is intelligence.  And confidence.  But those things just get you in the door, past the obnoxious bouncer so-to-speak.  It’s early in the night, you have to order a drink and do karaoke before you can hit the dance floor.  In other words, there’s a lot more to it than you think.

Rather than cover the business aspects of consulting, or even the how to make the jump article, I thought I’d share the secrets of successful consultants I’ve come in contact with over the years. These can make or break your career as an independent contractor. Ignore them at your own peril.

Secret #1: Your reputation is your livelihood. Protect it like your life depends on it.

Your career does actually depend on it. People hire contractors for two reasons: someone said they were worthwhile, or they thought you were the cheapest one available. You get far more jobs from the first recommendation than the second, and the quality of work is higher when they pick you for your reputation rather than your rate.

Secret #2: You are always looking for a new gig. Even when you have a current gig, keep your eyes open at all times.

Unless you’re lucky enough to get two year, iron-clad contracts, you’re always keeping an eye on the end date of your current gig, trying to figure out “What’s next?” If you get a job and think, “Oh, they’ll just keep extending me over and over”, you’re not a consultant, you’re thinking like an employee. You’ll be kicked out as soon as someone realizes it. Pay attention to new opportunities coming around, and have something in your back pocket if your current contract ends or is canceled early.

Secret #3: Assume that if you’re not adding value, you’re overhead and you’ll be fired.  Constantly find ways to add value to a project.

This ought to be a rule for employees in general, but sadly, there are too many companies that allow for this sort of inertia to exist in their ranks indefinitely. Often times, this is why a company hires consultants (to get something done), so make sure you’re the person that “gets things done”. I’m already assuming that you’re smart, now you need to deliver on it.

Secret #4: You’re never an employee at the company you contract for, no matter how long you’re there.  Maintain a certain distance.

Making friends at your contract jobs is good. Acting like you’re an employee and leaning against the water cooler, chatting with the gang mid-morning is not. Contractors are supposed to be billing every hour they’re on site. Managers tend to watch consultants more closely than employees because they pay top dollar for good contractors and are paranoid about wasting their money. Don’t be the reason you get tossed out.

Secret #5: Network constantly. Everyone is a potential new client.

You never know who is going to need a contractor at their company. You never know who will be promoted to a manager or executive and suddenly be looking for that certain person with some mad skillz to come in and take over some failing project. People end up in unexpected places, and making enemies is never a good idea. Leave companies gracefully, never burn bridges, don’t speak badly of others, even if you have some personal problems with them. These things always come back to haunt you at some later point, without exception. Don’t believe me? You won’t be the exception. Trust me.

While I don’t advocate the smarmy salesman approach, handing out business cards in the restroom at the local bar, you should stay connected. Make sure you keep in touch with people. LinkedIn is a great way to do that. Facebook is another. MySpace, maybe not so much.

Secret #6: Finish what you start. Deliver what you promise.  Uphold rigorous ethical standards.

The four fastest ways to end your consulting career:

• Fail to finish a project you were given.
• Leave before your project is finished because you wanted a better, more lucrative job.
• Lack the expertise you claim to have and accept a job based on that premise.
• Do something unethical.

Did I mention your reputation was everything? Software is a small community wherever you practice it. You’ll run into the same folks again and again. If you make a bad impression because of those items above, you can bet they will be remembered somewhere down the line when you’re interviewing at another company, if those people already work there.

To confirm reputations, many managers will ask if anyone on their staff already knows this person (Yep, Secret #1 again). A negative reputation will give the manager a reason to move on to the next candidate. Don’t give them any ammunition.

You need to be cautious about finding new work while employed with another company.  You want to maintain your reputation while still networking.  It’s a tricky balance, but we’ve already established you’re smart, so this should be no problem.

As far as ethics goes, do I even need to talk about how unethical behavior has screwed pretty much everyone in the world in the last year?  OK, good.  Moving on.

Secret #7: Consultant is generally synonymous with expert. Make sure you qualify.

You shouldn’t just start consulting right out of school or with little experience. (Accenture, I’m looking at you here with your army of under trained and overpriced pretty boys and girls that you send into large companies like a virus where they try to spread all over the place without being able to tie their own shoes) There are notable exceptions, like where your field of study was highly specialized and you acquired expertise in school along the way from your professors, internships or colleagues. But for software development, you need a few solid projects under your belt to really be able to call yourself a serious consultant. If I had to put a number on it, maybe no less than 5 years experience prior to consulting. There’s no hard-and-fast rule to it, but you should actually know something, otherwise you can’t add value (Secret #3).

Secret #8: You can try to fake it before you make it, but you’ll only get so far.  Represent yourself honestly.

There are folks out there that will tell you that a consultant is only 5 pages ahead of the customer in the same book. Sometimes that’s true. Most of the time, it’s not. You need to know your field well enough to be the expert (Secret #7). Not just sound like one, but really walk the walk here.

Secret #9: Consulting in a niche or a general field can be both good and bad. Choose your specialty carefully.

You may get two kinds of advice about entering consulting:

• Stick with something that is popular, but crowded (like Java, C++, .NET)
• Stick with something that is specialized, but has less competition (e.g. Delphi, Embedded C development, etc)

There are pluses and minuses in both sets and neither in my opinion really wins out.

• The popular fields mean you have more work available to you, but competition is much higher which pushes the overall rate for contractors lower in that skill. Java programmers are in this exact situation today.
• The less popular subject areas command much higher rates because of the skill rarity, but finding new projects is much harder and you might be out of work more often. Google Go programmers are in this situation today.

Ideally, you want to be the early adopter of a new technology that will take over the industry, like learning Java in 1995. Predicting those events is nearly impossible, so I’d suggest sticking with the things that interest you. After all, you’ll spend 30% of your life doing them. No sense in punishing yourself needlessly.

Secret #10: You are a professional. In every situation, make sure you act like one.

New contractors are tempted to act just like those around them in a consulting gig: playing politics, cracking jokes, drinking too much at social events, and the like.

The short answer is: Don’t. Not ever. Even when you’re off the clock, you’re still being judged on your behavior. Remember that reputation thing (Secret #1)? Yeah, it still applies at happy hour, or launch parties, or even casual conversation. What about office politics? Understand them but steer clear of participating as much as you can. You’re not an employee (see Secret #4), so stand above that. You’ve been hired to add value, not noise (Secret #3).

If by now, you’re yelling at the monitor saying, “Dave, these aren’t secrets! Everyone knows them!”, then I applaud you. You’re in the absolute minority and you have what it takes to succeed wildly in this field. So go forth and conquer the consulting world with your intuition.

The rest of us will just have to follow this list.

No related posts.

Top Ten Reasons Babies are better than the iPad

1. Babies eventually grow up to be better than their fathers.
2. Babies get cuter as they get older.  Think those fingerprints on your screen will get any cuter?
3. Babies don’t have a daily purchase limit.
4. After you get a baby, most people are satisfied for awhile and don’t want to upgrade for a long time.
5. Babies make other people smile when they see them.  iPads just make people ask, “What’s that?”
6. iPads look dorky in strollers.
7. Putting an iPad on your back makes you look like a Borg.
8. Babies don’t care if you use Flash, Objective-C, or Lua.  Babies just want you to talk to them in any language.
9. Babies don’t require you to dress in black turtlenecks, khakis, or attend MacWorld for any reason.
10. Babies are a LOT more fun to make than standing in line to buy an iPad

AND!

Top Ten Reasons iPads are better than Babies

1. iPads never require college educations.  The closest you get is $4.99 for the Encyclopedia app.
2. You can shut the iPad off at 10pm and turn it back on at 7am every day, without social services knocking at your door.
3. When the iPad is hungry, you can plug it in and leave it alone for two hours.
4. One word:  Diapers.
5. The iPad will never go on a first date or drive your car.  Ever.  Not even with iPhoneOS 4.0.
6. You’ll never hear the iPad asking where it came from.
7. The iPad will never walk in on you having sex.
8. Trying to swipe a baby will land you in jail.
9. Best game with a baby:  Peek-a-boo.  Which gets boring pretty fast.  Solitaire, on the other hand…
10. You can’t make menstruation jokes about babies.

No related posts.

Software has long since lost its glory-days status.  We’re not the go-to field anymore.  Geeks are no longer revered as gods amongst humanity for our ability to manipulate computers.  We get crappy jobs just like everyone else.

So, what is it that still motivates you to work as a software developer?

Is it your fat salary, great perks, and end-of-year bonuses?  Unless you’ve been working on Mars for the past two years, I think Computerworld would disagree with you.  We’ve been getting kicked in the nads just as hard as everyone else.  Between budget cutbacks, layoffs and reductions in benefits or increases in hours, clearly our paychecks are not our primary source of satisfaction.

If money was our primary motive, right now we’d be seeing a mass exodus from the tech sector.  So, if it’s not the money, then what is it that we hang on to when we get up each day?  Are we really working for those options?  That salary bonus?

Turns out, we’re kidding ourselves if we think that’s our real motive as developers.

The assumption: People perform better when given a tangible, and even substantial, reward for completing a task.  Think bonuses, stock options, and huge booze-driven parties.

The reality: In a narrow band of actual cases, this is true.  (And by narrow, I mean anything that isn’t a cognitive task, simple or complex, according to the research I quote below).  By and large, the reward-based incentive actually creates poorer performance in any group of workers for cognitive tasks, regardless of economic background or complexity of the task involved.  (Sorry, outsourcers…dangling the reward under your workers noses doesn’t help even when your home country is considerably poorer on average than Western economies.  Yet another surprising finding of their research.)

I’m not making this up, nor am I just drawing on anecdotal experience.  Watch this 18 minute video from TED and I’ll bet you’re convinced too:

Daniel Pink gave this lecture at the 2009 TED.  It’s mind-blowing if you’re stuck in the carrot-and-stick mentality.  And I’ll just bet, unless you work for Google, are self-employed, or extremely worldly, you probably are.

I’m not saying that to be mean or controversial.  I’m saying that because this mentality has pervasively spread to every business, industry and country on the planet over the past 100 years.  It’s not just software development, but we’re hardly immune from its effect.

While we’re not immune to the impact, we do have a lot going for us that gives us an advantage in stepping outside this mentality:

• Developers tend to be social oddballs and the normal conventions seem awkward to us. Social oddballs tend to question things.  We don’t like what everyone else likes because, well, we’re nerds and we don’t think like sales people.  Or accountants.  Or athletes.  We’re willing to try things others find weird because we’re weird too.
• Because we’re odd, we tend to be forward thinking and revolutionary in our approaches to workplace advancements. Think about the good aspects of the Dot Com era:  pets in the workplace, recreation rooms with pool tables and ping pong, better chairs and desks for people, free lunches.  Those innovations didn’t come out of Pepsi, Toyota, or Price Waterhouse Coopers, they came out of tech companies.  Every one.
• In doing so, our weird becomes the new normal. Witness the output of the Dot Com era:  Aside from the economic meltdown, how many companies now regularly practice some, if not all of those things we did back in the late 90s?  (Albeit with more restraint, thankfully)

With that in mind, let’s take Daniel’s idea of the results-oriented work environment (ROWE) forward and create something new for the 21st century.  It focuses on three important ideas, which developers already love and embrace:

1. Autonomy
2. Mastery
3. Purpose

Autonomy: What developer out there doesn’t like to be given the freedom to do their own thing, on their terms, with their preferred hours, using their tools, environment, IDE, language, operating system and favorite t-shirt?  Find me a single developer anywhere that doesn’t crave this kind of freedom and I’ll pay you $10.  Seriously.  Drop me a contact above.  I’m good for it.  Of course, you’ll search for the rest of your life and won’t be able to do it.

Mastery: Every developer on the planet wants to get better at what they do.  We crave new knowledge like some people quaff coffee after a hangover.  Fortunately, the side effects of getting better at development are far more benign than caffeine binging.

Purpose: Nothing is more tedious, horrific, or uninspiring to developers to work on projects that lack any real meaning in the world.  Or lack any real direction.  Or lack any substantial need from the company.  In fact, you can probably point to the brightest points of your career all stemming from those projects that had the deepest meaning to you personally.  Maybe the darkest points are those soul-sucking projects that you waded through because you were glad to have a job but desperately waited for things to improve so you could find a better job elsewhere.  Preferably where soul-vacuums didn’t exist.

Google gets it:  They already advocate the 20% time concept and (near-)complete workplace freedom.  Atlassian gets it:  They have the Fedex challenge where everyone in the company gets 24 hours to work on something they are interested in, with the caveat you have to deliver it at the end of 24 hours and you must present it to the company.  Think those don’t create passion for the company?  How about the Nine Things Developers Want More than Money?  These points all touch on the same three basic concepts:  autonomy, mastery, and purpose.

Does your company “get it”?  If the answer is NO,  what can you do right now to change your workplace to “get it”? And if that is too Sisyphean a task for you, how about starting your own company instead, that does “get it”?

That’s my challenge for you in 2010.  “Make software suck less in the 21st century”

Good luck.

No related posts.

I’ve mentioned a number of software laws in various posts, like Cargill’s Ninety Nine Rule, or Occam’s Razor.  And there are tons of laws that you probably already know, like Metcalfe’s Law or Moore’s Law.

I’ve found a very complete list of the laws regarding software development (I highly recommend reading that link. I’ll wait, go ahead).  But from that list, we seem to have developed a complete blind spot for five in particular.  Let’s look at these five and how our collective ignorance of them continues to impact software development today:

Law #1: Amdahl’s Law

Gene Amdahl first published this notion in a 1967 paper.  This law is about the mistaken notion that “All We Need Are More Parallel Processors and Our Software Will Run Faster”.

The Damning Evidence: Pop quiz:  have you bought a new machine in the past 4 years that was multi-core?  Were you a little disappointed when you checked the processor usage and found that not every one of those shiny, new cores was busy all the time, no matter which of your apps you ran?

We buy new hardware with the mistaken impression that our old programs will continue to run even faster than before because we expect our software to take advantage of all those friggin cores!  But software never runs as fast we expect it to on the multi-core hardware, because the parallel component of the program is often missing, underdeveloped, or poorly understood by the developer. Thus, our software continues to disappoint us on even on shiny, new multi-core hardware.

Exceptions: Some applications have been expressly written to be massively parallel and they continue to kick ass and take names on new multi-core hardware (e.g. rendering, scientific and encoding applications).  By and large, most applications simply don’t benefit from those extra cores because they weren’t written to do so.

Law #2:  The Law of False Alerts

First introduced by George Spafford in this article, the law states that the more the user is presented with false or erroneous alerts, the more they will ignore real alerts in the system.

The Damning Evidence: Windows Vista is the classic current example.  Every bloody operation in it required your permission from the user authentication module.  After while, you just madly clicked “Yeah, sure whatever…” for every warning that popped up.  This, of course, robs the operating system of any ability to protect you from a real threat because you’ve been annoyed by the feature in the first place.

Of course, people still design applications like this:

• “Are you sure you want to delete?”
• “No, really, are you REALLY sure you want to delete?”.
• “OK, look, I’ve asked already but just so I can’t be blamed for anything, are you SUPER-DUPER-ABSOLUTELY, 110% sure you want to delete?”

Stop the insanity.  If they click delete and they weren’t supposed to, how about offering an undo operation?  Too hard you say?  Then you’re not trying hard enough.  Don’t punish the users for bad design.

Law #3:  Jakob’s Law of Internet Experience

From Jakob Nielsen, web usability guru, who states that users only spend a small fraction of time on your site, compared to all other sites.  Therefore, your site experience should be similar to all other sites to minimize learning curve and maximize usability.

The Damning Evidence: Well, things like Firefox Personas aside, which distract your users from the actual content of the sites, we still can’t seem to come up with a consistent way to develop user interfaces on sites.  Thanks to Web 2.0, everyone is now trying to copy the success of sites like Facebook, Twitter, and other social networks to create wild, experimental web pages that are just plain awful to use.

Don’t get me wrong here:  I’m not saying different is bad, I’m saying that different is hard to get right.  Users (especially “Normals”) don’t like to be made to think how to use things.  But that doesn’t seem to stop us creating web pages with crazy stuff on them.

Exceptions: Sometimes, user interfaces are giant evolutionary steps that simply lie outside the normal boundaries we’ve come to expect and that’s acceptable.  The iPhone was a perfect example:  no one really had mastered the touch interface until Cupertino & Co came out with it and they didn’t exactly follow any of the old school rules.  But it was still a major success and now sets the standard for all smartphones.  However, most everyone else thinks they’re creating the exception when they’re just breaking the rules poorly.

Law #4:  The Pesticide Paradox

Attributed to Bruce Beiser, the law states that every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.

The Damning Evidence: Things like Test Driven Development and Unit Testing give us the false impression that we’ve quashed the major bugs in the system when all we’ve really done is quash the obvious bugs, leaving the more subtle, painful, and difficult ones behind.  Many of these types of bugs are related to concurrency or particular complex data conditions that are difficult to express as unit tests.

Before anyone rants about this comment section claiming I think TDD is bad, or unit testing is evil, please hear me correctly:  Unit testing and TDD leave a false sense of security that we’ve managed to create stable software. They are a starting point to more complete testing, but they are not the end.  The meaningful problems are often in integration with other systems and modules, that are often left out of testing plans because of time constraints, schedule pressures, laziness and sometimes plain arrogance.

Exceptions: Small, simpler systems rarely suffer from these issues because testing is much easier.  This is mostly a complex software problem, at a level of enterprise development, large applications (e.g. Microsoft Word), or operating systems.

Law #5:  Fisher’s Fundamental Theorem of Natural Selection

While this law stems from genetic research by R.A. Fisher, the application in software is somewhat obvious:  The more highly adapted an organism becomes, the less adaptable it is to any new change.

The Damning Evidence: We strive to create complex, interesting, and highly useful frameworks:  Hibernate, Struts, Flex, ExtJS, and jQuery to name a few.  But every version we release generates new requests by the users for missing features or enhancements.  Each change adds more complexity.  And the more complex the software, the lower the chance those changes can be easily accommodated in subsequent versions.

For example, Struts went through a major rewrite for version 2.0, which speaks volumes about the original version’s adaptability to change.  Spring did a major update for AOP that was a breaking change from 1.0.  ExtJS did the same for their 1.0 and 2.0 releases.

Exceptions:  Probably none–this seems to be the inherent nature of frameworks.  But if you know of something, please prove me wrong in the comment section.  I’d love to hear about some piece of software that didn’t follow this rule.

No related posts.

So far we’ve established a few things from the last post on Jedi Software Training:

• We already implicitly and perhaps accidentally practice the Apprentice/Master model in software development today.
• The route to Software Master is paved with ambiguity.  Most who achieve it can’t tell you how they got there.
• A Software Apprentice often flails around in a sea of information without much guidance unless they are insanely lucky in their first job.
• We need a systematic way to train apprentices and grow the craft in a reproducible manner.

Someone at this point might be saying, “Don’t we already have internships?  Isn’t that enough?”  Let’s distinguish an internship from an apprenticeship.

	Internship	Apprenticeship
Exploring career option?	Yes	No
Short time frame?	Yes	No
Project-based opportunity for career exposure?	Yes	No
Employer is looking for permanent hire?	No	Yes
Provides training in all aspects of field?	No	Yes

The important difference here is about goals and time frame. The internship is about checking things out and deciding interest, the apprenticeship is about getting serious and learning the craft from start to finish with the intent of employment.  So no, the internship doesn’t address the need in the same manner.

To create successful apprentices, we need several things:

• A road map to understand the most important aspects of the craft
• A master who already understands the map
• A place where the master practices and can freely train the apprentice

Who, what and where.  The how is being addressed here, and the why was already established in the last post.  Let’s focus on who now.

The Ideal Master

Taking a page from the book of Star Wars:  To train a Jedi Apprentice, we first must find a Master.  The master clearly must know software engineering inside and out.  They should be versed in every aspect of project development from soup to nuts:  analysis, requirements, design, development, testing, release and maintenance.  Ideally, they should have practiced for 15+ years and on a wide variety of projects–from small to large, from internal to highly public, and from successful to dismal failure.

A master should be fluent in many tools, languages and frameworks.  A good master is productive no matter what tool they use once they practice for a short time, given their vast prior experience on other tools.

A master must be able to communicate difficult concepts with simple and easy-to-understand analogies or other descriptions that even the newest apprentice can grasp.  True mastery of material means the ability to describe it in multiple ways.  The master should have infinite patience to describe things in various ways to give the apprentice many different ways to understand new concepts.

Most importantly, a master must be fluent in the business of software, since many software decisions are driven through the business, not the craft of software itself.  A master that cannot understand the balance of these two forces is no master at all.

The Path to Mastery

To turn an apprentice into a journeyman, we need a training program.  Electrical apprentices currently mix actual practice on the construction site with nighttime course work in electrical theory, such as the study of Ohm’s Law.  A similar program for software could be useful:

• How to use the tools of the trade (e.g. IDEs, makefiles, testing scripts, automated build & testing tools, source control, etc)
• Understanding the theoretical side of software engineering:  requirements analysis, functional decomposition, design practice, project scheduling, developer estimation, and so on.
• Putting these theories into practice using a small scale, but real, project

Some of you may be thinking that this sounds precisely like a university program for computer science (at least in part).  Most computer science curricula are geared toward creating academics who are skilled in the field of computer science.  The fact that software engineers might pop out on the other end of the coursework is an afterthought.  It’s the same distinction between physics majors and mechanical engineers:  science vs. applied knowledge to practical situations.  Clearly we want the practical here.

The Master’s Workshop

There seems to be no perfect environment in which to train the apprentice in a modern setting.  This, to me, is the crux of the problem.  There are several potential possibilities, but each fraught with peril.  Let’s examine each of them individually:

• Universities
• Open source projects
• Companies of various sizes

The University Setting

The first and most obvious is university setting.  But universities already have problems as a training ground for three major reasons:

• Most professors are professional academics without the benefit of real-world software engineering experience in a business setting.  This makes them less than ideal Masters, even though they may be quite accomplished in their own right.
• Their goal as academics is to create more academics, not to create masters in the field.  Generally, they prefer the theory to the practical aspects.
• The end result of a professor’s research is a published paper, not a finished software product.  Often time, prototypes are more than enough for them.  Prototypes are not good examples of production grade software.

Dude, what about Summer of Code?

An excellent question:  What about Open Source projects such as  Google’s Summer of Code or the Apache Project?  There’s an unlimited number of them, creating limitless opportunity for apprentices.  They sure seem like a great place to start, but:

• Most of these projects are working remotely, using distributed development techniques.  A master/apprentice relationship functions best when the master and apprentice are close in proximity to keep the apprentice from getting too far into the weeds.  SoC and other such setups do not offer that proximity and many apprentices are generally unprepared for that level of freedom, at least initially.
• An apprentice is trying to develop confidence in their skills.  These projects often assume some self-taught skill with the tools involved.  Newcomers are often ridiculed and derided for lack of understanding of even basic tools.  That doesn’t foster confidence in apprentices.
• Some are treated more like internships (e.g. Google Summer of Code).

Be A Company (Wo)man!

That leaves only one place left as fertile ground for training:  technology companies.  But which ones–large, medium, small, or startup?  Let’s make a case for each.

Large companies (more than a few thousand employees) seem to be a good choice:  well funded, plenty of good facilities, and larger numbers of job openings per year.  Yet I don’t think this setting would enable the craft of software to flourish and expand for several key reasons:

• Large companies tend to hire large development groups.  Inside these large groups, mediocre and completely untalented engineers can reside undisturbed for years because of the inertia of large organizations to get rid of them.  This unseemly presence of incompetence could give the apprentice the wrong impressions of development techniques if left unchecked.
• Large companies tend to foster painful bureaucracies that would stifle the creation or gestation of such apprenticeship programs.
• CEOs and technical managers of such companies are highly focused on budgets and value-add for every person in the company.  Apprentices are the very definition of unproductive learners, taking up substantial resources up front in order to become productive.  Managers would have to place personal reputations on the line in order to start and maintain such programs in the face of budgetary pressure.
• Large companies have very siloed groups, sometimes with company-specific processes, tools or standards, that would expose an apprentice to a very narrow view of development practices, making them less marketable to the broader economy as software engineers.

Medium size companies (say between a few hundred and a few thousand employees) would face some of the same pressures as large ones, but the right company with the foresight to create such a program might be successful.  It comes down to the courage of the management and the strength of will of the development team to make it happen in the face of current company policy.  However, in this current economic climate of 2009-2010, few companies would take such a chance, in my opinion.

I think the biggest bang for the buck will come from small companies (less than 200 employees, but more than a dozen) for several reasons:

• Small companies because of their size must attract the best talent in their development teams.  Small companies don’t have time or money to waste with unproductive people. But they are usually willing to develop good potential talent because of the value proposition.
  
• Small companies tend to be nimble in their thinking and can change their internal culture to support such a movement the easiest of the 3 described so far.
• A smaller company can use an apprentice more easily across groups and teams, making more efficient use of the apprentice’s time and energy, and exposing the apprentice to more of the business of software.

But it’s not all sunshine and roses either:

• Small companies fight tooth and nail for whatever money they get.  Budgets for developers are tough.  Add an apprentice to this and it gets even tougher.
• Many small company developers are often over tasked in their current roles, making it harder for them to act in the role of a master developer.  A part-time master is a difficult and potentially inconsistent kind of teacher.

Finally, we’re left with startups (less than 20 employees).  While they have some of the small company advantages such as agility and talent attraction, I believe that the chaotic environment of startups is best left to more experienced engineers:  journeymen or at the very least, previously trained apprentices.  Startups rarely have the time to spend training new people in techniques beyond those that implement the vision of the founders.  And what meager monetary resources exist in small companies, even fewer are available with startups.

In short, the apprenticeship idea requires tremendous courage on the part of the company that fosters it.  Small companies are used to taking big risks with substantial payouts in the future.  I believe small companies are the ideal place to grow this notion and allow it to root.

The Bottom Line

Creating an apprentice program would require jumping a number of substantial obstacles, not the least of which have been enumerated here.  But the company that creates such an environment would become a magnet for the latest talent in the industry because so few places offer the right opportunities to train new graduates into real software engineers through a systematic approach.

Related posts:

1. Jedi Software Training–Part 1

Seriously, has the Mozilla team run out of important things to work on in Firefox?

My Firefox browser updated to 3.6.2 today and I’m greeted with this page, asking me to try their new personas:

I’m not really a customize-my-browser-to-look-like-a-teenage-girls-Twitter-page kind of guy, but I thought I’d give it a shot and see what happened.

In a word:  horrific.

My browser’s link bar went from easily readable to complete obscured:

And not just with one, but pretty much ALL of their “recommended ones”.

WTF Mozilla?  Have we decided to throw out 30-odd years of user interface practices in favor of ponies, rainbows and unicorns? Really?

There are 30,000 MORE of these monstrosities to choose from!  Keep in mind that Mozilla would, of course, showcase the best and most interesting on their update page…But if these are the best, I’m frightened to dig any deeper.

This reminds me of the skinning snafu of the early 21st century where every damn audio application (WinAmp for example, but certainly not limited to them) had to come out with 368 cool skins to go with their app.

Not only did you have to learn an entirely new interface with the application’s custom look-and-feel, but you often had to relearn it for each damned skin you switched to.  The same was true of Linux usability.  Reminds me of a quote:

Whenever a programmer thinks, “Hey, skins, what a cool idea!”, their computer’s speakers should create some sort of c*ck-shaped soundwave and plunge it repeatedly through their skulls.

This is a usability nightmare**. No, wait, usability nightmare doesn’t even begin to cover it.  And now Mozilla wants to do that with my browser?  As if MySpace pages didn’t make the web awful enough…

For the love of all that is good and easy to read in the world, stop.  Just please stop.  Tell me where to send the money to make it stop.

My eyes are still bleeding.

UPDATE (part 1):  Looking to REMOVE the personas?  Do this:

Go to Tools -> Add-ons ->Themes Panel.  Click on Uninstall on the persona.  Then restart Firefox.

UPDATE (part 2):  Since I have the (un)fortunate Page 1 Google ranking in “firefox personas”, and the comments seem to fall into 3 categories in rough order:

1. Dude, you’re a grouch.
2. Dude, you’re an idiot AND a grouch.
3. Dude, I totally agree with you.

I realize I fell prey to a classic issue often happened in math class too…I skipped straight to the answer and failed to show my work.

** When I say “usability nightmare”, what I mean (hyperbole aside) is that personas violate some well-known principles of web usability all posited by Jakob Nielsen, the guru of web site usability.  He doesn’t just guess on these things, he actually researches them, observes behavior and reports results.

Which principles?  Well, I can probably dig up a dozen if I try hard, but without going too crazy, here’s a short list of the ones that FF personas pretty much violate right out of the box:

• Distract users on eyetracking for important content.
• Violating web-wide conventions
• Vaporous content and empty hype
• Custom GUI widgets
• Ignoring aesthetic and minimalist design

Usability matters.  And grouchiness aside, the more we infect these kind of eye-Twinkies (think:  eye-candy, but far less nutritious) on people, the less capable they are of actually using the web in the first place.

No related posts.

Software engineering is perhaps the youngest of all the engineering disciplines (and some would even argue, we don’t practice an engineering discipline at all).  But like all disciplines, an engineer must be trained in order to achieve a level of competence to practice their craft with any proficiency.

The Fresh Out Of College Problem

Software engineers run the gamut–amazing, mediocre, and step-away-from-that-IDE-before-you-hurt-someone levels.  Some are naturals and some probably will never be able to program their way out of a wet paper bag.  In terms of how we get trained, it’s all very informal in most cases.  Universities rarely focus on any software engineering courses as a serious part of a Computer Science program.  Indeed in my own Alma mater, Computer Science was part of the Engineering School but did not constitute an official engineering discipline.  The focus is entirely on data structures, programming, algorithms, and nuts-and-bolts sorts of topics.

Don’t get me wrong–those topics are incredibly important, but almost every college graduate I’ve ever worked with directly out of school (within the first 3 years of their graduation) generally has poor knowledge of how to run multi-engineer, moderate-size software projects.  I’m talking about simple things that are bread and butter for a successful complex project: source code control, project module management, requirements definition, functional design, and documentation.  This seems like a huge gap in training, considering that everyone knows these things are absolutely essential when they get out of school.  Employers are frustrated because graduates are unable to take the reins of a software project in a meaningful way without starting out in (very) junior positions that are rarely available and only modestly tolerated by most companies.  Graduates are frustrated at their lack of opportunities because they don’t have enough ‘real world’ experience to qualify them for real software development positions.

The question is, why are we training our engineers so sloppily today?

The Apprentice Model

Engineering training and Star Wars Jedi Knights have something in common:  they both start out as apprentices.  Let’s look at the history of this practice and see how it can apply today.

Back in the Middle Ages, if you wanted to practice a craft, such as blacksmithing, baking, masonry, or butchery, you needed to become an apprentice to a master craftsman.  That master had been practicing for years, had a demonstrated level of mastery, and often belonged to a guild of other masters who judged this master to be fit enough to earn that title of master in the first place.

Apprentices lived a hard life back then.  They were exploited as cheap labor for the master for a long period of time (5-7 years was typical).  The master provided tutelage in the craft, food, and lodging in exchange for their work.  After this period of time, the apprentice was shown the door and expected to fend for themselves as a newly minted journeyman.  Journeymen could either practice in solitude or work under a willing master (if available).  After some long period of time, the journeyman could try to produce a masterpiece of his or her craft in an attempt to demonstrate mastery, join a guild and attain the rank of master.

Some guilds were comically harsh in the training methods.  For example, up until 1791 in France an apprentice worked under a master for a long period of time through their journeyman rank.  If they failed to produce a masterpiece during their journeyman period, they were subsequently executed.  I, for one, am glad we don’t train people in this manner anymore.  The labor shortages this could create aside, the pressure to produce would be excruciating.  And in truth, not everyone is cut out to be a master.

Modern Apprentices

Apprenticeships are still used as a model for vocational training in many professions in a number of countries (in America, we still use this for plumbers, electricians, and carpenters among many other trades.  Across Europe, this is true as well), this training serves to bring new workers into the trade, give them baseline skills in a controlled environment under a specific instructor, and a route to advancement in the field.  But this formalization happened over years when best practices were easily formulated into codified documents and classroom formats, allowing the apprentice to reproduce a master’s work in a repetitive and formulaic manner.

If we characterize the differences between the three levels, you might see the following:

• An apprentice can take a specific set of instructions, created by journeymen or masters, and faithfully reproduce the steps to create a result of lesser quality than a master or journeyman.  An apprentice is unable to work without a concrete framework of rules to abide by while practicing the craft.  Apprentices generally need constant guidance and intervention from others in order to complete a project successfully.
• A journeyman is an apprentice who has completed their training period and achieved a baseline level of skill in the trade.  They are competent enough to work alone, but most often seek the continued education under a master to improve their skills.  Journeymen are comfortable in a wide variety of techniques as taught by the master, and many of these techniques are now second nature to the journeyman.  However, journeymen lack the ability to create new skills or perform the baseline skills in the effortless manner of a master.  Journeymen generally can do a skill, but have trouble expressing the exact reasons why one skill is preferable over another in a certain context outside of the rules given by his or her master.
• The master is the culmination of years of practice into “effortless skill”.  The master’s abilities require no conscious thought to manifest.  Masters have the ability to see patterns in projects and skills, arbitrarily combining things into new and unique ways of using them, often pioneering new techniques as a result of this artful mixing.  Masters understand the rules to the point where they can break them at will, knowing which limits are completely arbitrary.  A truly excellent master has the ability to teach their skills to apprentices in a way that creates excellent apprentices.  These masters are exceedingly rare.

The apprenticeship method is a time-tested, battle-worn method for bringing people of low-to-no skill into a field and training them to a level of success. Dozens of fields practice this, and almost all formal engineering disciplines have a similar model (engineer-in-training and P.E. (professional engineer) certification).  Academics have similar models to produce scholars, with undergrads promoted to graduate students, then post-doctorates and finally becoming associate professors, full professors, and professor emeritus.

My question is:  Why don’t we do this with software engineers?

Apprentice, Journeyman, Master – The Path Less Traveled

The path from apprentice to master must be reproducible and documented through time in a formal manner by previous masters.  Developers today are largely left to hack out their own destinies at random with high variability in the results.  If they inadvertently become masters, they attribute it to their own skills rather than the lucky circumstances they managed to find themselves in early on in their careers.

Transferring that mastery becomes incredibly difficult when a master’s path lacks any formal system to follow along the way.  Software masters are revered for their superhuman abilities to swoop in to save the day on a doomed project through superhuman coding effort via nights, weekends, major refactoring on a level that ultimately undermines the morale of the rest of the team.  This “cult of the software hero” approach ultimately shrouds the master’s path rather than illuminates it for others to follow.

We know this model is still at work in software today, albeit informally.  We see the various levels at work in our own teams and workplaces.  Here’s a simple OO experience hierarchy that you probably can relate to:

• An apprentice OO programmer will struggle with encapsulation and polymorphism, but can put together systems that have been well-specified.  They need highly detailed object designs to see the various interactions of a system come together with elegance.  Left without such guidance, they will inevitably create God Objects, Yo-yo Object Hierarchies, and other monstrosities.  Design patterns are something of a mystery to them.  Simple algorithms will require large amounts of effort to understand and encode into a language.  Their skills with the tools of the trade will be low to moderate (e.g. IDEs, source code control, automated build systems, bug tracking) and significant effort will be expended in using/learning them.  Design and architecture are generally beyond them.
• A journeyman OO developer will understand the value of data hiding and be able to participate in interface designs for obvious parts of the system.  They are comfortable with the tools of their trade and know many time-saving shortcuts, along with several flavors of tools.  Journeymen are typified by rigid adherence to systems, tools, frameworks or architectures because of their comfort level with them (“The Golden Hammer” effect).  They will recognize some design patterns and have regularly applied them, but may not know all cases when they are applicable, or apply new ones with ease.  Journeymen dabble in architecture with increasing skill and ease, but complex problems are often met with complex, obscure solutions when designed by journeymen.
• Master OO engineers can hear domain problems and suddenly see object models dancing in their heads before the description is finished.  Architecture is second nature and design patterns require little to no effort to apply or recall.  A master OO developer generally tends towards tool, language, and framework agnosticism because they understand their relative weaknesses and strengths, choosing only those tools that stay out of their way, make the most sense for the domain and allow for ease of creation.  A master’s trademark is the ability to create a simple and elegant solution to a complex problem.

So the $64,000 question becomes:  How do we create an apprenticeship model in software development that can succeed in cultivating better engineers? We’ll start to answer that in Part 2.

Related posts:

1. Jedi Software Training–Part 2

There seems to be a fundamental debate raging:  Is software more like mad science or plumbing?

This debate came to my attention via Mike Taylor’s article:  What Ever Happened to Programming? Mike’s argument is that programming is nothing more than plumbing today and it’s no longer fun.  He believes that there’s more fun to be had as a “mad scientist” developer, building everything from scratch with materials at hand.  So let’s look at the two major camps of this argument:

Software Developer as Mad Scientist

Reading the classics of software literature like Knuth, Brooks, et al, you get some notion of the programmer as hiding out in an arcane computer lab, late at night, pounding away at the keyboard as if to build some Frankenstein program.  Unlike Shelley’s creation, the outcome is far more benign and often even useful.  But the creation is always that:  pure construct from the mind of the programmer.  No assistance from the outside world aside from a few borrowed organs to create the Magnum Opus.

This image has historical truth in it–Knuth himself created TeX in a similar fashion.  Supposedly Woz and Jobs did the same with early models of Apple computers.  Every programming language we have at our disposal today clearly had some singular human force behind it:  Ruby, Haskell, Java, C, C++.  The list goes on ad infinitum.

These creations required intense and detailed knowledge of the hardware and operating system to create their monsters.  Whatever they required in their tasks, they often built from scratch by themselves.  They are the pioneers of our fields, the first wave of migrants on the digital frontier.

Without the Mad Scientists, we would be language-less, tool-less, and probably stuck with punch cards on ENIAC.

Software Developer as Plumber

I hear our job derogatorily compared to that of a plumber:  “We just put stuff together instead of build it”.  I don’t think that gives plumbing it’s due, nor does it really consider the rich history of the field.

Plumbing back in the late 1800s and early 20th century was a dicey business.  The entire practice was inconsistent, lacked any standard methods, and training was haphazard (for much more background, check out this article).

Appropriately, the National Association of PHCC (formerly the National Association of Master Plumbers), first met in committee in 1883 at the old Astor House, the hotel that provided the impetus to modern plumbing back in 1834. Many new plumbing inventions had appeared and too many plumbers were ill-prepared. Close on their heels would be the Mechanical Contractors Association of America, the American Society of Heating, Refrigerating and Air Conditioning Engineers and the American Society of Sanitary Engineering.

Wholesalers banded together, too, starting programs to prod manufacturers into standardizing such things as sink and basin outlets, faucet drilling, trap gauges, etc. The Central Supply Association, for example, was formed in 1894 and soon made contacts with the old Eastern Supply Association, the Plumbers Association of New England and the National Association of Master Plumbers. But it would take another 30 years to accomplish the standardization which everybody takes for granted today.

That means roughly the first 50 years of plumbing (1883-1925) was effectively like the Wild West:  Every man for himself and standards be damned.  The public often suffered as a result of this:

An outbreak of amoebic dysentery in Chicago during the 1933 World’s Fair was traced to faulty plumbing in two hotels. Tragic results were 98 deaths and 1,409 official cases. One year later, Major Joel Connolly, Chief Inspector of the Chicago Bureau of Sanitary Engineering, spoke these prophetic words:

“One of the lessons to be drawn from the amoebic dysentery outbreak … is that plumbing demands the very best, painstaking effort that thoroughly qualified, certified plumbers can give in every building, and especially where the systems are complicated and extensive, and where large numbers of people may be affected by contamination of water.” (emphasis mine)

Clearly standardization of materials, methods, and training gave plumbing a major shot in the arm for consistency, safety, reproducibility and public trust.  The plumber’s model started off as cowboy hacking of pipes in a haphazard way to a systematic method of standards, interoperability, training and licensing.  Along the way, there were glitches, problems, and issues.  Big surprise.  Sound familiar?

Our software legacy has taken us from raw register manipulation in assembly, through multi-generation languages (2GL, 3GL and god help us, 4GL), to huge amounts of frameworks, libraries and tools that give us unprecedented levels of productivity today that would be unheard of 10, 20 or even 30 years ago.  But the cost is that there is less of the low-level work to do, and more of the heavy-lifting at the business level.

We’ve complained, bitched, and moaned about spending too much time on things like low-level problems:  database connectivity, GUI frameworks, XML parsing.  And guess what?  People responded to those complaints by building libraries, tools and frameworks to make them happen. To give us what we always wanted:  the ability to focus on the business problems and not the lower-level constructs.  In essence, we’ve borrowed the plumber’s model.

You assemble the pipes, solder them together, solve local problems about how to route the sewer line around the funky wall shape, but you don’t get to set the pipe sizes, mold the elbows, or determine the ideal composition of solder for ease of melting.

You get to put things together for utility.  A large part of modern software development is nothing more than a utilitarian venture of “some assembly required”.

But even the first waves of migration to the American West by the military and trappers of the day had comparatively little impact to the settlers of the late 19th century.  And similarly, the mad scientists of software have a significant, but much smaller, impact in comparison to the plumbers of software.

So which are we?  Mad Scientists or Plumbers?

Both.  Neither.  It depends:  On who you work for.  On what you specialize in.  On where your interests lie.  There is a need for both:  mad scientists are the creators of new tools, frameworks, languages and OSes, plumbers are the integrators, the users, and the orchestrators.  Software requires both to survive.

This isn’t a question of what software is, but rather who you want to be.  But there are some facts that are hard to argue:

• There are more plumbers jobs than there are mad scientist jobs.
• The need for mad scientists seems to diminish over time, not because mad scientists are less important, but because more plumbers are needed once the mad scientists are done with their work.
• It’s very hard to be a good mad scientist OR a good plumber, but they are vastly different skill sets.

Be an mad scientist or a plumber, but don’t complain when you’re a plumber but you really wanted to stay a mad scientist.  The choice is, and always was, yours to make.

No related posts.