Facilitating Technical Discussions

One of the tech lead on my team excels at his job: he picks things super fast, is excellent at mentoring and helping others; he is a recognized leader, articulate and personable.

In other words, he pretty much has all the qualities you could wish for in a team leader; however, and this is not uncommon in our field, he's often too silent during technical discussions' meetings and doesn't drive them as efficiently as should be.

I'm a firm believer that it is the job of tech leads -and not the one of Dev Managers- to drive technical discussions, identify options, and make recommendations.

Here's a few reasons for that:
A. These gals/guys are closer to the issues than anyone else

B. It gives them opportunities to develop leadership skills

C. It gives them full ownership and accountability - which is probably the greatest motivation of all

That doesn't mean that as a Dev. Manager you should be removed from the technical issues- au contraire - but that's a separate discussion. Back to the facilitation issue.

My recommendation to my tech lead was to start applying 3 concrete steps during meetings, in order to more efficiently drive them:

1. Re-state what was said: a silent nod is not enough; you need to make a statement that starts with something like "Let me make sure I understand..." and summarize what was said. This allows you to maintain control of the meeting, and enables everyone to validate their own understanding - making sure everyone is on the same page.
2. Give everyone a turn to speak: Ask people who haven't expressed their opinion whether they have something to add. The most silent engineers are often the ones with the most thought-out ideas; so you need to make sure everyone participates. If you're in a geographically dispersed organization, with people dialing into your meetings, the issue becomes even more acute: being assertive on the phone while a group of people are meeting face-to-face requires a whole different level of confidence; so it's especially important to help these folks stepping-in the discussion and make it clear that you value their opinions.
3. Bring closure: acknowledge and summarize when a decision is reached on a given point. This helps making sure there is truly agreement, and allows the conversation to move on to the next topic.

These 3 facilitation tricks don't require you to be born outspoken and assertive, and they greatly help driving meetings in a more productive and efficient manner.

Thoughts on selecting a shared web host provider - a candid review

This is more tactical and technical than I'd like for this blog; but anyone looking to start-up a web application runs into this type of issue.
Impartial reviews on shared web hosts are hard to find - they're buried under a ton of commercial review sites which sell their souls to whoever gets them the fattest referral check.
So I'm sharing here my own experience at selecting a shared web hosting provider - hoping it will help someone at some point.

What do you need it for?

The first step in selecting a web host is to understand what are your requirements.
These requirements are typically in terms of technology stack (LAMP vs. .Net vs. Java vs. YouNameIt), disk space, bandwidth, computing power, and up-time/reliability.

The web application I was looking to deploy is based on fairly standard architecture:
Some PHP/MySQL, with the meat of the application being a fairly chunky Flex/Flash file (.5 MB in size) that spends most of its time querying the net to retrieve analyze and aggregate RSS feeds and other REST-based data.
Because of cross-domain security restrictions, most of that traffic needs to be proxied through my own server(s) before being able to reach the browser/Flash application.

My requirements bottom line is this:
a. I need a ton of bandwidth, but
b. I don't need much disk space or computing power: most of the action is happening in the browser/Flash client.
c. I need a standard LAMP stack (with PHP5.x and MySQL5.x)

Scalability and load-balancing
While the odds of overwhelming success are definitely not in my favor; I'm a firm believer that you need to prepare for what you're wishing for.

So, if overwhelming success means overwhelming bandwidth consumption; I'd better think about the problem in advance.
The solution I've devised consists in doing some poor man's load balancing:
a. have the html/php pages and database hosted on a very fast and reliable server
b. have the flash files as well as associated RSS proxy traffic distributed onto dumber/commodity servers

More contenders than I'd like to deal with

So far, in the past 3 months, I've tried a total of 6 shared hosting providers.
My primary sources for trying them has been dumb google searches and netcraft.

Netcraft is the best source I've found for starting your selection of shared web hosts: they report on actual quality of service and seem to be impartial.
However, you ultimately need to go for a trial in order to see what you're getting.

Here's the run-down - in the order I signed-up with them:

. LaughingSquid [http://laughingsquid.net/]
Plans are anywhere between $8 and $14 per month; but bandwidth is constrained to 60GB/month.
This is a great thing if you're not bandwidth hungry - it means no-one else is hogging on the wire while you're being civilized.

They're based in San Francisco, cater to artists and bloggers, and clearly state that they don't want your business if you're going to suck down their servers.
The fact that they enforce a strict policy while not being dirt cheap turns out to work pretty well if your requirements are met: tech support is very good, and server response times are always great.
They mention that their servers are actually hosted by rackspace.com; which led me to host #6.

. aplus.net [http://www.aplus.net/] - offers a $7.46/month LAMP plan where you get 2000GB of bandwidth.
I ran into a technical issue; and their tech support wasn't able or willing to address.

So I moved on.

Cancellation and refund was without any hassle.

. 1and1 [http://www.1and1.com] - is another big provider with attractive packages - 2500GB of monthly bandwidth for under $10 a month.

The tools are great; but instructions are sparse.

Tech Support is available 24/7, but they seem to be using cheap IP phone lines; so on one instance I just couldn't understand what the Tech Rep was saying on the other end of the line - I ended-up sorting out the problem myself.

Server response times were often 10x of those experienced with LaughingSquid. So I canceled.

The cancellation process was a bit funky for me -requiring 3 emails and the same number of phone calls- but in the end I did get my refund.

. PowWeb [http://www.powweb.com/] - provides unlimited bandwidth for under $8/month
The tools they offer are very good; and their tech support was top of the line.
Just like 1and1; server response times were often 10x of those experienced with LaughingSquid; and just like 1and1, the cancellation process was a bit funky but I did get my refund.
I did get the overall feeling that they were truly concerned about delivering great value.

. Yahoo small business hosting [http://smallbusiness.yahoo.com/webhosting/]- also offers unlimited bandwidth for under $10/month
Server response times are pretty good.
I ran into 2 limitations:
1. You can only send 250 emails per day from any email account (that's their way of preventing spam). The trouble is, my registration process includes an email confirmation, and in my dreams, I get more than 250 registrants per day (hasn't happened just yet)
2. They offer PHP4 support, but no PHP5 - and all my script were developed on PHP5.
This is where I currently host the Flash file and proxy the RSS traffic.

. Mosso.com [http://www.mosso.com/] - Mosso is Rackspace' shared hosting division. Much like Amazon Web Services [aws.amazon.com] it's a pay-what-you-use type of service.
It's not cheap: $100/month gets you 500GB of bandwidth (and more diskpace and computing cycles than I need).
Except for one instance where the tech guy was unresponsive, Tech Support has been top of the line.
And, I do get the best server response times.

This is were my html/php pages and MySQL database are currently hosted.

And the winners are...
It again really depends on what you need:
. If you're OK with limited features, but want dirt-cheap bandwith, go with Yahoo SMB Hosting
. If you need great service, great response times, and don't need a ton of bandwidth or diskspace; LaughingSquid is your choice
. Lastly, if you need a lot of bandwidth, great features, good tech support and great response times, then you'll need to pay for it, and mosso will be a good place to start.

You don't get a second chance

I mentioned in my last post the need to fail early, fast, and often in software development.
Guy Kawasaki recently posted a great piece on bootstrapping where he explains the point with his sharp style:
" Ship, then test. Perfect is the enemy of good enough. When your product or service is good enough, get it out, because cash flows when you start shipping. Besides, unwanted features, not perfection, come with more time. By shipping, you'll also learn what your customers truly want you to fix. It's definitely a trade-off your reputation versus cash flow so you can't ship pure crap. But you can't wait for perfection either. (Nota bene: life-science companies should ignore this recommendation.) "

Whether expressed as "fail early, fast, and often" or as "ship, then test", the concept is critical to successful software development and innovation at large.
However, as you get closer to impacting customers/consumers, the concept becomes more and more subtle - if not dangerous:
When it comes to impacting the external world, most of the time, you don't get a second chance.

There are at least 3 issues you're facing:
1. Trust:
You cannot test/release/fail early and often with your users, customers, or prospects, and come back to them, and keep on trying stuff on them - only with less issues than the previous time.
You'd be toast.
While the fail early and often concept means that you should hit something; it doesn't mean that there isn't a trust issue.
Kawasaki expresses it as a trade-off between reputation and cash-flow; but unless you have exceptional reputation in the marketplace or with a strong customer fan base, that's actually not an account you can draw funds from - when you're a small player, reputation is more like a Swedish match: you can use it only once.

2. Attention Span:
Trust is one thing at risk; interest and attention is another.
You cannot expect people to wait patiently, with unaltered attention for your next product release.
This is true for enterprise customers expecting a set of feature that is critical to them.
This is even more the case in the consumer space - unless your name is Steve Jobs, Sergei Brin or Larry Page; people will rarely give attention to your product for more than a few seconds; let alone wait for a new better/stronger/faster solution.

3. Freshness of your story:
Closely related to the attention span of your audience is the interest of the media at large (press or blogs): When launching a wide-reaching product or consumer application, and in order to have a successful launch, you will need to have a good product and a great story - and you can use your story only once.
It becomes stale very shortly after it's released.


Back to the concept of failing early, fast, and often -
The fact that you're faced with risks on trust, attention and interest, certainly doesn't mean that you can take a big-bang approach, isolate yourself from the World for a couple of years, and -deus ex machina- put out a solution people will rave about.

There is probably no silver bullet as to when and how to test your solution in the market; but there are a number of common-sense principles:
A key concept is that you need to keep on hitting something - getting feedback about what you're building.
At times, the temptation will be great to say "screw-it, let's just launch".
The question will then be whether you've been obsessive-compulsive or whether you're becoming impatient and sloppy.
There is definitely tradeoff and some balancing act:
. Are you confronting something new that is getting you closer to address the needs of your market?
. Are you reaching diminishing returns in what you do, or how you do it?
. Are you in tunnel vision, absorbed on the current issues and having lost sight of the big picture?

The trick is probably to identify if you're past the point where you will get a second chance - either because the audience you're going to impact is carefully selected (a self-selected fan base or a restricted set of users); and/or because you know you're providing enough of a compelling reason for users to come back.

The Technology Acceptance Model can probably be of help here:
You will get a second chance when there is enough usefulness that your users need to have your solution; and enough usability that they can actually realize benefits out of it.

Fail early, fast, and often

Note to self and others: this is one of these post that starts with a thought (the central one here being that when it comes to user adoption, you rarely get a second chance), and where I'm circling around, ending-up pulling a whole thread of rumblings - not necessarily in the most coherent and articulate manner. In the spirit of failing early fast, and often; this is the first part of the rumbling.

A central concept to agile programing practices is that no matter how hard you try to plan and design a software solution, you're going to be off the mark.
And the distance by which you'll miss your target largely depends on how far away you're aiming at it; i.e. how much in advance, how far and how deep you're designing your solution.

This is true for your technical architecture: if designed exclusively along a waterfall model; it will most certainly either be under-sized or over-designed - and will rarely perform well, or be cost-effective to maintain.

The problem is even more acute when it comes to specifying functionality and designing end-user experience: if you start coding some functionality according to some written specs frozen 6 months ago, and then throw them over the wall to your users; you can pretty much be guaranteed that your solution will suck - either from a usability standpoint, or from a usefulness standpoint; and most likely from both.

The largely accepted concept that addresses this issue is that we should test early and often, also known as "check-in early and often", "release early and often"; and which I like most expressed as "fail early, fast, and often" - the emphasis being on the idea that it's ok to screw up, and that the earlier you do that in your development cycle, the better off you are.

This is closely related to Risk Management: issues are bound to happen, so the earlier you hit the greatest risk areas, the less likely you are to fail.

This is also very much in line with Gall's Law which states that "a complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: a complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system."

The concept works great and is actually critical to the development of successful solutions - up to a certain point:
There are a number of situations where you can't go out if you don't have enough or don't have the right thing - i.e. situations when you don't get a second chance.
I'll try to expand on this in my next post.

What makes a good technical design?

What makes a good technical design?
This is one of these questions I get by way of search engines.

The first answer would be that it depends on who you ask:

In a discussion thread on "What makes a good technical design?", people list a number of great attributes (though often being unclear as to whether these traits are related to good technical design documents or good technical designs):
. "That it does everything it needs to and nothing it doesn't. :)" - KC
. "It's a good design if the thing you design is used in the future in a way that the original developer never anticipated, and it works correctly with no modification." - cbmc64
[Agreed on the first; disagree on the second]

In a similar thread, The Server Side proposes a list of top 10 elements of good software design:
10. Considers the Sophistication of the Team that Will Implement It
9. Uniformly Distributes Responsibility and Intelligence
8. Is Expressed in a Precise Design Language
7. Selects Appropriate Implementation Mechanisms
6. Is Robustly Documented
5. Eliminates Duplication
4. Is Internally Consistent and Unsurprising
3. Exhibits Maximum Cohesion and Minimum Coupling
2. Is as Simple as Current and Foreseeable Constraints will Allow
1. Provides the Necessary Functionality
[1, 2, 3 and 10 are particularly important in my view]

Last but not least, Object Oriented Programming guru Rob C. Martin gives 11 principles:

5 principles of class design:
. The Single Responsibility Principle
. The Open Closed Principle
. The Liskov Substitution Principle
. The Dependency Inversion Principle
. The Interface Segregation Principle

3 principles of package cohesion:
. The Reuse Release Equivalence Principle
. The Common Closure Principle
. The Common Reuse Principle

3 principles of package coupling:
. The Acyclic Dependencies Principle
. The Stable Dependencies Principle
. The Stable Abstractions Principle

These are all useful, deep and insightful - and can probably also make a great impression during dinner in some circles.

The second-level answer as to what makes a good technical design is that there is really no universal answer. It depends.

It should however not vary not based on who you ask, but rather based on what you're trying to accomplish:
What is the business problem that you're trying to address?
What are the resources and timeframe that you have?
What are the constraints that you're facing?

A good technical design is a design that requires the least amount of effort and money while meeting your business requirements; and in particular as they translate into your development speed and turn-around time, maintenance cost, deployment complexity, scalability, or security needs.
There is no right or wrong.

In order to build great designs, you need to be conversant with technical design principles to the point where you feel comfortable not only using them, but also ignoring them.

Technical Design Essentials

As far as the current state of software technology, pretty much everything has been written on technical design - but then again, it looks like Search Engines, with their current flaws, want us to write about the topic - and don't seem to lead to anything much relevant.

So here is a list of 3 books that are essential to better designing software and creating solid architectures.
These are classics that are worth much more than what they're being sold for.
With one exception, the list weights heavily towards pure Object Oriented design and concepts. This doesn't mean that highly-structured OO designs are invariably better than the light-structured ones - it just means that in order to make a technical choice, you need to truly understand what your options are.
Onto the list:

. UML Distilled - by Martin Fowler. The focus of this book is as much on UML (which is a critical tool) as on the development and technical design process (where Fowler truly shines). Fowler is able to articulate in a very concise manner the process, challenges, and best practices of creating great OO designs.
Here are two great quotes that convey well Fowler's tone and philosophy:
. "No user is going to help you for pretty pictures; what a user wants is software that executes"
. "A hard deadline works well in concentrating minds"

. Design Patterns - by Gamma, Helm, Johnson, and Vlissides; aka the Gang of Four
This is an absolute classic.
The book contains 23 software design patterns.
There again, the actual patterns are interesting, but what is truly unique and invaluable is the description of the thought process underneath them.
The authors also lay critical principles of good OO designs - including these two:
. "Program to an interface, not an implementation"
. "Favor object composition over class inheritance"

. Building Scalable Web Sites - by Cal Henderson
Cal Henderson played a key role in building the Flickr architecture; so he knows a thing or two about building scalable web applications.
There is a couple of things that make this book extremely compelling:
1. It reflects on a much less formal approach to build solid architectures - the technology stack is LAMP; with PHP intrinsically putting less focus on the high-structure OO principles that Fowler or the Gang of Four are outlining
2. It touches on all the elements required to build high-traffic web applications - this includes general architecture options; authentication and security requirements; localization and internationalization; hosting; partitioning, distribution and clustering; performance and scalability.

As a bonus - here are 3 websites that are worth looking at:
. OO Tips, by Yonat Sharon, contains a great collection of notes on Object Oriented design. The content is sometimes too deeply rooted into Object Oriented dogma - but there again, knowledge is power.
. Mapping objects to relational databases, by Scott Ambler, is a great paper on the key design issue of Object-relational mapping. The article balances depth and clarity, and perfectly articulate how design choices are a question of trade-offs.
.The Anti-pattern catalog, on Ward Cunningham's web site, has an extensive list of design traps, that you can get familiar with - in order to be better able to identify and avoid them.

From Flip-Flop to Kung Fu via the Red Army - an alternative maturity model

A number of years ago, the Capability Maturity Model was introduced as a way to measure the maturity of development organizations.
In a nutshell, the concept is that the least mature organizations are relying on undocumented processes, practices and knowledge; and that their success -if any- is the sole result of individuals' heroic prowess.
Grossly oversimplified, the CMM states that organizations mature by gradually developing repeatable processes; then measuring their efficiency, and finally optimizing the way they operate.
Somewhat of a Maslow pyramid of needs applied to organizations - from survival to ongoing self-improvement.
You can read here and there for a more detailed -and clearer- explanation.

The CMM is definitely a good way to "read" a development organization - and address its weaknesses.

Looking back at the development shops I've worked at, and the ones I've worked with; it seems that the maturity of these organizations could also be categorized along an alternative framework, based on 3 very distinct types:

. The Flip-Flop organization:
Flip-Flop organizations are on the lower end of the scale, and are a too common type.
Their essence and commonality lies in the fact that they are lacking focus and a sense of purpose.
Some employ very formal and repeatable processes, others are very Agile; yet they all achieve the same thing: not much, if anything.
A majority of the projects they engage in never come to fruition: they're either shut down in flight, experience a slow death, or do come to completion but with an outcome that has no significant impact.
The management style of these organizations could pretty much be anything - except focused, deliberate, and resolved.
The pattern is self-reinforcing as people quickly understand that there is no point in pushing their limits, that they won't be held accountable for slipping a date, and that they might as well focus on Resume Driven Development.

The common cure for fixing a Flip-Flop organization is to come down with a hammer, set objectives that you can't mess with, and transform it into...

. The Red Army development team
[Apologies to veterans who fought in Stalingrad and freed-up death camps; it's the Management I have in mind...]
The central concept in this type of organization is that it's a crime not to meet a goal or to slip a date - and you might get shot (or fired) for doing so.
Projects need to have aggressive completion dates before starting, and once initiated, there is no turning back unless an Act of God strikes.
There again, processes may or may not be formal/agile/repeatable - although processes do tend to develop very quickly when people need to set boundaries around their responsibility and accountability.
The overall throughput is much greater than in the Flip-Flop organization; the organization does get from point A to point B; but there are 2 major issues:
1. Efficiency is on the same order as driving a military-grade truck to go to the post office (a practice that was popular until recently). Mileage is poor and you need deep pockets in order to operate this way.
2. In software development, the point B defined when you were at point A, rarely has any relevance when you're about reach the said point B. Development is a process of discovery that requires constant adjustments, and markets are typically volatile and demand an ongoing validation of the strategy and objectives.

In order to fix a Red Army organization you need the organization and its leaders to develop a deep concern for both economy and impact - i.e. a focus on getting maximum results without expanding energy unnecessarily.

You then get...

. The Kung Fu organization:
If you're not a fan of Stephen Chow's movies or haven't read Sun Tsu the concept might sound obscure.
What lies within it is a constant focus on quickly assessing your current position, leveraging your strengths and the opportunities presented by the environment, and creating maximum impact with minimal effort.
This requires the organization to be deliberate and disciplined, yet flexible and creative.
In this type of organizations, you'll typically find:
- very lightweight processes
- very seniors engineers that have the respect of the entire organization, but yet are not unquestioned
- a leadership team with a constant focus on performance and results

In a way, Peter Drucker, might have been much closer to Kung Fu than one would have thought.

Persistence

I'm talking about perseverance here -i.e. "to persist in a state, enterprise, or undertaking in spite of counter-influences, opposition, or discouragement" [m-w definition] - not about data storage...

I recently came across 2 great pieces that are uplifting for any individual or team facing obstacles, and in particular the lack of understanding or empathy from the external world.

The first one is coming from Guy Kawasaki: in a guest post on Sun's blog titled "Five most important lessons I've learned as an entrepreneur" - item 4 reads:
"
Ignore schmexperts. Schmexperts are the totally bad combination of schmucks who are experts--or experts who are schmucks. When you first launch a product or service, they'll tell you it isn't necessary, can't really work, or faces too much competition. If you succeed, then they'll say they knew you would succeed. In other words, they don't know jack shiitake. If you believe, try it. If you don't believe, listen to the schmexperts and stay on the porch."

The other piece is proof to Kawasaki's point - it's a letter from the NY MOMA that was sent to Andy Warhol in 1956.
It reads:
"
Dear Mr. Warhol:

Last week our Committee on the Museum Collections held its first meeting of the fall season and had a chance to study your drawing entitled Shoe which you so generously offered as a gift to the Museum.

I regret that I must report to you that the Committee decided, after careful consideration, that they ought not to accept it for our Collection.

Let me explain that because of our severely limited gallery and storage space we must turn down many gifts offered, since we feel it is not fair to accept as a gift a work which may be shown only infrequently.

Nevertheless, the Committee has asked me to pass on to you their thanks for your generous expression of interest in our Collection.

Sincerely,
Alfred H. Barr, Jr.
Director of the Museum Collections"

The letter ends with a P.S.:
"The drawing may be picked up from the Museum at your convenience."

The 5 Easy Steps to Creating a Bullshit Management Methodology

I recently went through another 3 day management training class - and it was just excruciating; a real punishment for being a manager.
One of my lead, who was also attending the class, had me promise not to use this stuff at the office.

The management field is plagued by a slew of witch doctors that are selling management books and trainings, in the hopes of making millions, while attempting to impersonate true management theorists - such as Drucker, Humphrey, or De Marco.

There are 2 questions that keep on hitting me when I go through these types of trainings -I've stopped going through the books a while ago, but I don't always get, or choose, the option to skip the trainings-:
. What are the components of BS management methodologies?
. Why people eat the stuff?

I only have a partial answer to the second question and it probably has to do with why there are so many variations of Fast Food chains in the World.

I may have a more complete answer as to what is the fabric -if you will- of these management sub-methodologies.

So here are 5 steps to creating a Bullshit Management Methodology:

1. Pick a good, wide-reaching, and difficult problem - e.g.
. quality sucks in most products;
. communicating with people is difficult;
. change is hard;
. we rarely get to retire before age 30;
you name it...

2. Identify common-sense concepts and ideas that, if applied, do address the problem - BUT introduce a thick layer of theories, concepts, and acronyms on top of it:
Don't just say that Quality must be an overall process that needs to be constantly measured and improved; or that when discussing difficult topics with people, you should assess objectives, values, background, and emotions.
Instead, introduce concepts with unique names and acronyms: "method 3", "level 4", SSX, "black belts", whatever...

This step is absolutely critical and has 2 major benefits:
1. No one wants to buy a book that just says common-sense stuff. It's boring.
2. You put your audience on the edge - they need to memorize stuff, make an effort to follow you.
The idea here is that there should be just enough complexity for the audience to follow you, and yet prevent them from analyzing and questioning the virtues of what you're saying.
In other words, they should have just enough to chew on and digest.
And if it makes people hopeful and upbeat at the prospect of solving a pain or addressing a need - they'll ask for more.


3. Ignore other theories and authored work
You should avoid referring to other authors; especially if they're original, critical, thought-provoking, and articulate - they are your competition.
One caveat - it's OK, and actually recommended, to quote people who've given their names to a theory but haven't published much other than specialized research papers. So Maslow is always a great addition.
In any case, you should never list your sources and references in your books / training materials - unless your publisher forces you to mention other published authors.

4. Be assertive as if you had seen the Light and found the Truth
Do not invite self-criticism or doubt.
Push it as far as you can without ever showing an ounce of doubt.
It's OK to call your work a 'philosophy', but you should instead define a dogma.
The bulk of your audience isn't expecting to read Kant or Hobbes when picking your book. They're looking for a ready-made solution that can give them a comforting faith; not for some uncertain philosophical journey raising more questions than they have answers.

5. Be universal
The Management Methodology you're advocating should have the ambition to apply to all aspects of the audience's life.
The 7 Epsilons Quality Management theory is relevant to your kitchen, and Situated Power Communication should work on your spouse, kids, and dog.
And if it doesn't work; it means the reader didn't get it - and she should know it.

This raises the stakes -the Methodology is much more powerful and critical than the audience initially imagined- and simplifies the reader's problem: just buy this book, follow these steps, and all of your problems will be fixed. And you should sign-up for my seminars if they're not.


And when you're done creating your very own Power Management Methodology for the Enterprise - if you've ended up corrupting yourself, feel lost, burned out, or need to get back to the basics - you can always reflect on what Peter Drucker's was writing in the early '70s:
"An employer has no business with a man's personality. Employment is a specific contract calling for a specific performance... Any attempt to go beyond that is usurpation. It is immoral as well as an illegal intrusion of privacy. It is abuse of power. An employee owes no "loyalty," he owes no "love" and no "attitudes"--he owes performance and nothing else."

Measuring programming languages adoption trends

I talked a while ago about perceived market trends in the programming language world: Java's stronghold seems to remain untouched, PHP is all the craze despite its intrinsic lack of structure, and the Ruby fad seems to be losing momentum - I actually thought it was pretty much dead until I started discussing with friends and was told Ruby is in fact alive and kicking.

How can this easily be validated?
Beyond formal polling -which is quite involved, expensive, and not necessarily accurate-, there are a number of ways to measure language adoption trends: analysis of book sales volumes, result counts in general-purpose search engines as well as blog-centered ones, or number of related job postings.

One of the key to understand strengths and limitations of each of these approaches is to put them in the context of the Diffusion of Innovation theory. In a nutshell, this theory states that 1. technologies spread by gradually addressing the needs of 4 types of users (innovators, early adopters, early majority, and late majority); and that 2. they do so in a non-linear manner, by typically expanding slowly until reaching the early majority, at which point they start expanding at a much faster rate (if and when the technology is successful).

Here is a rundown of these trend analysis methods, followed by actual results:

. Book sales volumes is an interesting source - as introduced by Tim O'Reilly.
One limitation is that the numbers could reflect interest more than actual usage - buying a book on Ruby programming doesn't necessarily mean that I'm actually using it.
From a Diffusion of Innovation standpoint, books would also tend to be purchased not so much by innovators or early adopters (books are typically not published when these categories start adopting a technology) and more by the early and late majorities - so sales numbers would probably better represent technologies that are already in the mainstream as opposed to the emerging ones.
From a practical standpoint, a major issue with this approach is that the numbers are not published on a regular basis.

. Volume returned by search engines is probably the simplest and most accessible method - I've tried to use it in my low-key analysis of web services adoption trends, and TIOBE publishes indexes established based on this.
One objection with TIOBE's index is that -as with any index- numbers can be taken at face value, with the audience ignoring their actual source and its intrinsic limitations:
1. You need to pick your search keywords wisely; and you'll always get a level of noise in the results. TIOBE uses " programming" as their keywords (as in "java programming" or "PHP programming") which might not be the best choice for say Actionscript (the more relevant keyword here being "Flex"). I use the language name as the search keyword - and I'm not sure how many of my results are related to jewelry when searching on "Ruby".
2. Web count probably tends to favor languages on which there is a large established volume of data -i.e. those who've reached the majority-, or those who are creating an unusual amount of buzz (and thus generating a large number of references).

. Technorati, or other similar blog-centered tools, is a useful variation on search engines - these probably tend to favor languages generating a lot of debate and discussions.
The volume is likely to be generated by innovators and early adopters, and is therefore expected to tend to over-amplify early technologies rather than those in the mainstream.

. Job postings are another place to look at.
In fact, the job market could be one of the best indicators of the penetration of a technology: this is where the data is not affected by the volume of interest, publicity, or discussions; but in fact represents where organizations actually stand in terms of executing on a technology.
One major caveat, is that organizations could more heavily recruit skills they don't have rather than the one they already have on staff – i.e. technologies entering the mainstream rather than established ones.
The location of the job offering could also say a lot in terms of the diffusion of the technology: in tech-focused areas (e.g. the SF bay area), the numbers probably tend to favor technologies that are at the stage where they access the early adopters and early majority; in more traditional areas, they would reflect more or less on the early vs. late majorities, depending on how conservative the area is from a tech standpoint (e.g. NYC vs. Washington DC).

To validate my hunch, here are the results of the above 3 approaches (excluding book sales because of lack of current data) for Java, PHP, Ruby, and Flex:
I'm using craigslist for job postings, and also normalizing results to 100% to help the comparison across sources.

. Google:
Java 487 millions; php 12,230m; ruby 130m; flex 95m
[4%, 94%, 1%, 0.7%]

. Yahoo
Java 908 millions; php 2590m; ruby 267m; flex 195m
[23%, 65%, 7%, 5%]

. Technorati:
Java 228k; php 141k; ruby 51k; flex 60k
[47%, 29%, 11%, 13%]

. Craigslist – SF Bay Area (all jobs)
Java 1125; php 773; ruby 257; flex 250
[47%, 32%, 11%, 10%]

. Craigslist - NYC (all jobs)
Java 406; php 606; ruby 143; flex 218
[30%, 44%, 10%, 16%]

. Craigslist - DC (all jobs)
Java 228; php 267; ruby 47; flex 98
[36%, 42%, 7%, 15%]

What does this say? Probably that I need to do some more digging...

- The google results seem to be out of control.
I initially thought this was because of all the pages ending in .php, but the results are not different by much if searching on text only. So I went onto Yahoo, which is different, but still weighted towards php beyond what I was expecting.
Plausible explanations:
1. Php is indeed predominantly used
2. Because of its nature and audience, php is much more "documented" on the web than java is

- The Technorati weighting is indeed quite different from the regular search engines and instead closely aligned with the SF bay area job posting distribution - but I'm not quite sure what to make of it except that I'm not really seeing the amplification of emerging technologies that I was initially expecting.

- In terms of job postings, the balance is leaning towards java in the Bay Area and PHP elsewhere.
Is it really about being conservative, or does it have to do with the nature of the projects in different locations?
Note also that there are less job postings in NYC and DC combined than in the Bay Area - so the question of what constitutes the "majority" might not be that straightforward.

-As far as Ruby is concerned; there is not much that the numbers reveal - is it still in it's infancy, or slowly winding down?
The only way to tell would be to look at the trending over time - which is a little bit more involved.

I might take another checkpoint in 6 months to see where we are.

Guidelines for good technical design documents

I recently realized that Yahoo! search gives me a great ranking on these keywords - but then doesn't link to anything relevant.
That's a great question nonetheless... So here are some guidelines.

What makes a good technical design doc varies greatly based on the environment, the level of uncertainty and changes expected, the level of formalism expected or required, the audience and stakeholders, and how the document fits into your SDLC.

There are some guidelines that are good to keep in mind in any case:

• Only be as formal as you need to be - sometimes you don't need a document; discussions and brainstorming on a whiteboard might be sufficient
• Start on the right foot by giving your document a relevant and clear title, a file name consistent with its content, a version history table, and a table of content
• Don't write a Victorian novel - use bullet points and tables, and keep sentences short
• When documenting object oriented designs, make sure readers can quickly find and grasp each class' name, responsibility, and relationships to other classes (see CRC for more details)
• A picture is indeed worth a thousand words - block diagrams, class diagrams, ER diagrams, and interaction diagrams are great tools that are easy to create. Keep them simple (their intent is to give the big picture), and make it easy on yourself (if you're good at MS Visio great; otherwise paper and scanner might be sufficient)
• Make it as short and condensed as can be - the objective is not to write as much as you can; the objective is to describe the technical design as precisely and concisely as possible - most of us are not great writers, and most of us hate reading verbose tech docs
• Do yourself a favor, read the Elements of Style by Strunk and White - it's short and to the point, and will help make the writing process less painful for you

Happy writing.

Fixing performance reviews

Performance reviews can be a great tool - both for managers and contributors - but most often, their intent is missed and their essence diluted in corporate formalism.

Here is a rundown on some of the issues, and some thoughts on how to fix them.

In most organizations performance reviews are comprised of:
- a written self-review, where employees can reflect on their own performance
- a written manager's review, where managers should structure and formalize their evaluation
- a review meeting, where managers can formally deliver their feedback, and discuss with employees

I believe these 3 components are essential, as they allow proper reflection and communication - on both ends.

To spice things up, performance reviews are often driving, and followed by, salary adjustments.
This sounds quite reasonable as you do want to compensate people based on their performance; but this is also putting a lot of weight on the exercise - most employees get less open to feedback when they think that the perception on how they performed on such or such project is solely guiding their paychecks
[People should instead be compensated based on the overall value they provide to an organization in relation to market rates, as opposed to strictly on short-term performance]

The crux of the problem often starts with the manager's written part of the performance review. These written reviews are either too un-structured -consisting in a loosely defined "Manager's Review" section which will eventually be followed by "Employee's Comments"; or conversely, too formalized - I've had to deal with "systems" where managers needed to rate their staff based on over 100+ questions after which a score would be given.
These type of written performance evaluations result in review meetings with similar attributes - either a set of random recollections and thoughts, or a discussion around a metric system that employees try to game in order to get the highest possible score; hoping that their pay raise will follow.

To be truly effective, to both the reviewer and the receiver, performance reviews need to focus of performance, results and growth - and just that.

In order to accomplish this, I structure both self-reviews as well as reviews around 4 questions:
- What were your major accomplishments over the review period?
- What are the strengths you need to build on?
- What are the areas you need to develop?
- In which ways can we help you become more productive?

While the relevance of the first question is obvious; the remaining questions are meant to help both the manager and the employee understand what they're good at (performance and results are obtained by building on strength, not weaknesses); where they need to improve in order to get greater leverage on their skills; and how can they make a greater contribution.

The last key principle to better performance reviews is that the feedback provided during the review should be no surprise to the reviewee.
People can only get better at their game if they receive constant feedback on their performance throughout the year.

Modeling development capacity in offshore teams

When discussing roadmap options -estimates, plans and development capacity- with Executives, I'm often challenged by their assumption that x staff-weeks of effort here (in the San Francisco bay area) is equivalent to x-staff weeks of effort for our offshore team (located in India).
The challenge for me has been as much to understand what variables affect offshore development productivity, as to communicate it to upper management - in a short and clear manner.

Before going into further details, I should say that
- Our overall development team is medium-sized (20-30 developers)
- Our offshore team represents less than half of that size
- Members of the offshore team are generally very solid and competent engineers; but
. They're on average 5 years more junior that the SF folks
. They have 6 to 18 months tenure with us, when the majority of the SF folks have been working with the product for over 5 years

When talking about differences in productivity, I'm therefore NOT referring to a potential difference due to one of the team being intrinsically less competent or talented than the other; but differences related to experience, background, and distance.

When I have to explain it in less than 10 seconds I usually say that because of the overhead created by the distance as well as the difference in seniority and tenure, we're typically able to get 90+% productivity (compared to the SF team) on 1/3rd of our projects.
For the remaining projects, the overhead is typically so important that it doesn't make any sense to engage on the project at all if it has to be done offshore.

So far, I've identified 3 major factors that affect offshore productivity:

1. Technical competence - meaning abilities, skills, and experience. In my situation, we're mostly affected by the difference in experience. In other cases, you might be affected as well by engineering abilities, meaning having the offshore team not being "as good" as your local engineering team.
The difference in technical competence can be modeled as a percentage of productivity of your local team.

2. Distance - no matter what you do, you will get some overhead related to distance: communication is more challenging when engineers cannot just walk to the next cube to discuss an issue or validate a solution.
The best way I've found to mitigate this is
a. To have the offshore team work on their own set of projects – i.e. minimize the amount of cross-continent communication by allowing the offshore team to work on the same issues and resolve them internally
b. To have an engineer on the local team being dedicated to be the point of contact for the offshore team. This helps understanding where they stand, what their progress and challenges are, and channel and follow-up on communications between the 2 teams
c. To have proper communication solutions in place such as conference calling and/or video conferencing, instant messaging, servers for file transfers, and the like
d. To have regular checkpoint meetings (at least twice a week) with the remote team

While you can very significantly reduce the distance overhead, some will remain.
This can also be modeled as a percentage of productivity of your local team.

3. Nature of work. This is where it gets tricky. There are a number of components that make the nature of the job more or less appropriate to do offshore.
This includes factors such as
. Amount of background info required to do the job. - e.g. Rewriting some SQL code for performance would require close to no new context for a qualified developer to do the job. Conversely, writing a new piece of functionality at the core of your business domain, and using some proprietary framework, would require some significant ramp-up effort as well as ongoing knowledge transfer.
. Amount of iterations required or expected - how much design reviews and other cross-team validations are going to be required
. Size of the job (i.e. small vs. large projects) - this might greatly affect the above factors; e.g. a significant training/ramp-up effort might be more appropriate on a large project

Ultimately, the nature of the work might require both a fixed-cost increment of effort to start the job, as well as an ongoing increment of effort, that could also be modeled as a percentage of your baseline productivity.

To sum it up; if we had to model the above, it would give something like

Offshore Effort = (Baseline Effort x Skill Percentage Adjustment x Communication Percent Adjustment x Nature of Work Percent Adjustment) + Nature of Work Fixed Cost Adjustment

That's probably a little over-the-top for managing day-to-day R&D with an offshore team; but it helps understanding the challenges.
This is one of these cases where modeling is more interesting than the model.

The most interesting part of this is that the productivity of offshore development efforts can vary widely based on these –and potentially other- factors.

What this also says is that offshore development initiatives are bound to be unsuccessful if the distribution and execution of work is not properly thought out.

Management Styles - Beyond X and Y

I was mentioning in my last post 2 widespread -and related- management theories: MBO on one hand, and Theory X / Theory Y on the other.

Both theories are quite general and high level.
They're both deeply rooted in assumptions about what motivate people.
Theory X, which is authoritarian and directive, assumes people on the job are primarily motivated by money, fear, and/or the need for protection.
On the other hand, MBO as well as Theory Y, are centered on fostering a collaborative environment based on trust, commitment and reciprocity where the focus is on end-goals and results.
MBO/Theory Y assume that people on the job are primarily motivated by self-development and the search for personal achievement.

A number of additional management theories exist.
Some of them are bringing fresh perspectives both in terms of styles and methodology, while at the same time providing different spins on underlying motivational assumptions.
These ultimately give us more keys, tools, and options to do better jobs and bring the best out of our teams.
There are 2 of these I was familiar with -Joel Spolky's as well as Ken Blanchard's-.
I also took a look at Wikipedia's entry on Management Styles.
Here is a rundown.

Wikipedia's entry lists 4 styles:
- autocratic; where the leader makes all the decisions and drives them down the organization - the closest thing to Theory X
- paternalistic; where the leader also makes all the decision, but is more concerned about the well-being of employees - a softer variation on Theory X
- democratic; where decisions are made collectively - probably the closest thing to Theory Y; for as long as you understand that corporations' primary goal is not the well being of their employees, and as such are not meant to be democratic organizations
- and laissez-faire; where decisions are not made in a concerted effort - and which is probably Theory Y applied with no drive; or simply the absence of Management.
This nomenclature is interesting because it relates management styles to a form of political government.
Beyond the fact that, in my view, governments and corporations should have radically different goals [the protection and well being of their citizen on one hand vs. making a profit on the other], I can see a couple of limitations here:
The first one is that most of us put a judgment value on each of these forms of government -from unacceptable to desirable- with democracy being the only mature or reasonable system for the vast majority of us.
The second limitation is that these form of management -and the way they're described- seem to be primarily a by-product of the personality traits of the person in charge. The type of work at hand as well as the attributes of the team being managed -size, skill level, or seniority- seem irrelevant to the choice or application of these styles; when in fact they should be key drivers.

Joel Spolsky has his own -very original- way of describing 3 distinct Management styles.
. Econ 101 Management - this style is assuming that behaviors and performance can be driven by the prospect of financial rewards; and therefore assumes that the main motivation for people to work is money.
It uses compensation mechanisms as the main leverage to get things done; and ignores other motivating factors.
As pointed out by Spolsky, this style is both expensive and inefficient.

. Command and Control - this style consists in applying general infantry military methods in the workplace.
It is relying on imposed authority; and is based on the assumption that the subjects are ultimately motivated by fear, or a deep sense of duty.
If you want to play Drill Sergeant on a team of developers, you'll notice that it's quite inefficient and you'll probably soon end up on your own.
[While on the topic; I should mention that in a number of elite defense organizations, where members are highly trained and skilled, the management style can actually get quite sophisticated - see the book from David H. Friedman, Corps Business]

Spolsky' preferred management strategy is what he calls the Identity Management Method - it consists in encouraging people to identify with the organization, and implicitly assumes they will be primarily motivated by the need for Group Belonging.
Team building exercises as well as treating people with genuine care are critical as this strategy heavily relies on reciprocity to foster performance and loyalty.

The identity management method is a great approach that is virtuous and efficient when applied to development teams and other knowledge workers.
The only limitation that could be objected is that it is somewhat uni-dimensional - it doesn't tell me if and how I should vary my management style between junior and senior developers, or how to deal with an upbeat team after an IPO vs. a depressed and disgruntled team going through multiple rounds of layoffs.

To be most efficient, management styles should not only be applied at the overall level of a team, but should also vary based on general circumstances and individual situations.

I've found the most interesting bit on this from management seminars' Ken Blanchard
The method, called one-on-one leadership, consists in identifying where each individual stands in terms of
1. Competence - i.e. knowledge, skills, experience and capabilities; and
2. Motivation - i.e. determination, confidence, and commitment
Blanchard posits that management style needs to be adjusted considerably based on these 2 variables.
More specifically, they need to be adjusted in terms of the amount of direction as well as support being given:
Adjusting direction consists in providing different levels of granularity when stating objectives and giving instructions. A "low competence" individual typically needs a high degree of direction, with a clear and detailed description of what and how things need to be done. Conversely, a superstar developer would only need measurable goals with a description of the desired outcome and underlying intent -providing specific instructions, such as mingling into her code and telling her how to use such or such language feature, would instead be counter-productive.

Management Support consists in the degree of encouragement and validation that is given to an individual.
There again, this should vary significantly based on the level of motivation of the individual: an ecstatic new-comer might need to be paced, a developer going through the challenges of learning and growing needs a lot of encouragement and feedback; and our superstar developer might not need anything but a soft reinforcement of how much she is valued and appreciated.

To take it further, good management style should not only factor levels of competence and motivation, but also what specifically motivates each individual what they like, do best, and aspire to do better or grow into.

Management Styles - MBO, X, and Y

[Ed Note:
1. Apologies to my regular readers for not having posted in so long - I got sidetracked... should be back to a 2 to 4 weeks schedule
2. The email subscription has been fixed - you can now use the feedburner email widget on the right to get notified whenever a new post is published]

Shortly after being promoted a manager, someone asked me which management style I was using.
I did find the question a little odd for, at that time, there didn't seem to be many choices in "style" given the field and the environment - technical, volatile, and with most people hating the idea of being "managed".
There are in fact quite a few approaches to managing technical teams; and understanding them is a good step in becoming a better manager.
Here is a tentative rundown:

One of the most known style of management is Management By Objective (MBO).
Popularized by Peter Drucker in the late '50s, its key concept consists in getting buy-in from employees on defined objectives and specific tasks, and asking them when they'll be able to complete their assigned tasks.
Commitment is here essential, as the method relies on:
1. the assumption that the contributor knows better about the task at hand than their managers
2. Once a commitment is made, the contributor will make every effort to deliver on her promise -see the concept of cognitive dissonance for a deeper explanation-

MBO was somewhat of a revolutionary concept at the time, for it was in stark contrast with the generally accepted idea that managers should get results by either knowing more about the job than the people they managed or by having the ability to coerce subordinates into doing things -Management By Intimidation if you will-.
These two approaches are completely irrelevant in the tech field where, if your team members are indeed solid contributors, they'll know more about their jobs than you do; and if you try to "intimidate" them they'll eventually raise their finger -not the one you want- and find a job elsewhere.

Another spin on MBO is Theory X vs. Theory Y, established in the '60s by Douglas Mc Gregor.
Applying theory X consists in managing people by putting emphasis on controlling -meaning not trusting and instead directing and measuring.
Theory Y on the other hand relies on empowering employees and giving them proper level of ownership to get their job done - which is very close with Drucker's MBO.

The key to both theories, and when they would be relevant and effective, lies in what motivates employees in their lives and on the job.
More specifically, theories X and Y are deeply rooted in the Maslow Pyramid of needs:
Theory X can only be appropriate for jobs that only satisfy the lower levels of the pyramid -i.e. "hygiene" needs / jobs that pay the rent.
Theory Y is conversely only relevant in environments where the higher levels of the pyramid (need for group belonging and "self-actualization") can be satisfied:
The need for group belonging makes it compelling for a contributor to deliver on her commitment to her peers, and not let her team down.
The need for self-actualization drives people to learn new technologies, apply new skills, and try to improve their performance.

In my next post, I'll go over a couple more theories on Management styles.

Lumberjacks, hackers, scientists, and engineers

I was rambling in my previous post about 2 types of developers you might think you want on your team but you really don't.
Conversely, there are a number of profiles that are great assets to any team.

Developer's types is a larger question that has been addressed in a number of ways:
The Myers-Briggs Type Indicator (MBTI) is probably the most accepted way to categorize people and interpret behaviors. This is a great tool to better "read" people, understand what they might enjoy or dislike, and where they might excel.

In the Psychology of Computer Programming, Gerald Weinberg describes a number of personality traits required for being a great programmer:
- ability to tolerate stressful situations
- adaptability to rapid change
- neatness
- humility
- assertiveness
- sense of humor
These are all good attributes to look for when recruiting, or to develop for oneself.

In Managing Technical People Watts Humphrey makes a couple of distinctions among developer types:
- locals vs. cosmopolitans (grossly oversimplified: commitment to employer vs. technology or personal development)
- scientists vs. engineers (I'll talk about these in a bit)

The distinction between scientists and engineers is a great one.
Looking at the folks I have worked with, I see 2 additional profiles that are great assets to a team: lumberjacks and hackers.

Here is a shot at describing these 4 profiles:

Lumberjacks enjoy getting things done, and doing it well.
Lumberjacks are not necessarily great at learning, trouble-shooting, designing, or problem-solving but they're great at executing - often on a fairly established path.
There are a lot of people who do enjoy marking things done, so to be clear, if being done is the name of the game, we're only interested in those who mark things done when they're truly done and done properly.

Scientists enjoy learning, discovering, and inventing.
They enjoy finding order and simplicity from chaos and complexity; or in Humphrey's words "Scientists strive for knowledge and personal meaning. They see themselves as discoverers searching for hidden order and simplicity"
Scientists can appreciate the aesthetics of a solution - often more than the practical application of it.

Hackers enjoy getting problems fixed.
Aesthetics of a solution is typically secondary to end result, and any mean is valid to reach the end.
Hackers are typically great at learning and problem-solving but do not put as much emphasis on structure and meaning as scientists do – which can result in messy solutions if some discipline is not enforced.

Engineers enjoy building solutions and "seek to create their own monuments" (Humphrey).
The emphasis here is on structure and outcome, as opposed to the method or approach (as might be the case for scientists).
There is not necessarily a craving for learning new things, excelling at trouble-shouting, or for elaborating sophisticated solutions, but these qualities are important to make a great engineer.

So which one is best?
In a very small team (3 developers or less) versatility is key, and "engineers" tend to make things more efficient and predictable.
In larger teams, the mix and balance of these types is probably the greatest asset to an organization.
In fact, the challenge for recruiting managers is to make sure that not everyone is formatted along the same mold.
Diversity is what makes an ecosystem strive.

I have a java certification and I'm not afraid to use it

Every now and then, my team and I embark into a large-scale recruiting effort in order to bring in a few good developers.
The effort is only "large scale" in the context of the size of the current team, the deliverables we're working on, and the fact that the market is hot right now for tech folks - which means it's tough when you're on the recruiting side.
At any rate, I believe we ended-up interviewing over 100 candidates.
Our requirements are simple:
1. can code
2. can solve problems
3. not a jerk

I've outlined in a previous post some guidelines for recruiting developers.

Folks who just don't have the right skill-set or experience are typically easy to identify.

There are however 2 types of candidates that can be particularly deceiving and -I hate to say it- you do not want on your team.

I have a java certification and I'm not afraid to use it
In retrospect, the thing that stroked me most on this recruiting round was the number of candidates who knew programming syntax and libraries by heart, and quickly jumped into coding solutions; only to produce a massive amount of code that didn't solve anything.
The embodiment of maintenance nightmare.
They're the "I have a java certification and I'm not afraid to use it" class.
As the name implies, these candidates typically have one -or even a few- certifications.
Certifications were great back in the 90s. They did give you an edge, and the folks who were carrying them were typically in the top percentile.
Nowadays, certifications are the easiest way for anyone who has some good memory capabilities to get into the programming field.
The problem is that programming is about solving problems.
Not pissing code.

Chatterboxes
The second type of candidate you need to be watching for is the "chatterbox".
Chatterboxes are very articulate, and often very educated.
They can talk in great details about what they worked on; and it sounds really great.
Complex problems; brilliant solutions.
If your interview process is properly setup - you then move into the programming / problem-solving part of the interview, and then nothing much happens.
You may be puzzled at the end of your interview round - the candidate made a great first impression.
Don't be.
When there is a doubt, there is no doubt.

The evolution of Web Services - Darwinism

There are a couple of largely accepted theories that model or predict technology lifecycle and adoption patterns:
- The Diffusion of Innovations theory offers a model for how a given technology gets accepted and spreads through markets. Its central point is that technologies spread by gradually addressing the needs of 4 types of users: innovators, early adopters, the early majority, and the late majority (a fifth category, the laggards, might just never get it)

- The Technology Acceptance Model (TAM) offers some prediction to End User adoption. The key concept here is that individual users adopt a given technology based on its perceived usefulness and its perceived ease of use.

To my knowledge, there isn't an established theory or framework that models evolution trends of Technologies.
When looking at the history and evolution of web services, we seem to be in front of species that are spreading, adapting, and diverging much like finches in the Galapagos.

The immediate thought that then comes to mind is whether Darwin's Theory of Evolution has some or any relevance to Technology.

The theory of evolution defines three basic mechanisms of evolutionary change:
. Natural Selection is a process by which traits that are more useful in a given environment become more common over time (because they give better chances of survival), while traits that are harmful become rarer
. Gene Flow is the exchange of genes within and between populations, which translates in traits being transferred between populations and species.
. Genetic Drift is a purely random shift of the frequency of traits within a population - traits become more or less common in a population because of the long-term statistical effect of the random distribution of genes in each generation

How could these mechanisms apply to technology?

- Natural Selection is probably the mechanism most relevant to technology trending.
The fitter a technology is to the needs of its market, the more likely it is to stick around, and potentially supersede other technologies
This is why PCs are more likely to be found today than mainframes, why java is more often used than Fortran, and why soap-based web services have replaced xml-rpc.

- Gene Flow is also common in the tech field (although we'd probably want to call it something else).
Features and concepts are constantly exchanged between complementary or competing technologies.
That's how C# got a memory garbage collection mechanism similar to the one in java, and how row-level locking made it in MS SQL Server after years of Oracle claiming it as a key differentiator.
Gene flow is also at the root of hybridization, where traits of different species end-up being combined. This is what might be truly going on right now with REST - which is applying concepts of simpler web protocols, most notably HTTP and RSS, onto Web Services.

- Genetic Drift seems at first least relevant to the tech field, but might in fact be the most interesting bit.
The core concept in genetic drift is that the random distribution of genes in each generation can have a long-term effect on the frequency of traits in a population (because of the statistical law of large numbers, genetic drift is less likely to occur in large population than in smaller ones).
What, if anything, could have a similar impact in the evolution of technologies? What type of mechanisms, if any, can have an effect on the evolution and adoption of a technology, without being connected to its intrinsic fit or value?
Obviously there are a lot more forces that dictate the success or demise of technologies than just their core virtues.
A strategic alliance with IBM propelled MS-DOS into market dominance; technology companies like Oracle spend millions trying to influence the market; and there is a whole ecosystem of media, analysts, and venture capitalists who strive on generating buzz (PointCast or Twitter come to mind).

Who knows - if LISP had been able to be more hip, we might all be using more parenthesis today.

The evolution of Web Services - Measuring Trends

In my last post, I've tried to summarize a history of web services; the most recent developments being around RPC-style vs. Document style SOAP-based web services, as well as SOAP-based web services vs. RESTful web services.
I'd like here to try quantifying how each of these trends is progressing - the larger question being about finding efficient ways to measure the evolution and adoption of technologies.

Measuring trends
There are a number of heavy-duty tools for measuring technology adoption; the most obvious one is probably to poll organizations that might adopt them.
This is quite involved, can be expensive, and is not necessarily reliable.
Other -more or less reliable- tools include assembling expert panels, gathering feedback from your clients, discussing with analysts, reviewing book sales, or trusting your own gutfeel.

Web publication volume and search trends
Google is another tool that might be effective - not to find information, but to measure adoption levels and trends.
The main advantage is that it's simple and cheap to use.
The first step consists in identifying keywords relevant to the trends you're looking at and want to compare - in the case of web services, that might be "xml rpc", "soap web services", and "rest web services".
Once keywords are identified, a simple query on Google tells us how many documents are found - the number is approximate and is actually more or less reflecting the number of indexed documents that were deemed appropriate, but it should give a good order of magnitude of the volume of information out there.
Separately, Google Trends tells us the history of search volume on these keywords, i.e. how many searches were executed on these keywords over time.

Web Services publication volume:
3.7 million documents are retrieved for "xml rpc", 28 million for "soap web services", and 133 million for "rest web services".
If we assume that these terms are equally reflective of the amount of publication on the Web for their associated technologies (and that's a bold assumption); what this means is that rest is leading soap - at least in terms of publications and discussions on the web- and that xml rpc is marginal compared to REST or SOAP based web services.

Search frequency:
Here is the compared search frequency graph returned on the 3 sets of keywords:



It's important to remember here as well that we're making another major assumption in terms of accuracy of these results - which is challenged by the major blips we're seeing on the graph.

What this seems to say is that there is still interest in xml rpc, that soap web services are still going strong, but that there isn't enough search history and/or volume on REST web services to show any data.

Innovators, early adopters, and mainstream users:
The combined results seem to indicate that xml rpc is still being used but is on the decline, and that SOAP based web services are alive and well.
The more interesting bit is that there is a huge volume of content out there on REST web services but few actual people looking for that information - especially in comparison to the ratio of content vs. search volume on soap based web services.

Other than challenging the use of Google for measuring trends altogether - one interpretation is that REST web services are indeed not mainstream at this point, but they are generating a lot of buzz.
Innovators and early adopters are the most content-prolific type of contributors on the Net, and by virtue of being a hot topic, a lot of content is generated and discussions are occurring over REST on the Web.
Separately, because the technology is not fully embraced by the Masses, little search volume is generated compared to soap-based web services.

This is in line with the theory of diffusion of innovation, which I'll try to summarize in my next post along with other theories of adoption, before taking a shot at looking at the relevance of the Theory of Evolution applied to Technology.

The evolution of Web Services

There are a lot of changes going on right now in the area of Web Services technologies.
The transition from RPC to document style, and the recent buzz around REST, raise the question as to what the technology is evolving into, why, and along which patterns - if any.

I'll try here to do a recap on the history of web services.
In my next posts I'll try to take a look at the trends and review them against existing technology diffusion, acceptance, and evolution models.

A short history of web services
Distributed computing and interoperability is nothing new and the origins of web services should probably include efforts by DARPA to interconnect distributed computers.
In the enterprise world, flat file exchange via shared directories was a popular technique for exchanging information between different systems.
In the banking world, SWIFT -a private financial messaging network established in the '70s, with its own protocol and message format- is still a very much established standard.

Serializing the object model - CORBA, RMI, and DCOM
In the late '90s, a number of technologies allowing remote procedure calls (RPC) from one system to another became popular:
. CORBA allowed distributed software components to collaborate with a language and platform-neutral remote procedure call specification.
. RMI was essentially offering something similar, except it was entirely java based. As such it was somewhat easier to implement (at least for java programmers) but was also restricted to interoperability between java programs.
. On the Microsoft side, DCOM allowed native Windows programs to interact.

All of these competing standards had at least a couple major limitations:
1. They were restricted in terms of platforms they could interoperate with
2. It was a major challenge to have them work securely over Internet firewalls.

XML-RPC
The momentum java was gaining at that time, and its main value proposition to run anywhere, seemed to have prompted Microsoft to come up with a cross-platform interoperability standard, which could potentially help them get out of the corner they were slowly being pushed into.
In 1998, XML-RPC was advocated by Microsoft as a truly open technology leveraging the web infrastructure.
XML-RPC is essentially a remote procedure call protocol which uses XML format to encode its calls, and HTTP as a transport mechanism.
XML-RPC is a very simple protocol, defining only a handful of data types and commands - and this simplicity made it quite popular across major platforms.

SOAP
The XML-RPC standard quickly evolved into the more elaborate SOAP specification.
SOAP is a natural extension of XML-RPC that provides a greater level of structure and support for data types and operations semantic.

During the same timeframe, the need arose to provide more automation around message parsing, code generation, and discovery of web services over the network.
WSDL became the standard for defining web services interfaces; and UDDI became the standard for registering and finding web services on the Web.
There is today a considerable amount of tooling available for generating web services interfaces (i.e. WSDL) based on existing classes and interfaces, or for generating client libraries based on a WSDL.

Together, SOAP, WSDL, and UDDI form what is commonly referred as Web Services.

From RPC-style to Document-style
As Web Services usage gained momentum, a gradual transition occurred from RPC-style to Document-style.
In the RPC pattern, a client sends a request to a server, and the server immediately sends a response back to the client.
The RPC model is similar to existing programming models, and RPC-style Web Services are often implemented by simply generating services interfaces that map directly to existing language-specific function or method calls.
A major issue with this approach is the stability of the object model that these interfaces are based on: as classes’ attributes and operations change throughout their lifecycle, corresponding web service interfaces need to be re-generated accordingly, resulting in brittle and complex interfaces. A better approach is to provide a layer of abstraction by first defining the interface - a.k.a. Façade.
Another issue is that the RPC-style inherently requires synchronous processing – The RPC-style mandates the client end-point to be expecting the remote procedure call to return from the server-side.

Document-style Web Services offer a more radical alternative by putting the emphasis on publishing a schema, and exchanging messages, rather than on defining a rigid set of operations.
The Document-style approach is also known as the messaging model, or the SOA style (although the later term is overly abused, as can be seen on its Wikipedia definition page)

Unlike RPC Web services, loose coupling is more likely, because the focus is on the document "contract" that the WSDL provides, rather than on the underlying implementation details of operations.
As such Document-style services can be more agnostic to how, when, and by whom they are going to be consumed.
[A good technical article on Document-style web services can be found here]

Even when properly designed, SOAP-based web services often end up being complex to the point where directly coding against them would be a daunting task; and it is quite common to see enterprise web services packages heavily rely on the client-side libraries they’re delivered with - essentially reducing web services to some kind of middleware messaging protocol.

REST
In reaction to this complexity, RESTful web services are now becoming quite popular -at least in discussion forums-.
REST is a concept that was presented in 2000 and lays out a number of principles. The concept is still quite new when applied to web services, and a number of divergent definitions of what constitute RESTful web services are available.
The general focus is on interacting with stateful resources, rather than messages or operations.
Grossly oversimplified, RESTful web services heavily rely on standard operations already defined by the http protocol: GET, POST, but also PUT and DELETE. These operations allow to interact with stateful resources uniquely identified on the network, and to specify whether a given resource should be created, updated, or deleted.

The prototypical example of a RESTful type of "web service" is RSS, which consists of an http GET operation that returns an XML formatted representation of the state of resources exposed by the server.

A good programmatical comparison of XML-RPC, SOAP, and REST can be found here.

While there is a great level of antagonism in forums between proponents of REST vs. SOAP (the REST folks are typically more vocal), it's not clear that both technologies conflict: SOAP is well suited for supporting complex business operations, while REST is most appropriate for implementing simpler operations on simpler schema. In other words, REST seems to be well suited for broadly distributed, consumer-focused, web services (e.g. that of amazon.com or Google) while SOAP seems to be more suited for the enterprise world, where the challenges are around exposing complex legacy services, with requirements on security and transactionality.

In my next post, I'll try to look at some sources and data points for further analyzing the adoption trends.