Re: [tlug] programmer competency matrix

[Curt Sampson <cjs@example.com>]

Joe,

Thanks for your great comments. Many of them are things I already
know about and agree with, others I now agree with after further
thought.

You're right that the matrix has a lot of faults. 

I'd forgotten about the silliness of the API memorization thing (I'm in
API docs all the time, myself), and so I need some phrasing to fix that.
Any suggestions on phrasing?

The "higher levels include the lower" thing works for a fair number,
though not all, of the rows, especially if you take it in the general
sense, rather than relying on the specifics. Take the data structures
row as an example. For others, the more common cases, at least, are
covered; how many people learn FP before they learn OO, or more
precisely, how many functional programmers are not familiar with OO? We
could fix the others by working out some sort of branching system, but I
don't think that this is worthwhile, for reasons I'll describe below.

For all the problems it has, I think that the matrix is far above the
level of a checklist of acronyms. Surely you aren't going to argue that
"no RDBMs experience" vs. "uses MySQL once in a while" vs. "understands
the relational algebra" is no more useful than "RDBMS: yes or no".

Moving beyond all of that, you're right that programming tests are a
better measure. Are they the *best* measure? No: for example, working
with someone for three months is better. So why do we bother with
anything else? Cost.

Having someone work with us costs a lot of money. We obviously don't
want to offer a three month contract to everybody who applies. Even
the relatively short pair-programming sessions are quite expensive,
both for us and the candidate. Two non-billable days for me will easily
cost me 150,000 yen or more in lost revenue (yes, Starling is currently
turning down billable work), and the candidate as well loses two days
he could be spending doing other things. It makes sense to have a
graduated process that tries to weed out the obviously unsuitable
candidates before we hit that stage. So long as we run the process in
the appropriate way so as not to produce too many false negatives, we're
only saving everybody time and money.

If you look at where the matrix is in our process, you'll notice that we
ask for it with the resume. We consider resumes to be moderately useful
to disqualify candidates, and basically useless as a positive indicator.
The matrix is in the same category. However, it's quick to fill out,
will probably cover some interesting ground that the resume doesn't, and
serves one further purpose.

This is not something we normally talk about, but in trade for such
a good response from you, I'll tell you our little secret about the
matrix. One thing that it gives us is the ability to get some sense
of how a candidate rates himself, which gives direct insight into how
clueful he is.

The Dunning-Kruger effect[1] explains that people who are less competent
tend to overrate themselves, and very competent people tend to underrate
themselves. By comparing experience on a resume and the answers to the
matrix, I can tell if someone's rating of himself seems completely out
of whack. Given an interview, I can start filling in my own mental
copy of the chart and comparing the answers. This, as with the resume
itself, is highly inaccurate as any kind of positive indicator, but as a
negative indicator it can be useful.

[1]: http://en.wikipedia.org/wiki/Dunning-Kruger_effect

Given that the matrix takes only about five minutes to fill out
(significantly less time than writing a cover letter), it's hardly
a burden on the candidate, and I think it provides enough useful
information to justify the cost.

cjs
-- 
Curt Sampson       <cjs@example.com>        +81 90 7737 2974
           Functional programming in all senses of the word:
                   http://www.starling-software.com