Re: [tlug] programmer competency matrix

[Joe Larabell <fred62@example.com>]

> I'd forgotten about the silliness of the API memorization thing (I'm in
> API docs all the time, myself), and so I need some phrasing to fix that.
> Any suggestions on phrasing?

You could phrase it in terms of API capabilities. For example, I know 
pretty much nothing about the Windoze APIs because I try my best to avoid 
that platform at all costs. But I know pretty much everything that can be 
done with the LibXML2 API, even though I still have to keep a FF tab open 
on the docs and look up almost every call to be sure I have the right 
arguments in the right order (my memory isn't what it used to be ;-).

Either that or something like "knows what various APIs can do and where to 
find the details of how to invoke the functions". I dunno... that also 
sounds too lame. Maybe you can leave it as-is and ask the person in the 
interview how they felt about that question. If they took it as meaning 
"memorizing things is more important than knowing how to find them", you 
may have another useful negative indicator.

> For all the problems it has, I think that the matrix is far above the
> level of a checklist of acronyms. Surely you aren't going to argue that
> "no RDBMs experience" vs. "uses MySQL once in a while" vs. "understands
> the relational algebra" is no more useful than "RDBMS: yes or no".

Yeah... you're certainly right there. The 1/2/3 rating scale on the list 
of acronyms gave me no clue as to which box to check for most of them. At 
least the matrix levels were more clearly spelled out. And it might help 
prevent the self-overrating problem, since the candidate would be able to 
actually see the various things he might be called upon to discuss in the 
live interview.

> ... working
> with someone for three months is better. So why do we bother with
> anything else? Cost.

Understood. I've never worked anywhere where we took in anyone who came 
along with the intention of weeding folks out later. I'm not even sure 
that's fair to the candidate, even if he was willing to take the 3-month 
contract.

> If you look at where the matrix is in our process, you'll notice that we
> ask for it with the resume. We consider resumes to be moderately useful
> to disqualify candidates, and basically useless as a positive indicator.

That's interesting. My last three employers asked for a resume after the 
fact -- that is, after I'd already been hired -- just because they needed 
it as part of their HR process.

The thing is... you'd probably be better off hiring a really smart guy who 
has absolutely no experience in your field of business than hiring a 
mediocre duff who can talk around all the buzzwords. I've seen plenty of 
the latter and they *do* interview well. However, when the rubber meets 
the road, all you get is a lot of squealing and smoke.

It didn't escape me that you could (and possibly were) using the matrix as 
a way of figuring out how self-aware the candidate might be. I believe 
someone once said: "he who knows little thinks he knows a lot but he who 
knows a lot has learned his own ignorance". Had I been desperate for a job 
when I saw that acronym list, I may well have checked off half the stuff 
there, figuring I could wing it long enough to gain some real knowledge of 
the technologies involved -- all before anyone found out. At least I might 
have been tempted to try.

So, up to a certain level, you might want to use the inverse function on 
the matrix to get a good idea where someone is *really*. If, by chance, 
you have a real superstar in front of you, you'll probably set the matrix 
aside early in the interview anyway.

I'm thinking that maybe each "cell" in the matrix could have a checklist. 
The prose already lists a number of "for instance" items in most cases. 
Maybe turn those into checkboxes. It's a lot easier to check boxes than to 
write out a list, it might solve the "higher includes lower" problem, and 
it would give you more data points for averaging.

> ... By comparing experience on a resume and the answers to the
> matrix, I can tell if someone's rating of himself seems completely out
> of whack.

Have you done that? I would think a person with an over-inflated idea of 
their own accomplishments would embellish both thei resume and the matrix. 
I've interviewed candidates who mentioned things like "leading projects" 
and "designing systems" that, when asked how the thing worked, answered 
with something like: "Oh... that was so-and-so's job -- I just did what he 
told me needed to be done". Oh... so much for "leading" the project.

I don't know how you can easily weed out those guys without at least a 
screening interview. Their martix will scream "superstar" just as loudly 
as their resume. At least screening interviews aren't as costly but you 
still have a limit as to how many you can screen in a given amount of time 
and I agree with your suggestion that being able to weed out the duffs 
early saves a lot of time/money for everyone involved.

Maybe subjective essay questions. Maybe get them to explain why they think 
FP is better/worse than OO. Or what makes Ruby such an attractive language 
to ex-Perl fans. Or their own views on why there may never be a "silver 
bullet" in the area of software design. Or maybe give them a list of a 
dozen such subjective questions and ask them to make one up on their own
and answer it. I've been wanting to write at least one "advocacy" paper to 
send around to my colleagues but I never seem to have the time.

These kinds of things you can't just look-up on Google. But it might take 
quite a bit of original research to come up with a coherent argument. And 
if the questions are flexible, the potential candidate can draw on their 
own experience, which gives you yet more insight into whether they are 
likely to be a problem solver or a problem maker.

After all, that's all you need to know. Will this person, when faced with 
a problem at Starling, be able to draw on his own experience and the vast 
pool of data available on the Internet, and come up with an elegant 
solution even if the problem is not like any he's already solved before.

> Given that the matrix takes only about five minutes to fill out
> (significantly less time than writing a cover letter), it's hardly
> a burden on the candidate, and I think it provides enough useful
> information to justify the cost.

Yeah... I guess I have to agree. And having done a reasonably thorough job 
of filling out the matrix, irrespective of its content, tells you about as 
much about the candidate as the cover letter does, eh? At least you know 
if the person is interested enough in the job to put in the time. (BTW, I 
use a similar tactic with headhunters. If the initial contact doesn't show 
that the recruiter at least *looked* at my web site (which is linked from 
my online resume), the mail goes in the bit-bucket.)

---
Joseph L (Joe) Larabell            Never fight with a dragon
http://larabell.org/                    for thou art crunchy
http://thelemicleague.org/        and goest well with cheese.