[tlug] Re: [Hikari] Work on new VPS servers

[Curt Sampson <cjs@example.com>]

On 2009-02-12 22:23 +0900 (Thu), Henri Servomaa wrote:

> The downside is that not many people know how QAM works...

That's not a downside at all; it's just irrelevant, for reasons I'll get
into in a longer, and much more philisophical, post on the main list.

But to sum up the management decision you guys need to be dealing with
right now, look at it this way. Not many people know how your current
system works. If you handed this system to anybody who'd only interacted
with it as a user and said, "set it up again on another machine," it
would be an enormous amount of work. (Look at how much time it's taking
now, even with people who know the system doing the work.)

The point of QAM is that it turns a lot of this sort of knowledge into
code and scripts, so that even someone who doesn't know the details
can at least know if things are working or not, which is probably the
hardest part of dealing with these sorts of situations. (Typically, the
solution is, "roll it out and see if any users complain.") You just do a
checkout, run the "Test" script at the top level, and then start fixing
errors.

As an example of the sort of situation in which we use this, you can
look at our "big web site project" which you're somewhat familiar with.
This is a system with two different types of hosts running web servers
(currently sixteen dynamic content servers and five static content
servers), a host running background processing, and a host running the
DBMS. It currently includes several different server programs (lighttpd,
FCGI backeends for lighttpd, memcached, mogilefs, a DAV server, etc.
etc.), about ten diferent distinct server and daemon configurations, and
the code spans at least three different languages (C, Perl, and Ruby)
and uses more than 30 different libraries of the sort that don't come
preinstalled with most full-featured linux distributions (plus a lot
more that do).

If we had to manage and test all of these pieces individually on each
of the thirty or so production servers and yet more development servers
that pieces of this system have run on in the last half year, we'd be
sunk with sysadmin work. I developed QAM specifically to deal with this
sort of thing.

The philosophy behind this is: if you're given a new server with a fresh
OS install, and you can't get get a new copy of your production system
up and running within a couple of hours, something wants to be fixed.

On 2009-02-12 22:55 +0900 (Thu), zev wrote:

> How would it be better than say Capistrano, Chef or Puppet?

Well, keep in mind that I'm only familiar with these in passing, so I
might well be missing some important details here. But from what I've
seen of these, they're pretty similar in focus to other tools I've
built in the past (and that I'm rebuilding now--*sigh*) in that they're
designed to maintain the configuration of hosts.

QAM does something superficially similar, at first glance, but it's got
a different focus: QAM maintains the configuration of applications,
ideally in a way that's fairly independent of the host.

One example would be that some of the special libraries that an
application uses are kept in the project, and built for that project.
So not only does this deal with the library not being present on the
host, it deals with a different version of the library being present,
or having to have a special patched version of the library for some
particular thing.

The consequences of this different orientation go beyond that; we also
can usually run multiple instances of a project or application's servers
on a single host, and we can have multiple applications running at the
same time, even if they do things like use incompatable libraries or
php.ini files or whatever.

QAM is also oriented towards testing the application; while Capistrano
and the like can look at a configuration and change it, these systems
don't appear to me to be able actually to, e.g., make requests of a web
server and make sure it's actually talking to the database, much less be
able to load up a full set of test data, start all the servers, and test
that the code is working correctly.

All that said, as we've seen before when discussing this, QAM has some
weaknesses, especially when it comes to dealing with suid things, or
where it can't get enough control over all of the pieces that it can do
things like change the ports that servers and clients run on in order to
run tests. For pieces that involve that, it would likely be better to
to turn other tools. I'm not proposing that we try to shovel everything
into QAM, just that we find the pieces that are suited to it, use QAM to
manage those, and see if we can save ourselves some effort in the long
run, as opposed to saving the world.

BTW, I'll be at the nomikai tonight if anybody wants to discuss this in
person.

cjs
-- 
Curt Sampson       <cjs@example.com>        +81 90 7737 2974
           Functional programming in all senses of the word:
                   http://www.starling-software.com