[tlug] Build Systems

["Stephen J. Turnbull" <stephenjturnbull@example.com>]

I don't think I sent this earlier but it's been sitting a couple days
so who knows....  If so, sorry for the noise.

Curt J. Sampson writes:
 > On 2022-02-03 03:21 +0900 (Thu), Stephen J. Turnbull wrote:
 > > Curt J. Sampson writes:

 > > I don't recall pair-programming as being a requirement for:

 > >  > >  > regular get together and do real programming on real programs
 > >  > >  > on a regular basis.
 > 
 > Ah, I guess there are groups that "get together" only to chat for a few
 > minutes and then work alone.

That's not what I meant.  I just took your "pair programming" to mean
*formal* pair programming, one-on-one with rules about who types, and
who talks and about what.  Dev sprints (all the projects I mention
except Ghostscript, where I didn't meet any of the other devs until
literally decades later AFAIK) of course involve multiple people
looking at the same screen in an informal way, although not
necessarily continuously (eg, in Mailman sprints at PyCons we usually
end up with about a 3:1 visitor:core ratio, so 1-1 is not really
possible, and sometimes you'd have 3 devs looking at one visitor's
screen with a half-dozen visitors looking over our shoulders from the
peanut gallery).

 > I've fallen back to screen sharing when most of the programming is in a
 > non-graphical environment and we just need to share a bit of graphics,
 > but the case of "one of a pair can only watch" is a pretty bad bandwidth
 > limit on your communications, somewhere between "both parties can type"
 > and "can't see other screens at all; communicate in text chat."

Assuming that the screen updates and file shares (git push/pull on a
dedicated "scratch" branch, not via Zoom!) come through in real time,
I don't find it particularly limiting for a typical mentoring
session.  But then, I never tried to walk anybody through debugging
an "XEmacs on X.org on Mac OS only" problem, even in person. ;-)

 > When I hear "sit down with," I assume either actual sitting down
 > physically together in front of a computer or a session with voice
 > chat, text chat (for sharing URLs and suchlike) and shared
 > terminals.

So do I.  Not sure why you think I think being in shouting distance is
a sufficient condition.

 > you can do that with both Emacs and Vim and we all know that folks
 > who use TextMate or Sublime Text or worse yet an IDE (*shudder*)
 > aren't Real Programmers anyway. :-P)

Unfortunately it's against Google Rules to impose a text editor on
your students. ;-)

 > The most common block to using those is not technical but an emotional
 > insistence on the part of developers with power in the project (or even
 > managers) that only certain tools are orthodox

In Mailman we only allow Greek Orthodox tools (we had a Ukrainian dev
for a while so Russian Orthodox was right out).  Is that wrong of us?

 > When you've got issues like this, remote pair programming is only
 > one of the things you lose.

Not an issue anywhere I've contributed (of course in XEmacs everybody
had to use XEmacs, but it wasn't an issue somehow...).

 > Perhaps there's a misunderstanding between us about what "trail"
 > means.

There was.

 > I'm not referring to a situation where the developer completely
 > ignores the guy trailing him and just goes on with his day as
 > usual,

Which is *precisely* what it means for management interns.  (If you
know any managers, I doubt that will surprise you. ;-)

 > > Well yes, D'OH, that's *exactly* what I meant.  Is that part of your
 > > normal worflow?  Installing the source on a system you'll never touch
 > > again?  Except maybe as the typist in pair programming?
 > 
 > Well, yes. I do my tmate sessions in Docker containers (built and
 > used with dent[1]) so that I can give root access to participants

You have "participants" in your normal workflow?

 > Developers also have the ability merely to think about the likely
 > effects of their particular means of adding a dependency, and
 > change how they do it based on that. If they just added one that
 > required finding obscure web sites, downloading stuff from them,
 > and going through a laborious install procedure, they ought to be
 > thinking about how the next guy's going to do that and what they
 > should put in the repo (scripts, even) to smooth this out.

Not something I've dealt with, as explained elsewhere.  In Mailman,
all dependencies are *really* external (the RDBMS) or they're
pip'able.  In XEmacs, Python, and Ghostscript the dev's responsibility
was to ensure configure can find system resources, not install them.

 > >  > In my world, all you need to do is `git clone ...` and then `./Test` in
 > >  > that repo, and that top-level build/test script will fetch and build any
 > >  > dependencies needed or at least inform the developer in some reasonably
 > >  > comprehensible manner what she's missing.
 > >
 > > I gather you're not working in C. ;-)
 > 
 > I am indeed working in C! Well, it's not my code, but I build it, and have
 > to deal with exactly those dependency issues. For example, I use the
 > linapple Apple II emulator in my 8-bit development, and the most convenient
 > way to deal with this dependency is just to fetch and build it. So I list
 > out the dependency tests with the Debian package names:[2]
 > 
 >     DEPENDENCIES = (
 >         ('git',                     ('git', '--version')),
 >         ('imagemagick',             ('convert', '--version')),
 >         ('libzip-dev',              ('pkg-config', 'libzip')),
 >         ('libsdl1.2-dev',           ('pkg-config', 'sdl')),
 >         ('libsdl-image1.2-dev',     ('pkg-config', 'SDL_image')),
 >         ('libcurl4-openssl-dev',    ('pkg-config', 'libcurl')),
 >         ('zlib1g-dev',              ('pkg-config', 'zlib')),
 >     )
 > 
 > [2]: https://github.com/0cjs/8bitdev/blob/5ec968039580d7a3df4f0bf0d062d5159c43a841/b8tool/pylib/b8tool/toolset/linapple.py#L31-L39
 > 
 > The code that uses this prints out the names of all the missing packages in
 > the form of an `apt-get` command you can run. Here I assume that Red Hat
 > users will be able to figure out the equivalent package names on their
 > systems, but this could easily be extended to record those too, if someone
 > felt it was worthwhile.

That's fine if you can assume that your code needs to build only on
very mature Linux distros.  I don't think this would work so well if
you had to support Homebrew on Mac or random "since 2021" Linux
distros, let alone *BSD, etc and Windows (as Ghostscript, XEmacs, and
Python do, which is where the "not working in C" crack came from).

 > > Snide "a plague on all concerned" language bigotry aside,
 > > building the software is very little of what I'm talking about,
 > > except for people who are quite new to programming as well as to
 > > the project in question.
 > 
 > Huh. My experience has been that there are a lot of projects out
 > there where the build is really, really difficult.

There certainly are.  I haven't worked *on* any of them, though I've
occasionally tried to build some of them to work *with*.

 > > I gather you haven't paid much attention to how a Python venv works?
 > 
 > You couldn't be more wrong about that; I'm a religious user of Python
 > virtual environments.

Then you would know that you have to keep the build really clean if
your "make test" runs tox for six or nine different combinations of
Python + Django + backend RDBMS.

 > having Python interpreter itself available) that I don't even
 > require you to have pip available,

Are you using Python 2 still?  python -m pip goes back to at least
Python 3.5 ....

 > That brings us to the second point: there actually are (or should be) what
 > are at least conceptually two separate build systems here. One is for
 > people who want to build a working version of the software to use on their
 > system,

Isn't that "./configure && make && sudo make install", vs

 > and the other is for developers who want to test that the software
 > can be built on a wide variety of systems in a wide variety of
 > configurations.

"""
for opts in 'with-bignum=gmp' 'with-bignum=mpir' 'with-bignum=mp'; do
    rm -rf "$opts"
    mkdir "$opts"
    cd "$opts"
    configure --$opts && make maintainer-clean && make beta && make check
done > test-all-`date +%Y%m%d`.log
"""

?  Or whatever variant is appropriate for your build tools, that's
almost how I used to build XEmacs back when we were releasing betas
2X/month (except with about 50 variations on $opts, those are just the
options for multiprecision integers, and a make target that included a
special version string for betas).

 > > That's not been my experience, but most of my experience (except for
 > > Mailman) has been with applications that need to work on every system
 > > known to man (and some known only to woman).
 > 
 > Yes, open source projects designed to be built by a lot of people other
 > than the developers are usually not going to end up in the state I
 > described above. That's more likely to happen with proprietary projects
 > where having a "the build is your problem" attitude isn't able to drive
 > customers (of the build) away.

Proprietary projects aren't my problem (yet -- watch this space!) and
certainly not my experience.

But I'm not clear on why proprietary projects are in scope for this
discussion.  It seems to mean that making sure onboarding and build
systems are done well are among the useful functions of managers.  If
that means making experienced devs handle interns and new hires
trailing them, and detailing somebody to test and maintain the build,
that's what you do.  If your managers aren't making sure that new
hires get up to speed quickly, as you wrote earlier you've got the
bigger problem of an organization that's probably dysfunctional in a
lot of ways.  It's voluntary organizations like most open source
projects where these are some of the bigger problems that kill
unsuccessful projects and successful ones solve somehow.

 > Right. [A "code ownership" issue] seems to me exactly the sort of
 > thing you'd be likely to encounter with more than a trivial amount
 > of trailing, at which point surely the developer would explain why
 > he's making the decision not to touch some code or seek advice or
 > whatever, right?

For several values of "sort of thing", yes.  But as I wrote above, I
would consider this kind of trailing to be a special onboarding
workflow, not my daily workflow.  And I'd probably be a lot more
aggressive about many decisions in my normal workflow (one of the
privileges of being a core dev, at least in the projects where I've
been core, is EAFP-ing these decisions, where newbies are expected to
LBYL).

 > I don't see these as "big project" problems. The smaller projects are
 > dealing with the same core problems; they just (almost naturally, due to
 > being smaller) usually have much better communications within the project
 > that gets these things resolved more quickly. Often, with experienced
 > developers, so quickly that you probably don't even see it as having
 > encountered and solved a problem.

That's what I mean by them being "big project" problems, I guess.

 >     - L Peter Deutsch

Funny you'd mention him. :-)  I do enjoy name-dropping him (in company
that gets the joke), it gets attention. :-þ