[tlug] memcached / alternatives

[Miles Colman <mcolman@example.com>]

Hi TLUG,
I wrote earlier about SSDs, but I should have asked a simpler question.

I read a little bit about linux's automatic caching of files that are regularly accessed, but it seems to be turned off at my company. At least, most of our systems show free memory in the performance monitor (and 'in use as cache' shows up there). Should I dive right in to memcached to try and cache files and see if things are faster? or is there something else I should look at first?

Background:

The guys who write the file accessing code for post-processing gave me a 'yes-and-no' answer about whether files are accessed randomly or not:

Q:

> I was just curious if [the software's] disk reads during post-processing  are
> sequential or random access.
> The reason I am asking is that I was thinking of caching snc and fnc
> files on an SSD to improve [the software's] speed during post-processing.
> Would this only make a difference in speed if we were changing which
> frame we were looking at a lot?

A:

We do not explicitly code here to consequently stream the files.
Nevertheless a scalar variable for example is read chunk wise but in successive chunks which is kind of a streaming behavior. But there are also parts of the file where we probably jump around something more.

Also keep in mind that the load times for the datasets is not purely IO bound. There are larger parts which just do algorithmical work to prepare the internal structures for later faster access. These parts are recognized as load time by the user but do not do IO operations at all and consequently cannot benefit from an ssd.

So the answer is somehow: 'yes and no'.