Re: [tlug] Lightning Talk Proposal for January 23 – The Fastest Batch Is the One That Does Less

[Chris Salisbury <chris.salisbury@??>]

I like the overall message, and would be interested to see how people are doing this kind of thing in Ruby these days, but it seems like expectations are a bit off with the 250 million records per month metric -- it's not unusual for people to process billions of records per second using ClickHouse, for example. I'm used to it being 10-100x faster, but it's literally 10 orders of magnitude faster than the case presented here...sometimes you might actually still want to ask, "How can we make it faster?"

On Sat, Oct 24, 2026 at 1:33 PM 越智翔一 <dekamintv@??> wrote:

Hi TLUG organizers,

My name is Shoichi Ochi, and I'm a software engineer working with Ruby
on Rails in Tokyo.

I'd like to propose a lightning talk for the January 23, 2027 technical meeting.

Title: The Fastest Batch Is the One That Does Less
Duration: 5 minutes
Language: English

Abstract:

How would you efficiently process 250 million records every month
without overloading your database?

We built a batch processing pipeline using Ruby on Rails and MySQL.
Initially, we focused on improving throughput through bulk upserts and
parallel processing. However, we encountered high database CPU usage.

After investigating the problem, we realized that fewer than 5 million
records actually needed updating out of 250 million.

By reducing unnecessary writes, we were able to simplify our
processing from four parallel workers to sequential execution while
reducing database load.

In this talk, I'll briefly introduce our batch architecture, explain
the challenges we faced, and share a simple lesson:

Before asking "How can we make it faster?", ask "Do we really need to
do all this work?"

Reference:

This lightning talk will be based on a longer presentation I gave at
Kaigi on Rails 2026. I'll focus on the batch optimization part and
adapt it into a five-minute talk in English.

You can find the original slides here:

https://speakerdeck.com/occhi/persisting-massive-data-with-rails-designing-for-60-billion-rows-and-fighting-the-bottlenecks-rails-de-idomu-tairyou-deta-eizokuka-600-okuken-o-eizokuka-suru-tame-no-sekkei-to-botorunekku-tono-tatakai

I'd be happy to share this experience with the TLUG community.

Best regards,
Shoichi Ochi