Re: [tlug] Anyone alive out here ?

[Bernie Innocenti <bernie@example.com>]

On 2024/09/05 20:58, Darren Cook wrote:
> > Later, I installed ollama and blogged about my experience with a few 
> > freely available LLMs:
> >
> >    https://mstdn.io/@codewiz/112527717194517544
>
> That was interesting. Which AMD GPU are you using?

Radeon RX 7900 XT. It's 2 years old, but very stable and performs well 
on Linux for desktop, gaming and compute workloads. AMD's 
CUDA-compatible stack no longer sucks as it used to.

The main issue is having only 20GB of RAM. Not sure when they will 
announce the next gen of desktop GPUs with at least 40GB, which would be 
very useful for LLM inference and training.

If you need more GPU RAM, I'd recommend getting a VM on Google Cloud or 
other provider that charges by the hour.


> > What's the current state of the art? Interfacing models with text has 
> > big limits...
>
> Quite closely related, I've been wondering what the state of the art for 
> open-source OCR is, particularly of Japanese text.

I'm waiting for the first Llama-like LLM with image recognition similar 
to ChatGPT.

It's not the same of classic OCR and has obvious limitations, but a 
multimodal LLM can figure out things from context, like "oh, this is the 
hand writing of a young child... and this word is misspelled, I think 
they meant to write hamburger... pages 3 and 4 appear to be swapped".

> All the links go to Tesseract (or wrappers around it), which is simply
> not good enough, even for English. Or to online tools where you upload
> your private data, and pay for the privilege.
>
> This could then lead on to the greatest unsolved computing challenge
> of the 21st century, which is a PDF to text converter. Yes, yes, I
> know toy examples work. I mean PDFs that were made in Microsoft Word,
> contain multiple columns and figures, and inset boxes, like a
> magazine, or a company's annual report. And extracting all the
> sections with headings, in a reasonable reading order (suitable for,
> say, a screen reader).
>
> I thought Google, OpenAI, etc. had solved it, as they train the LLMs
> on PDFs, and ChatGPT can be given a PDF and answer questions. But, as
> far as I've been able to find out, they either feed raw PDF bytes in,
> and hope, or they use pdf2text, and hope.


Ultimately, narrow models for OCR, speech recognition and translation 
can't improve beyond a certain point because they lack semantic 
understanding. They also can't be instructed in plain English to adjust 
the output (e.g. "use informal language in the dialogue part, but formal 
Japanese for the professor").

Just yesterday, I asked ChatGPT to compare a 5-page lease contract in 
Japanese with an older one from 5 years ago, then summarize the 
important differences along with any clauses I should watch out for.

I could have asked "translate this PDF to English", but then I would 
have to do all the hard work by myself :-)

--
_ // Bernie Innocenti
\X/  https://codewiz.org/