Technology

We do know that they can be coaxed into spitting out the original work, though, which sure implies it is in there.

Only very rarely, under extreme cases of overfitting. Overfitting is a failure state that LLM trainers want to avoid anyway, for reasons unrelated to copyright.

There simply isn't enough space in a LLM's neural network to be storing actual copies of the training data. It's impossible, from a data compression perspective, to fit it in there.