Chinese room, called it. Just with a dog instead.
The Chinese room experiment is about the internal process; if it thinks or not, if it simulates or knows, with a machine that passes the Turing test. My example clearly does not bother with all that, what matters here is the ability to perform the goal task.
As such, no, my example is not the Chinese room. I'm highlighting something else - that the dog will keep doing spurious associations, that will affect the outcome. Is this clear now?
Why this matters: in the topic of existential threat, it's pretty much irrelevant if the AI in question "thinks" or not. What matters is its usage in situations where it would "decide" something.
I have this debate so often, I’m going to try something a bit different. Why don’t we start by laying down how LLMs do work. If you had to explain as full as you could the algorithm we’re talking about, how would you do it?
Why don't we do the following instead: I play along your inversion of the burden of the proof once you show how it would be relevant to your implicit claim that AI [will|might] become an existential threat (from “[AI is] Not yet [an existential threat], anyway”)?
Also worth noting that you outright ignored the main claim outside spoilers tag.
I also apologise for the tone. That was a knee-jerk reaction from my part; my bad.
(In my own defence, I've been discussing this topic with tech bros, and they rather consistently invert the burden of the proof. Often to evoke Brandolini's Law. You probably know which "types" I'm talking about.)
On-topic. Given that "smart" is still an internal attribute of the blackbox, perhaps we could gauge better if those models are likely to become an existential threat by 1) what they output now, 2) what they might output in the future, and 3) what we [people] might do with it.
It's also easier to work with your example productively this way. Here's a counterpoint:
The prompt asks for eight legs, and only one pic was able to output it correctly; two ignored it, and one of the pics shows ten legs. That's 25% accuracy.
I believe that the key difference between "your" unicorn and "my" eight-legged dragon is in the training data. Unicorns are fictitious but common in popular culture, so there are lots of unicorn pictures to feed the model with; while eight-legged dragons are something that I made up, so there's no direct reference, even if you could logically combine other references (as a spider + a dragon).
So their output is strongly limited by the training data, and it doesn't seem to follow some strong logic. What they might output in the future depends on what we add in; the potential for decision taking is rather weak, as they wouldn't be able to deal with unpredictable situations. And thus their ability to go rogue.
[Note: I repeated the test with a horse instead of a dragon, within the same chat. The output was slightly less bad, confirming my hypothesis - because pics of eight-legged horses exist due to the Sleipnir.]
Neural networks are a different can of worms for me, as I think that they'll outlive LLMs by a huge margin, even if the current LLMs use them. However, how they'll be used is likely considerably different.
For example, current state-of-art LLMs are coded with some "semantic" supplementation near the embedding, added almost like an afterthought. However, semantics should play a central role in the design of the transformer - because what matters is not the word itself, but what it conveys.
That would be considerably closer to a general intelligence than to modern LLMs - because you're effectively demoting language processing to input/output, that might as well be subbed with something else, like pictures. In this situation I believe that the output would be far more accurate, and it could theoretically handle novel situations better. Then we could have some concerns about AI being an existential threat - because people would use this AI for decision taking, and it might output decisions that go terribly right, as in that "paperclip factory" thought experiment.
The fact that we don't see developments in this direction yet shows, for me, that it's easier said than done, and we're really far from that.