Couldn't agree more, your take on the ephemeral nature of LLM 'agency' really hits home. Do you think this disconnect comes from a fundamnetal limitation in how these models represent true understanding, or something else entirely?
I speculate that LLMs having a deeper sense of understanding (or rather a higher fidelity simulation of understanding) might make things better or worse depending on the incentives baked into the system on a level that can not be adjusted by the user. I instructed Claude in the system prompt clearly to stay away from any kind of judgment calls ("because you are not able to judge things!!111“) and limit his coaching to pointing out patterns in data while leaving the judgment to me. That kind of works, but not always. I think the more these LLM products are designed to capture our attention, the harder it will be to avoid agency traps, because perceived agency (not ACTUAL agency) seems to be the new currency in this game, replacing attention.
Couldn't agree more, your take on the ephemeral nature of LLM 'agency' really hits home. Do you think this disconnect comes from a fundamnetal limitation in how these models represent true understanding, or something else entirely?
That is a great question!
I speculate that LLMs having a deeper sense of understanding (or rather a higher fidelity simulation of understanding) might make things better or worse depending on the incentives baked into the system on a level that can not be adjusted by the user. I instructed Claude in the system prompt clearly to stay away from any kind of judgment calls ("because you are not able to judge things!!111“) and limit his coaching to pointing out patterns in data while leaving the judgment to me. That kind of works, but not always. I think the more these LLM products are designed to capture our attention, the harder it will be to avoid agency traps, because perceived agency (not ACTUAL agency) seems to be the new currency in this game, replacing attention.