Creativity (and imagination) are the aspects of cognition that most intrigue me in terms of the advancement of AI. Probabilistically, it's easy to be creative; you just generate many variations on a theme. But it's choosing the theme imaginatively and exploring those ramifications of the change that lead to surprise and intrigue in the mind of the audience that make a variation fruitful and compelling. The cognitive skill to choose just the right variation that leads to wonderment is ineffable. I think it exists NOWHERE in words, thus the essential cost function needed to be creative is both unstated and unintuitable (at least by a mind that lacks the ability to reason). Thus I see no way a LLM will be ever able to acquire that skill through stochastic parrotry from the available fodder -- the human narrative.

Even after the impressive advances we have seen in the fluid expression of natural language in LLMs, I remain convinced that blindly generating and testing variations in prose will NEVER create a Shakespearian sonnet. So whither superintelligence if it lacks creativity?