This is exactly how it works, and how output from it should be interpreted.
What a generative language model does, at a high level, is take the prompt it has been given (which is what you supply plus any earlier interactions in the conversation, plus any “hidden” parts of the prompt which may have been supplied by the system you’re using to interact with it) and then try to predict the most likely response a human would make to that prompt based upon the material with which it has been trained. At a lower level, it does essentially the same thing—what is the most likely thing to come next after what it’s just written?
There is no connection whatsoever to “fact”, “reasoning”, or “correctness and consistency in calculation” except to the extent they happen to be embodied implicitly in the training material. This is why ChatGPT will frequently answer a complicated question in physics correctly because it has seen a homework problem and answer that matches it closely enough, then trip over its shoelaces trying to calculate the numerical answer because it can’t do arithmetic, only regurgitate problems it’s seen worked in its training set.
It’s this behaviour which caused me to call these systems, from my first encounters with them, “bullshit generators”. They’re precisely like that guy you knew in college who could spit our answers to questions in class or on exams based on only the most cursory knowledge of the topic but which were phrased so smoothly and with apparent authority that, unless you had your own independent factual knowledge of the subject matter, would slip right past as plausibly correct.
A common phenomenon with large language models is “hallucination” or, if you prefer, “making stuff up”. If you ask it a question about something it doesn’t have in its training set, it may match something else that sounds similar, plug in some of the words from your prompt, and present it as if it is genuine. If you’ve asked for references, or the style of writing requested usually includes them, it will make them up, inventing journals, article titles, and author names (all of which may actually exist in its training corpus, but never in connection with one another) to support its argument. This is just like the bullshitter on the debate team (I find it difficult to read this without imagining it spoken in the voice of Bill Clinton) who responds, “Clearly, the speaker for the affirmative has never read the 1996 meta-analysis by DiBenetto and Kaplan published in the Kansas Journal of Economics which found no connection between the policy she advocates and outcomes in over 300 papers published over the preceding three decades.” where everything is completely fabricated.
As an example of GPT-4’s prowess as a bullshitter, here is where I posed Chip Morningstar’s challenge to write a postmodern deconstruction of the text “John F. Kennedy was not a homosexual” in the style of Derrida, Foucault, and Baudrillard, complete with source citations.