Question-Answering Systems: Single-Turn to Conversational

The simplest way to use a large language model is to ask it one question and read one answer, with nothing before and nothing after. The most common way people actually use one — a chat window open for twenty minutes, a running back-and-forth about a document, a coding problem, a piece of writing — is nothing like that. Understanding the gap between these two modes, and what changes underneath when a system moves from one to the other, is the difference between using an AI tool competently and being quietly misled by it.

Single-Turn Versus Multi-Turn

A single-turn question-answering system takes one input and produces one output: no memory of anything said before, no expectation that anything will be said after. Classic search-engine "answer boxes" and early QA benchmarks were built this way, and it remains a useful mental model even when the underlying tool is conversational, because every reply an LLM produces is still, mechanically, a single forward pass conditioned on whatever text currently sits in front of it.

What makes a chat interface feel like a continuous conversation rather than a sequence of unrelated single-turn answers is that the system re-sends the entire visible transcript — your first message, its first reply, your second message, and so on — as the input to every new turn. There is no separate "memory" being consulted; the model's only knowledge of the conversation so far is whatever text has been included in that turn's input, known as its context window. This has a very practical consequence: a context window has a fixed size, and once a long conversation exceeds it, earlier turns have to be dropped or summarised to make room. That is why a long working session can appear to "forget" an instruction given twenty minutes earlier — it has not been overruled, it has simply scrolled out of what the model is currently being shown. Liu and colleagues found something sharper than a hard cutoff, too: even well inside a model's stated context limit, information placed in the middle of a long input is retrieved and used noticeably less reliably than information placed at the very start or the very end [3]. In practice, this means the position of an instruction in a long conversation or a long pasted document, not only its presence, affects whether the model actually acts on it.

Closed-Book Versus Open-Book Answering

Underneath the single-turn/multi-turn distinction sits a second, more consequential one: where the answer is coming from. A closed-book system answers purely from what was baked into the model's parameters during training — there is no document in front of it, no search happening, just whatever the model absorbed and compressed months or years earlier. Roberts, Raffel and Shazeer's foundational study of this mode found that a model's ability to answer a factual question correlates with how much relevant text it saw repeated during training [1], and a later, larger-scale study by Kandpal and colleagues sharpened that finding into a specific warning: a model's accuracy on a fact drops off steeply once the entities involved appear only rarely in its training data, which means closed-book answers are least reliable precisely on the specific, obscure, or recent facts a user is most likely to actually need looked up [2]. This is the direct mechanism behind hallucination in plain factual Q&A: the model is not retrieving a stored fact and reporting "I don't have this," it is generating the statistically most plausible continuation of the question, which for a rare fact can be a fluent, confident, wrong answer.

An open-book (or grounded) system changes what the model is being asked to do: instead of answering from memory, it is given a specific document, or a set of documents fetched by a search step, and asked to answer using that supplied material. This does not just add a source — it changes the task itself, from "recall a fact" to "read this passage and report what it says," which is a task language models are considerably better and more checkable at. It is also the exact mechanism Retrieval-Augmented Generation automates: instead of a person manually pasting in a document, the system searches for and retrieves the relevant passages itself before generating an answer [4].

Evaluating QA Quality: Accuracy, Relevance, Faithfulness

The answer sounds right" is a judgement about fluency, and fluency is precisely what large language models are optimised to produce regardless of whether the content is correct — which is exactly why it is such an unreliable signal to grade an answer by. A more useful evaluation separates out at least three different questions, which the RAG-evaluation literature treats as genuinely distinct axes rather than one blurred impression of "good answer" [5]:

  • Accuracy — is the answer actually correct, checked against a real source or ground truth, independent of how it reads?
  • Relevance — does the answer actually address the question that was asked, rather than a nearby but different one?
  • Faithfulness — when a source document was supplied, does the answer only say things that document actually supports, rather than blending in unsupported claims from the model's own closed-book knowledge?

Faithfulness is the axis most worth deliberately checking, because it is the one a confident, well-written answer gives you the least intuitive signal about. An open-book answer can be fluent, relevant, and still quietly wrong if the model has let closed-book "background knowledge" leak in alongside what the supplied document actually says — which is precisely why, in any AI-interaction design, verifying a claim against the actual cited passage (not the model's paraphrase of it) has to be a deliberate step, not an assumption.

Practical Exercise: Closed-Book Versus Open-Book, Head to Head. Choose a topic you know well enough to fact-check without help — a niche hobby, a module you've already studied, a local topic. First, ask a web-based AI tool a specific, somewhat obscure factual question about it with no document attached (closed-book) and record the answer verbatim. Then supply the tool with a genuine source document on the same topic (a PDF, a web page) and ask the identical question again (open-book). Compare the two answers against what you actually know and against the supplied source: where did the closed-book answer get something subtly wrong or vague that the open-book answer got right by quoting or closely paraphrasing the source? Where, if anywhere, did the open-book answer still add something the source didn't actually say? Write up both answers side by side with your verdict on accuracy, relevance, and faithfulness for each.

References

  1. Roberts, A., Raffel, C., & Shazeer, N. (2020). How Much Knowledge Can You Pack Into the Parameters of a Language Model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 5418–5426. https://doi.org/10.18653/v1/2020.emnlp-main.437
  2. Kandpal, N., Deng, H., Roberts, A., Wallace, E., & Raffel, C. (2023). Large Language Models Struggle to Learn Long-Tail Knowledge. In Proceedings of the 40th International Conference on Machine Learning (ICML), 15696–15707.
  3. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638
  4. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, 33, 9459–9474.
  5. Es, S., James, J., Espinosa Anke, L., & Schockaert, S. (2024). RAGAs: Automated Evaluation of Retrieval Augmented Generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 150–158.