Language models will behave properly on prompts they have been trained on but may produce nonsense for untrained prompts.
the system will behave properly on the prompts that it's been either trained, pre-trained or fine tuned. But then there is an entire space of things that it cannot possibly have been trained on because it's just the number is gigantic