Current vision systems can convert images into high-level representations similar to LLM input tokens, which are then fed to LLMs to aid decision-making.
we have a number of ways to train vision systems, either supervised, unsupervised, self-supervised, all kinds of different ways. That will turn any image into a high level representation. Basically, a list of tokens that are really similar to the kind of tokens that a typical LLM takes as an input. And then you just feed that to the LLM in addition to the text, and you just expect the LLM during training to kind of be able to use those representations to help make decisions.