Shawn Wang: belief

Transcript context

…Yeah. That's a good question. I don't know. I think there's a limit with respect that we grow with respect to the model size. So bigger models means possibly bigger vocabulary without affecting too much the training. But yeah, there's a lot of people, that's not my domain of expertise, but a lot of people are discussing the interest of having this kind of tokenizer, which doesn't fit like natural. Could we go to character level tokenizer? Could we go to actually multimodal tokenizer, which will like decompose at pixel level? I don't know. Future directions that could be very promising. I would say the diffusion people have actually started to swing back to pixel level and probably that will presage the language people also moving towards, you know, 1 million vocabulary and then, you know, whatever the natural limit is for character level. I think we can maybe transition towards some of your personal stuff. We kept you here for a long time. We also, this is a very distributed podcast, you know, I'm in the Bay Area, you're in France, Sean is in Singapore, so everybody is on a different time zone. You also do, you know, some startup investing and advising, you know, we also meet Chantal on the podcast. He also mentioned he always enjoys kind of working with founders and researchers. Any company you're involved with that you want to shout out that you think is super promising, requests for startups that you've had, anything around that space would be awesome.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.