Andrej Karpathy observes that transformers are extremely powerful and general, learning patterns from various inputs like audio, video, or text.
we've stumbled by with the transformer neural network, which is extremely powerful, very general. You can train transformers on audio, or video, or text, or whatever you want, and it just learns patterns and they're very powerful, and it works really well.