Relying too much on pre-trained knowledge may hinder neural networks' ability to generalize beyond existing internet data.
You don't need or want the knowledge. I think that's probably holding back the neural networks overall because it's getting them to rely on the knowledge a little too much sometimes.