why is non youtube web video still so bad?

[bloomen]

Every time I wander onto a site and there's a video present that isn't hosted by youtube, it's borderline guaranteed that it's going to buffer forever or wreck the page load.

Vimeo works smoothly eventually but I don't think I've ever watched a vid there that I didn't have to wait to load.

My connection's good enough for HD netflix and BBC so I assume it's something at their end. What does youtube do differently to so many other systems?