I can't say I've ever found it a problem with mine, but then I've never cleaned the rest of the car either. Driving is sooo much more fun than cleaning.
I guess you need to use some kind of wax to stop dirt adhering to them. Much the same as any other paintwork.