In the UK, in normal use (ie. transiently, rather than racing WOT at all times), short answer, NO!
Also, a lot of the aftermarket coolers claim amazing cooling performance, but have such poorly designed cores and end tanks that air charge density is actually reduced..........
As far as I can tell the E46 BMW diesel was only ever fitted with on intercooler, part no. 7786351. That was for the 318, 320, and 330d. Unless you're pushing more CFM than one of those you won't have a problem.