By far, a Japanese film called In The Realm of the Senses. The BFI screened it a few times last year. I've seen more pornographic but none still considered arthouse.
The Texas Chainsaw Massacre. I'd never seen it and watched it about 10 years ago, when the ban was lifted. At the end of it my mate and me looked at each other and said simultaneously "that was seriously fked up".
I think the OP forget the key words-in a "mainstream film" -heard ANTICHRIST is pretty bad? - although in lots of films the description is worse than the reality