Effective testing of A/V equipment - can you *really* tell?
Discussion
This thread is started as a reference point and for discussion about testing A/V equipment coming out of the epic thread here: http://www.pistonheads.com/gassing/topic.asp?h=0&a...
Posters on this forum spend a lot of money on equipment with the hope that it is 'better'. On the flipside there is the argument of how much difference there *really* is. Out of this of course comes discussion.
As a point of reference, common terms discussed in this thread will be:
'Blind testing' - Where the viewer/listener does not know the model they are testing so their opinion in not swayed.
'Double blind testing' - Where as well as the above the person setting up the test does not know which model is being tested, or they are separate from the viewer/listener so not to influence them.
'ABX Testing' - Where the viewer/listener experiences a comparison between two components then the tester challenges the viewer/listener to choose which one out of the first two a third test is identical to.
Open to the floor...
Posters on this forum spend a lot of money on equipment with the hope that it is 'better'. On the flipside there is the argument of how much difference there *really* is. Out of this of course comes discussion.
As a point of reference, common terms discussed in this thread will be:
'Blind testing' - Where the viewer/listener does not know the model they are testing so their opinion in not swayed.
'Double blind testing' - Where as well as the above the person setting up the test does not know which model is being tested, or they are separate from the viewer/listener so not to influence them.
'ABX Testing' - Where the viewer/listener experiences a comparison between two components then the tester challenges the viewer/listener to choose which one out of the first two a third test is identical to.
Open to the floor...
Edited by JustinP1 on Sunday 16th January 13:47
Kind of a side issue, but in recent years I've done all my demoing of 'new' equipment at home. In some cases this has meant me buying used for a good enough price that I could sell it on later at little or no loss if needs be. In some cases it involved putting a credit card deposit down and bringing home the shop's own demo unit. I even had a dealer bring two JVC projectors round on one occasion as I wasn't sure my room was good enough to reveal the differences.
Some gear has gone back, some sold on and what I feel was worht keeping is still in my rack. It's mostly used stuff (Arcam processor/power amps, PMC speakers and the 'piece de resistance' an Isco anamorphic lens to compare against simply zooming my projector for 2.35:1 content). I figure if it doesn't work in my system regardless of being 'blind' tested then it goes/goes back.
Going back closer to the subject matter of cables, I bought what I thought was decent cable purely as it was going into a wall and wouldn't be easy to swop. It happened to be biwire as well, so recently as I had three spare power amp channels I biamped the front three speakers. It took me about an hour of faffing about behind the equipment rack and if I'm honest I couldn't really say if it sounded better or not, but I couldn't be bothered to rearrange it back to single wired. Previously it was bi wired, but again I don't think I could really have told if someone had disconnected the second pair and put the links back on. Maybe I'm not that refined in my requirements, but I can spot if my processor is on the wrong mode soon enough, so maybe not...
Some gear has gone back, some sold on and what I feel was worht keeping is still in my rack. It's mostly used stuff (Arcam processor/power amps, PMC speakers and the 'piece de resistance' an Isco anamorphic lens to compare against simply zooming my projector for 2.35:1 content). I figure if it doesn't work in my system regardless of being 'blind' tested then it goes/goes back.
Going back closer to the subject matter of cables, I bought what I thought was decent cable purely as it was going into a wall and wouldn't be easy to swop. It happened to be biwire as well, so recently as I had three spare power amp channels I biamped the front three speakers. It took me about an hour of faffing about behind the equipment rack and if I'm honest I couldn't really say if it sounded better or not, but I couldn't be bothered to rearrange it back to single wired. Previously it was bi wired, but again I don't think I could really have told if someone had disconnected the second pair and put the links back on. Maybe I'm not that refined in my requirements, but I can spot if my processor is on the wrong mode soon enough, so maybe not...
Work on the same system as Oldskool, buy secondhand or borrow new, try it out, if I like it I keep it, otherwise it gets sold on/goes back. So much sounds different in your room compared with any other room (vendors/mates/etc) that this really is the only way to do it. The room can make or break a system
I think it depends. AV equipment like amps/receivers/subwoofers are more difficult to tell IMHO due to the sound output of movies, which is mainly dialogue and special effects.
Loudspeakers and TV's are a different matter though as all have a very noticable presentation.
Listening to music is the easiest way to measure differences in loudspeakers and IMHO there are definite differences in quality from spending £100 and £1000.
Loudspeakers and TV's are a different matter though as all have a very noticable presentation.
Listening to music is the easiest way to measure differences in loudspeakers and IMHO there are definite differences in quality from spending £100 and £1000.
It is very hard to know how much difference certain items make and at the end of the day, everyone's eyes and ears are different so what looks and sounds good to one person, may not to another. We always try and specify kit that we have heard and had a play with, but if a client is an early adopter that isn't always possible. I recently swapped out any amp at home from a £500 one to a £1200 one and even my wife instantly noticed a significant improvement in sound and picture quality, but I can make an even bigger improvement by moving my speakers from their current compromised position (chosen because it's a cottage living room and not a cinema!) into the optimum position - that really does make an astonishing difference.
When I demo'd my system (which is mainly for music) it was relatively easy to tell the difference between the speakers, power amps and even AV processors. Admittedly I have a relatively high end set up but the difference between say the power amp's that I tried was night and day. I think CD players and DVD players can be harder to tell apart once you get to a certain level. Having said that the difference in CD playback between my Copland CD player and my Arcam DVD player (if used for CD playback) is huge but so is the price difference between the two but of kit. I've never tested cable so have no clue about them. I used to have some old bookshelf speakers which I bought some granite speaker stands for....they made a massive difference.
grumbledoak said:
While there is definitely a difference as price increases, there is a lot of bulls
t in the world of HiFi too. What proportion of people actually trial a 'better' component than their current one, in situ, and think it worse?
That's a very good point.t in the world of HiFi too. What proportion of people actually trial a 'better' component than their current one, in situ, and think it worse?Edited by grumbledoak on Sunday 16th January 18:55
I also must admit that I have a very high end system by any respect and I use it for work, mixing, so I am using it for many hours a week.
Therefore, when I need to change the system I do it only rarely, simply because it is important I am 'used' to the sound as it is a reference when I mix.
When I have done however I've set up A/B tests to that I can switch very quickly from one to the other, if possible, remotely without getting up.
I've tested a few things, for example two very good DACs and the difference was only apparent if I 'honed' my listening to certain key things that were only apparent in some musical sources. One sounded 'warmer' as it had a different tonal characteristic, and one slightly more 'open' maybe due to the opposite effect. One was not 'better' than the other by any means, they were slightly, slightly different.
Indeed, I've also tested the digital output from a £3000 reference CD transport and the sound file from £300 Mac mini. In that case, the transport went to ebay, because even after hours of listening with as many A/Bs as I could possibly do, they are identical as can be.
I would say that unless the change in your system is large - like changing an amp, or speakers, and especially if you have to change listening position and wait a minute or more to change things over, that it would be very, difficult, or even impossible to be objective in comparison.
FlossyThePig said:
This may not be HiFi but read this article on Ben Goldacre's Bad Science website.
I was going to post the link on the speaker wire thread but that has gone on too long.
No, it's better here. Interesting stuff. I'd known about auditions behind curtains for a while. It is interesting that although we think we are impartial we will always have natural and nurtured signifiers that make up our opinion.I was going to post the link on the speaker wire thread but that has gone on too long.
There's some stunning examples about our perception of people around such as two 'identical' photos, where the subject *for some reason* looks more attractive in one than the other. You'll look at it for a minute, round and round, but you don't know why. Then, when it is revealed, it is obvious - one has the pupils artificially dilated. Just a few pixels.
Linking back to testing, what is clear is that when testing you must ensure than any other outside stimuli are taken out, however much you think you can overlook them if you try.
If you really want to test this stuff properly you need white noise generators and frequency analysis equipment. Anything else is subjective, and those decisions will include factors including "How much did I pay for it" as well as "How much do I like it".
Where is that wonderful article on HiFi bks, including "Audiophile wood" (expensive shelves) when you need it?
FWIW, I have a teeny tiny Technics system, upgraded wires, and the speakers firmly nailed to the wall in "something close to optimal". I love it. I've also watched a mate buy so much high end stuff that the shops would simply give it to him to test. He's scaled back a bit, sold a few bits, and bought an Audi S4 with the proceeds. Yes, his system sounded better.
Where is that wonderful article on HiFi b
ks, including "Audiophile wood" (expensive shelves) when you need it?FWIW, I have a teeny tiny Technics system, upgraded wires, and the speakers firmly nailed to the wall in "something close to optimal". I love it. I've also watched a mate buy so much high end stuff that the shops would simply give it to him to test. He's scaled back a bit, sold a few bits, and bought an Audi S4 with the proceeds. Yes, his system sounded better.
Edited by grumbledoak on Tuesday 18th January 14:09
IMO,
Yes, you can tell. Both in blind, and double blind. I've never done this ABX thing (not had the facility), but you can definately tell. However - there are 2 caveats:
1 - The law of diminishing returns very much applies to all things Hifi and AV - for all components, but especially things like racks, cables (I've never heard any diffrenence in a rack, but I've nevre spent much time in the 60k a 2 channel system arena, either)
2 - After a while in the industry, you almost become a "trained listner". I belive that a lot of proper Hifi beard types are the same, but also get as much enjoyment out of their ability to hear the tinyest thing on their mega money system, as they do from listening to the track at hand.
As an aside, for a vast quantity of "high end" AV guys, I think the best "upgrade" they can ever do is to set the system up properly / add an EQ. Sadly, I've seen loads of expensive setups sound horrible in room due to bad placement / treatment / setup - if anyone wants an example go to the cinema on Deansgate, Manchester. Whoever mounted the centre channel above the screen needs to be shot twice, before being fired for incompetence. Bet they spent a fortune as well!
Yes, you can tell. Both in blind, and double blind. I've never done this ABX thing (not had the facility), but you can definately tell. However - there are 2 caveats:
1 - The law of diminishing returns very much applies to all things Hifi and AV - for all components, but especially things like racks, cables (I've never heard any diffrenence in a rack, but I've nevre spent much time in the 60k a 2 channel system arena, either)
2 - After a while in the industry, you almost become a "trained listner". I belive that a lot of proper Hifi beard types are the same, but also get as much enjoyment out of their ability to hear the tinyest thing on their mega money system, as they do from listening to the track at hand.
As an aside, for a vast quantity of "high end" AV guys, I think the best "upgrade" they can ever do is to set the system up properly / add an EQ. Sadly, I've seen loads of expensive setups sound horrible in room due to bad placement / treatment / setup - if anyone wants an example go to the cinema on Deansgate, Manchester. Whoever mounted the centre channel above the screen needs to be shot twice, before being fired for incompetence. Bet they spent a fortune as well!
grumbledoak said:
If you really want to test this stuff properly you need white noise generators and frequency analysis equipment. Anything else is subjective, and those decisions will include factors including "How much did I pay for it" as well as "How much do I like it".
If you want a scientific answer - so you can be sure of what you are hearing is not swayed - that is the only way.As David says though, one mans 'accurate' is another man's 'too revealing'.
For example, if you wanted to 'smooth out' your system some nice thick cable should do that. If that's what you want then that's cheap.
grumbledoak said:
If you really want to test this stuff properly you need white noise generators and frequency analysis equipment. Anything else is subjective, and those decisions will include factors including "How much did I pay for it" as well as "How much do I like it".
Some Gump said:
As an aside, for a vast quantity of "high end" AV guys, I think the best "upgrade" they can ever do is to set the system up properly / add an EQ. Sadly, I've seen loads of expensive setups sound horrible in room due to bad placement / treatment / setup
resurrection 
regarding testing stuff properly.Many moons ago I was lucky enough to have an anechoic chamber + £35k worth of Bruel & Kjaer gear to play with
. Today, assuming there is a computer already available, it now costs less than £125 to buy all the kit ( competent instrument mic, mic pre amp, duplex sound card + freeware ) needed to make high quality MLS, Sine, RTA, Energy time curve, distortion and waterfall measurements of your Hi-Fi and AV gear in situ. Improvements can be huge at low frequencies when the measurements are used to optimise Hi-Fi speaker postioning or AV subwoofer phase/integration. Is there anyone on PH who is using similar systems already ??The cost of high quality room correction is tumbling too ~ pro company KRK have a sub £450 product called ERGO which uses Lyngdorf's algorithms. http://www.krksys.com/krk-ergo.html
RC freeware is available now too http://www.juicehifi.com/index.html albeit without the years of TACT/Lyngdorf development behind it.
Edited by Crackie on Thursday 22 December 23:28
Crackie said:
Improvements can be huge at low frequencies when the measurements are used to optimise Hi-Fi speaker postioning or AV subwoofer phase/integration. Is there anyone on PH who is using similar systems already ??
Certainly for subwoofer use I have an Antimode 8033C which comes with a mic and can be set up to correct at one main seat and another position (to create an averaged correction IIUIC). Many AV amps these days have Audyssey EQ built in too, plus I know a few guys over on AVForums have the external Audyssey processor.My simpler 8033C solution certainly tames the peaks compared to without: In fact it took a little getting used to as at first you might wonder where some of the bass has gone. My Arcam AV9 doesn't have Audyssey built in so it's a half way house solution as ti upgrade my AV9 would require some thing like the £3000+ AV888 so not likely to happen until they drop to a good price used (as my AV9 was, since it too was similar price when new).
OldSkoolRS said:
Certainly for subwoofer use I have an Antimode 8033C which comes with a mic and can be set up to correct at one main seat and another position (to create an averaged correction IIUIC). Many AV amps these days have Audyssey EQ built in too, plus I know a few guys over on AVForums have the external Audyssey processor.
My simpler 8033C solution certainly tames the peaks compared to without: In fact it took a little getting used to as at first you might wonder where some of the bass has gone. My Arcam AV9 doesn't have Audyssey built in so it's a half way house solution as ti upgrade my AV9 would require some thing like the £3000+ AV888 so not likely to happen until they drop to a good price used (as my AV9 was, since it too was similar price when new).
I've not used one but have seen nothing but good press for the 8033. Did you experience any issues with one 'main seat position' ? Fixing peaks at the listening position is ideal but if there's a null at the measurement point(s) it's easy to run into problems with amp clipping and driver excursion. My simpler 8033C solution certainly tames the peaks compared to without: In fact it took a little getting used to as at first you might wonder where some of the bass has gone. My Arcam AV9 doesn't have Audyssey built in so it's a half way house solution as ti upgrade my AV9 would require some thing like the £3000+ AV888 so not likely to happen until they drop to a good price used (as my AV9 was, since it too was similar price when new).
I do indeed have a null point at 'my seat' but the beauty with the 8033C is that is only cuts peaks and doesn't try to boost troughs. There isn't any point as if it's a true null point you still won't improve it.I did the set up at the main seat and also in a second postion so it could average the results for other seats. My experience with Audyssey wasn't so good: My old Denon 2808 had Audyssey and it added a 13db boost at 15Hz which combined with 'War of the Worlds' played a bit lound damaged my Monolith's driver. Thankfully BK replaced it free of charge and I sold the 2808 to get a secondhand Arcam AV9 and Arcam power amps setup, hence adding the 8033C as I think that the sub is most critical in terms of needing EQ. With this setup even when I'm using my Oppo '93 to play SACDs via it's analogue outputs, the sub is still EQ'd.
I hope to add a second sub at some point, positioned so that it 'fills' in the 40Hz null at my seat (it doesn't need to be so big as even my spare XLS200 can drop to 40Hz). The 8033C will apparently be able to EQ the pair if connected to the both via a splitter.
I've spent a fortune on outboard devices, but I do appreciate their effects: The Antimode 8033C and my Lumagen Radiance Mini3D both give pleasure long after the price is forgotten...
I hope to add a second sub at some point, positioned so that it 'fills' in the 40Hz null at my seat (it doesn't need to be so big as even my spare XLS200 can drop to 40Hz). The 8033C will apparently be able to EQ the pair if connected to the both via a splitter.
I've spent a fortune on outboard devices, but I do appreciate their effects: The Antimode 8033C and my Lumagen Radiance Mini3D both give pleasure long after the price is forgotten...
Edited by OldSkoolRS on Friday 23 December 12:42
OldSkoolRS said:
I do indeed have a null point at 'my seat' but the beauty with the 8033C is that is only cuts peaks and doesn't try to boost troughs.
I hope to add a second sub at some point, positioned so that it 'fills' in the 40Hz null at my seat (it doesn't need to be so big as even my spare XLS200 can drop to 40Hz). The 8033C will apparently be able to EQ the pair if connected to the both via a splitter.
I've spent a fortune on outboard devices, but I do appreciate their effects:
Re-positioning sub(s) isn't always an option but it may be possbile to shift the null away from your main seat if you have a real time display of what's going on.I hope to add a second sub at some point, positioned so that it 'fills' in the 40Hz null at my seat (it doesn't need to be so big as even my spare XLS200 can drop to 40Hz). The 8033C will apparently be able to EQ the pair if connected to the both via a splitter.
I've spent a fortune on outboard devices, but I do appreciate their effects:
In conjunction with £110 worth of Behringer's ECM8000, MIC800 & UCA22, these free measurement programs should help improve things....
http://www.fesb.hr/~mateljan/arta/
http://www.holmacoustics.com/holmimpulse.php
http://pensa.fr/freqresplot/indexe.htm
http://www.sillanumsoft.org/download.htm
Gassing Station | Home Cinema & Hi-Fi | Top of Page | What's New | My Stuff