RAW File Archives - long term support as cameras models age?

[LongQ]

Here is a thoughtful article that may be of interest to some people.

http://www.dpreview.com/articles/4167497780/the-lo...

I first came across this elsewhere. The writer makes some good points about longevity of support for what has become camera model specific file/sensor information.

To date things have been interesting but not overly complicated as far as Bayer array sensors have been concerned. The bulk of sales in the digital marketplace seem to have fallen into the 'get the colour profile right and all will be well' category. Not quite that simple but not so complicated that, for example, product relying on dcraw on their conversion process could not simply await an update and then come up with their own settings and processes for colour interpretation to act as their USP for conversion quality.

However, throw lens correction into the mix and it all changes ... and for many new cameras, esepcially the compact models that offer RAW, lens correction can be a fundamental part of the process rather than a 'nice to have' option. Why? Because the sensors now have enough detail capacity to hide the fairly large corrections sometimes required for smaller and cheaper (and therefore simpler) lens designs.

But rather than me rehearse the points here - read the article and see what you think.

[gangzoom]

Interesting read but I suspect most people wouldn't care. I use to shoot in RAW but not that I'm more confident with getting the shot right first time have switched to JPEG. I am now in the process of converting 13k RAW files into JPEGs simply because the RAW files were taking up so much hard drive space. I know RAW give the best quality, but unless your making money out of your photos I doubt the small amount of quality lost in the conversion matter??

[RobDickinson]

You think recoding raw files is going to go away?

Its not. Stop worrying.

Theres plenty of even open source raw converters, its a well understood problem with a defined solution that just needs tweaks for each different model that comes out.

[flat-planedCrank]

Wouldn't worry unduly - if you are mega-worried convert to DNG or save your edited shots as TIFs?

[LongQ]

RobDickinson said:

You think recoding raw files is going to go away?

Its not. Stop worrying.

Theres plenty of even open source raw converters, its a well understood problem with a defined solution that just needs tweaks for each different model that comes out.

So far, yes, mostly.

One could argue a case for Foveon being an outlier.

But the observation in the article is considering the future where there are already some indications that the latest and greatest features built into the cameras are giving some developers a challenge or two. The number and variety of non OEM and (as suggested in the article) non Adobe software products that can cope with all the options may drop dramatically. This may not ve a problem if you are an Adobe fan and prepared to live with whatever they decide to offer at whatever price they offer it.

The historic angle may not arise for a while either but if you take the development of computing equipment a lot of technologies have appeared, blossomed and then disappeared overnight a decade or so later. Support for obsolete products is not often carried forward for long - what would be the point? Well, other than to get at archived data that no one got around to converting, were it possible, to another format, or maybe several new formats in the period since it was created, so that it is still readily readable today.

If you dig out an old negative or glass plate or even a print you have a physical object that offers an opportunity to decide if it is interesting and, usually, do something to retrieve and reprint or digitise the image. With a digital file on some obsolete media you will probably never even know it was there.

Does that matter? Well, maybe not. Perhaps it depends on the content of the image.

Even if you know there is a file there that you want to recover it may not be possible without a lot of specialist work. Then when you get to the file if its jpg or TIFF (currently) you may be in luck but old RAW formats may well be dropped through lack of interest within a few years. In fact as core software development products change some of the older code may simply stop working as it did. With no real demand no one will put the effort in to fix that at a commercial level. Maybe they would as Open Source but that could be a big maybe.

Does it matter? Probably not. 99% of people viewing images probably don't care how good the absolute quality is. However for any one keeping an archive, whether privately or commercially,and intending that the archive should have an extended life, it's probably something they need to give some thought to.

Even, or perhaps especially, a continually developed application will be unlikely to guarantee legacy support for its own functionality. Which means it may well still be able to deal with the RAW format but maybe not understand its own old edit files. Less of a problem than not recognising the RAW of course - you can just do the edit again and maybe find a better result - but still a pain should the need come about.

But in ten years from now will any of the developers still feel a need to make lens correction routines for old Micro 4/3rds cameras? Will they still be prepared to multiple engineer Canon RAW files to cover mRAW and sRAW? Will Active D light adjustments be something that marketing people will say are 'must have' features for new products?

Who knows? Something to keep half an eye on maybe.

[LongQ]

flat-planedCrank said:

Wouldn't worry unduly - if you are mega-worried convert to DNG or save your edited shots as TIFs?

Tiffs maybe.

The DNG standard isn't really standard any more than RAWs are but the guidelines for Standard are set and modified over time by Adobe. So again if you are happy with a single point of control for development and are quite happy to work with whatever that source makes available from an entirely dominant market position then that's fine. It should certainly make choosing applications easier.

The potential problem of converting to DNG or saving as Tiff is, in the short term, an increase in storage requirements (and of course the presumption that you can, here and now, get as much out of the file as you will ever get - but then being concerned about that is probably a bit nerdy.)

Now none of this is of particular concern to me, other than as an academic exercise, so long as some sudden sequence of events does not change the game overnight. However it might be of greater concern to others at some point as their back catalogue grows. I would also guess that a few people may not be totally keen on the idea that their new camera a coupde of years from now will only have available whatever supported software the manufacturer might supply or Adobe's latest offering(s). Nothing else unless shooting JPG.

[RobDickinson]

From my own pov canon & adobe would have to vanish off the face of the earth taking all their converters with them and all the other ones too.

If things have got that bad I'm likely not worrying about my snaps so much.

[Kermit power]

gangzoom said:

Interesting read but I suspect most people wouldn't care. I use to shoot in RAW but not that I'm more confident with getting the shot right first time have switched to JPEG. I am now in the process of converting 13k RAW files into JPEGs simply because the RAW files were taking up so much hard drive space. I know RAW give the best quality, but unless your making money out of your photos I doubt the small amount of quality lost in the conversion matter??

I'm a little puzzled by your thinking. I could certainly understand you deleting the RAW files after processing, or even shooting RAW + JPG then deleting the RAW file unprocessed once you've checked you're happy with the JPG, but what do you do when you get that great, impromptu, unrepeatable shot, then find out the camera has made an arse of the white balance, for example?

[LongQ]

RobDickinson said:

From my own pov canon & adobe would have to vanish off the face of the earth taking all their converters with them and all the other ones too.

If things have got that bad I'm likely not worrying about my snaps so much.

True, although maybe they might also discover other reasons for not carrying legacy support forward, especially if it starts to get more complicated body by body, lens by lens.

And then one thinks of Kodak, Polaroid, maybe also Rollie, Contax and others in the camera world that are not the businesses they once were and a larger number of big brands in the computer world that have disappeared almost overnight. Food for thought, even if it is only a light snack at this point.

[gangzoom]

Just finished "processing" my lightroom library. 130gig of RAW files have now turned to 55gig of JPEG files...right time to now buy a SSD

Kermit power said:

I'm a little puzzled by your thinking. I could certainly understand you deleting the RAW files after processing, or even shooting RAW + JPG then deleting the RAW file unprocessed once you've checked you're happy with the JPG, but what do you do when you get that great, impromptu, unrepeatable shot, then find out the camera has made an arse of the white balance, for example?

The things I takes photos of are far too unimportant to worry about correcting the white balance in photos (use to do it) but now the only thing I ever do with my shots is the odd "crop". If I was a pro than ofcourse i would stick to RAW, if I could afford a 600gig Intel SSD I would stick with RAW. I accept the lower quailty of JPEG files and the lack of freedom you get...still am sure in the old days before digital, the pro photographers were taking good photos without having to "preview" everything on the camera

Edited by gangzoom on Friday 5th October 10:23

[14-7]

gangzoom said:

The things I takes photos of are far too unimportant to worry about correcting the white balance in photos (use to do it) but now the only thing I ever do with my shots is the odd "crop". If I was a pro than ofcourse i would stick to RAW, if I could afford a 600gig Intel SSD I would stick with RAW. I accept the lower quailty of JPEG files and the lack of freedom you get...still am sure in the old days before digital, the pro photographers were taking good photos without having to "preview" everything on the camera

Edited by gangzoom on Friday 5th October 10:23

Not sure why you need an SSD to store photos on?

You may as well buy a few TB's of conventional HD's and store them on there whilst using the SSD for operating system and some programs.

[gangzoom]

14-7 said:

Not sure why you need an SSD to store photos on?

You may as well buy a few TB's of conventional HD's and store them on there whilst using the SSD for operating system and some programs.

SSD is mainly for the OS, and the fact my wifes MacBook AIr simply flies compared to my MacBookPro all because the Air has a SSD.

I've had a fair number of HDs failures in both my laptop, MacPro desktop, and external hard drive (4 in 2 years) so now have 2 bootable backups the laptops in the house plus a Apple Timemachine, the MacPro has an RAID setup and bootable backup. If i put my photos onto an external hard drive I'll need to buy another 2 hard drives to back up that drive (one of the hard drives that failed was a back up drive hence my paranoia of only having 1 back up drive) and than it starts all getting very confusing!!

I completely agree RAW files are much better than JPEG for quality, I have 13K shots worth of RAW files...but for the last 6 months I've been shooting in JPEG only and am now quite happy to forget RAW and stick to JPEG simply for the smaller file size. For me been able to reduce my Lightroom Library from 130gig to 57gig is worth the small loss in quality comparing RAW to JPEG, but for most people I suspect HD space is probably not a problem

Edited by gangzoom on Friday 5th October 12:18

[Kermit power]

gangzoom said:

I've had a fair number of HDs failures in both my laptop, MacPro desktop, and external hard drive (4 in 2 years) so now have 2 bootable backups the laptops in the house plus a Apple Timemachine, the MacPro has an RAID setup and bootable backup.

Wow. I hadn't realised Apple kit was so poor! That certainly does change the perspective a little.

I've had 2 hard drive failures in the last 5 years across 3 laptops, 1 desktop (with twin drives), 3 backups and 2 PVRs, so I'm not quite so paranoid about failure as it seems I'd need to be in your position.

[LongQ]

gangzoom said:

14-7 said:

Not sure why you need an SSD to store photos on?

You may as well buy a few TB's of conventional HD's and store them on there whilst using the SSD for operating system and some programs.

SSD is mainly for the OS, and the fact my wifes MacBook AIr simply flies compared to my MacBookPro all because the Air has a SSD.

I've had a fair number of HDs failures in both my laptop, MacPro desktop, and external hard drive (4 in 2 years) so now have 2 bootable backups of everything for the three computers in the house plus a Apple Timemachine for the laptops. If i put my photos onto an external hard drive I'll need to buy another 2 hard drives to back up that drive (one of the hard drives that failed was a back up drive hence my paranoia of only having 1 back up drive) and than it starts all getting very confusing!!

I completely agree RAW files are much better than JPEG for quality, I have 13K shots worth of RAW files...but for the last 6 months I've been shooting in JPEG only and am now quite happy to forget RAW and stick to JPEG simply for the smaller file size. For me been able to reduce my Lightroom Library from 130gig to 57gig is worth the small loss in quality comparing RAW to JPEG, but for most people I suspect HD space is probably not a problem

That's a very high rate of disk failure.

In 17 years of personal PC use (longer if I include earleir computers and machines at work) I have only ever had one hard drive failure. That was on an external bacjup drive that was 3 months old and had been fired up only 3 or 4 times. Although it was exchanged under looking back on it I suspect that a possible reason for failure was using an incorrect power supply, albeit one that fitted the power socket on the device.

I have had 2 drives that, as boot drives, became corrupt somewhere and somehow in the boot sector and would not boot a machine. No way could be found to fix the boot problem (economically) and they were each, by that time, fairly old and short on capacity, so the drives were replaced in the machines but still work(ed) fine running in a caddy (or as a secondary drive) with no data loss.

My understanding about SSD drives, such as it is, is that they have relatively low expected read/write lifecycles compared to hard drives so using one as a primary drive needs to be a conscious decision to go for speed over longevity in a heavily used computer. That's not necessarily a problem and is likely a good trade off if you need the performance but it does imply that having a well thought out disaster recovery process along with the backup (to recover lost licences and so on ) is a sensible idea.

Plan D, of course, is to use Cloud storage - something many of the big players would like us to do and in many ways, if we assume coninuity of service can be assured, it does make a lot of sense as a concept.

[CommanderJameson]

Kermit power said:

Wow. I hadn't realised Apple kit was so poor! That certainly does change the perspective a little.

I've had 2 hard drive failures in the last 5 years across 3 laptops, 1 desktop (with twin drives), 3 backups and 2 PVRs, so I'm not quite so paranoid about failure as it seems I'd need to be in your position.

I've had 1 hard disk failure in 12 years across 6 Macs and 5 PCs (not to mention the HP PA-RISC server I had for a while with 4 disks in it) totalling about a dozen disks, so there's your counterpoint.

Disks fail. Some people are just cursed and have them fail a lot. Other people don't.