Mongo Loc: Western New York

Lots of times we hear people say, raw is uncompressed and jpeg is compressed.

Well not exactly.

Raw image files are not usually compressed, but it depends upon the camera system design and architecture. There are cameras which compress raw data. Perhaps not many consumer cameras, but there are many cameras which do compress raw image data. The most common compression, up until about 2000 was run length encoding (RLE). It could be cheaply implemented in hardware, ran at near maximum data transfer speeds, and did not require much extra power. RLE compression took advantage of identical pixel values in adjacent pixels.

By the same token, jpg files can also have compression shut off. Sometimes it is done for power and processing time considerations.

Also, both types of files can be treated as data files on computers, and have lossless compression applied. This often happens with utilities like zip, or xz.

A source data file which is already compressed will tend to have little advantage in getting compressed again, and often will take more space in the recompressed version. That actually forms a poorman's method of compression verification. Compressing a pre-compressed image file, and getting poor or even negative compression tends to show (but not prove) that information in the file was already compressed.

So if you keep raw files on your computer, and want to make a backup of them on a blu-ray or DVD, you can copy them to a scratch folder, and compress them with a suitable compression tool. I use xz -9e some of the time. The server I do the compression on is very fast, has lots of memory and is idle much of the time. And it helps me pack every blu-ray or DVD disc with as much information as possible. Because I don't expect to access the back up often, if ever, I am not worried about decompression processing time. I just want to reduce media costs.

For linux users, after the images are in a directory,

find . nice xz -9e {} \;

compresses each file. When that is done, and your working directory or folder are sized for your media,

mkisofs -dvd-compat -r -J -joliet-long -l -Z /dev/dvd .

will copy all the files in the directory, onto the media, in this example dvd, in an ISO file. That allows one to index to the desired file later, without reading the whole volume if an individual file is needed. Be careful to observe filename size restrictions of your operating system and of ISO9660 to make sure your files are both unique and can be accessed for later recovery.

Those running Windows can use zip or other similar utilities to compress and/or wrap groups of images together.

No form of backup is perfect. Most use harddrives, which are pretty reliable, but eventually fail. Optical media is also used, but can have failures. USB flash memory is not terribly reliable, long term. Having multiple backups is not a bad idea and having them compressed reduces the media needed.

cameraf4 Loc: Delaware

My D850 gives me choices on how to save NEF (Raw) files, either uncompressed or in one of 2 methods of compression.

Mongo Loc: Western New York

The D800 does as well. One of the methods is lossy. There is also a lossless compression. The lossless is said to reduce file size 20-40%. The lossy is said to provide 35 to 55% compression.

Retina Loc: Near Charleston,SC

It may help to clarify that compression can mean different things with respect to computer files. Lossless compression usually means a more efficient way to store (usually) repetitive data. A crude analogy is how a U.S. dime is a compressed version of ten U.S. pennies. The intent is for unpacking, or at least being able to retrieve all the original data. Altogether different is processing a RAW file into a JPG file. This is a file format conversion where the result is not intended to save all the data in the original file, and the resulting file is usually smaller by default. I never heard of JPG that contains all the data from the RAW or that a RAW file can be recreated from a JPG which is identical in content to the original RAW. Someone can correct me, please.

wireloose

Sony also compresses the Raw (.arw) files, at least on the A7Rii,iii etc..accidentally turned that option off and found myself with ~85mb files to process- so it cuts the size by half

camerapapi Loc: Miami, Fl.

First thing first, RAW is not a file but data. RAW data is a more appropriate way to address it. Nikon cameras can compress RAW data, mine to 12 or 14 bits. 14 bits is supposed to be uncompressed but when the data goes into an editor it gets 16 bits of information.
I do not believe I have a means of decompressing JPEG files in my cameras and to be honest I have never heard of it till today.

Retina I will not be the one to correct what is correct. A JPEG file DOES NOT contain all the information of a RAW data and as you know we loose the wide color space, if using a wide color space during editing and 16 bits of information when we convert the RAW data to a JPEG.
The big advantage of the RAW data is the amount of information it has and its flexibility during editing.

Retina Loc: Near Charleston,SC

Sorry, I use the term file loosely to refer to the stored form of the RAW data from a camera that an operating system handles, like those with name extensions such as NEF and NRW (going from memory.) I was just trying to make the point that types of compression can easily be confused since we are dealing with formats, data, files, etc. Even my money analogy falls apart when I go to Canada and my U.S. dollar is compressed by vendors to seventy five cents. Thanksfully, ATMs unpack my funds in a lossless manner into Canadian format.

f8lee Loc: New Mexico

JPEG, by its nature, is a compressed data storage format. You may choose the level of compression (when you save you see the scale for what level of compression you want - higher quality means less compression used). JPEG files are image files, in that each pixel is assigned a specific RGB value.

Raw files (raw does not get capitalized as it is not an acronym) are not image files, because each photo site reading from the chip has only one of the three primary colors - Ror G or B. Until it is demosaiced, that is, processed by software that decides what RGB value should be assigned for pixel 20,20 based on the readings from a number of adjacent sites, it dose not constitute an image. And yes, when you shoot "raw only" and chimp the images on the back of your camera, what you are seeing is the JPEG image generated by the software built into the camera's computer. Having to do those calculations for the 24M or 48M or whatever photosites on the chip for every shot is a lot of math, but that's what goes on. Same is true for your cell phone, and virtually every other digital camera made (not the Leica Monochrom nor the Foveon chips).

Anyway, in computerese the term compression when applied to any kind of digital file means using methods to reduce the space required to store it - so some cameras do offer that process. But just like compressing does not change the words in a Microsoft Word .doc file that is a 1,000 page book, this kind of compression does not imact the underlying data.

I know Nikon (and perhaps others) offer what they call "lossy" compression for raw files, but cannot say how they achieve that without losing the individual photosite data readouts.

blackest Loc: Ireland

Raw is mostly lossless in terms of Data information. Sony has a bit of a reputation of lossy compression of data. I know the Pentax K1 and K1 mark II differ in that the mark II applies noise reduction to raw files (it also has a massively expanded iso range). I do not appreciate the compression in the mark II. I would rather be selective in deciding where to apply it. There is no option to turn it off. If It could be turned off I might have chosen to buy the mark II (there was an option to retrofit the 'accelerator' to the mark1) If it's raw it should be raw and not some mock raw. I really don't think the expanded iso range is worth anything. It's still going to be pretty horrible.

selmslie Loc: Fernandina Beach, FL, USA

This can be explained with very simple math. Think about the distribution of values within each file.

An 8-bit image can display a range from black to white using 256 values. For nine steps that's about 28 values per step if they are evenly distributed - 28x9=252 is just about the whole range.

The values in a 14-bit raw file are not evenly distributed. The brightest step has 8,192 possible values and the step nine stops darker only 16. Generating 28 highlight values from 8,192 is clearly overkill but generating 28 values from 16 involves some compromising. The middle tone has 512 raw values that can easily create the 28 8-bit steps needed within each stop.

So we can compress some of the information in the brightest raw steps without seeing any visible impact on the JPEG. In other words, lossy compression does no visible harm. Not appreciating this, many photographers opt for what they feel is a safer option - lossless compression or no compression at all. They pay a penalty with larger files.

But an 8-bit image has already lost most of what it's going to lose in the first step, even if it is an 8-bit lossless TIFF. If it drops a few crumbs of quality with subsequent edits and saves it's not going to be as much as it lost initially.

Mongo Loc: Western New York

The jpeg standard is published, and a co-worker chaired the JPEG 2000 standards committee. The standard actually uses wavelet methods, and one can think of wavelets as a mini-FFT. The method does not arithmetically break grey levels down, but rather looks at the spatial distribution in a heirarchical manner. It is possible to set the algorithm for no compression, in which case the resultant image can be restored to the original image. Mappings of compressed images can take in to consideration the color responsiveness and spatial characteristics of the human visual system. This adds to the ability of jpeg images being able to have reasonable perceptual quality even though the are compress, or even highly compressed.

Not sure what was being explained above, but a 14 bit image would have the ability to represent 16,384 values.

JPEG 2000 can handle thousands of terapixels, with up to 38 bits/sample (pixel), with or without tiling, and has other features. It was designed with medical imaging, geospatial imaging, pre-press and more conventional photography in mind.

JohnR Loc: The Gates of Hell

Phew... Wow..... Cripes ...... None of the above posts are of any value in taking and editing photos. One can still get more out of a raw file than a jpeg without knowing any of this tech stuff. Happy snapping one and all

selmslie Loc: Fernandina Beach, FL, USA

Yes, that means 8,192 in the brightest stop, half of all the possible values. The next stop has 4,096 values, half of what's left. Then 2,048 1,024, and so forth.

It's the darker values that are getting short changed with only 16 values nine stops from the top then 32 values one stop brighter followed by 64, 128 and so forth.

There is an embarrassment of riches when it comes to the number of values in the brightest stops - a lot more values that anyone would ever need even if they were trying to save the image as a 16-bit TIFF. If you compress only the brightest values carefully you won't lose anything visible and you can save some space.

But the important zones for color and tonality are closer to middle gray. There are more than enough raw value levels there in a 14-bit file.

Retina Loc: Near Charleston,SC

The question touched on compressed data. Since compression is a term that has different applications in digital photography and personal computers, my reply seemed appropriate enough at the time. I offer my apologies to the members. Next time I will discuss taking and editing photos only.

bwana Loc: Bergen, Alberta, Canada

The different types of compression should be mentioned. Compression can be off, lossless or various degrees of lossy. Off and lossless yield the same image quality; only difference is image size. Lossy does not retain full image quality!

bwa