MD5 checksum within IFL

[Bob Coleman]

rustleg wrote:
How do I know that the process(es) which
> copied the orginal .tbi file from the original PC to the backup medium have created a
> faithful exact copy?

Could the process(es) be extended to compare the original file and the copy after copying?

[Bob Coleman]

TAC109 wrote:

> You run IFL, Validate mode, on the COPIED image file(s) I.e the ones on the 'other PC
> or server, etc'. This will check that the COPIED image file(s) are internally
> consistent, and complete. That's the point I'm trying to make.

I do more or less that though using IFW rather that IFL and am assuming that, as you say, the copy is internally consistent and complete. Does that mean it's an exact copy of the original? I don't know. It does seem pretty unlikely that anything could have happened to the file in copying and it still be internally consistent and complete. Is it impossible? I don't know.

[TAC109]

rustleg wrote:
> TAC109 wrote:
> >
> > You run IFL, Validate mode, on the COPIED image file(s) I.e the ones on the
> 'other PC
> > or server, etc'. This will check that the COPIED image file(s) are internally
> > consistent, and complete. That's the point I'm trying to make.
>
> I don't think there's such a facility in IFL. Hopefully support will confirm this.

IFL, IFD, and IFW all have the facility to validate an image, whether it consist of one file or many files. For example, if you start any of the GUI's (IFL, IFD, or IFW) it is right there on the front window, the 'Validate' option.

Here's a quote from one of the User Manuals:-

"Validating Backups with Image for Windows -

You can use Image for Windows to validate backups at the time you create them and also before you restore a backup. You also can validate a backup at any time using the instructions provided below.

When you validate a backup, Image for Windows performs internal consistency checks on the backup file(s), helping to ensure that the backup will be reliable if you need to restore from it.

Tip: You should consider validating your backup images using your Image for DOS, Image for Linux, or TBWinPE/RE recovery media to make sure that you can see your backups while working in the recovery environment.

..... "

There is similar text in the other two manuals.

I hope this clarifies the point I was trying to make.

[rustleg]

TAC109 wrote:
> ...
> IFL, IFD, and IFW all have the facility to validate an image, whether it consist of
> one file or many files. For example, if you start any of the GUI's (IFL, IFD, or IFW)
> it is right there on the front window, the 'Validate' option.
>
> ...
This validation is possible because at the time of validating the partition you are imaging is available and the file that IFL/IFW/IFD has created is also available. So the file can be read and checked back against the partition. This protects against things like disc errors.

When the image file has been copied on to another PC, the original partition isn't available so there's no way to compare the contents of the file against the partition.

The only way to perform a validation in this instance is for a checksum (also called message digest or hash) such as MD5 to be calculated from the validated image file and saved on a text file. MD5 checksums have the property that any change to a byte of the source file results in a change in the checksum. The chance of accidentally finding a different source file with the same MD5 checksum is millions to one. MD5 isn't perfect for security purposes but is widely used for file integrity checking - see Wikipedia.

If you have a MD5 checksum you don't need the original partition to be available to check the integrity of the file. You can store the file on a different computer and calculate its MD5. If you get the same answer as the MD5 originally calculated when the file was on the original computer you can be sure the file hasn't been changed in the process of copying to the other computer. This is not just a theoretical requirement. USB sticks can be unreliable and can introduce errors. It has happened to me.

Of course its possible that Terabyte can create an image, hash it, then put both image and hash into one .tbi file. This would allow them to remotely validate a tbi file. But I doubt they have done this or they would have said so and also provided a feature in their programs to verify a .tbi file without the original partition being there.

[Bob Coleman]

The below is all correct, I think. You and TAC109 are both correct, I think.

The programs contain two types of validate: validate byte for byte and just plain validate.

The points you making are, I think, correct for validate byte for byte especially the point that the original partition(s) which were imaged must be available.

However, as TAC109 says, the programs can perform a validate (not byte for byte) on any image including a copied image. This doesn't involve or require the original partition(s) that were imaged. It doesn't verify that the image being validated is a accurate copy of anything, but it does verify as TAC109 said that the image being validated is internally consistent and complete.

You say "The chance of accidentally finding a different source file with the same MD5 checksum is millions to one". No argument there.

TAC109 and I believe that the chance of an image which is validated to be "internally consistent and complete" also being incorrectly copied and/or corrupt is vanishingly small. I won't attempt to debate whether the chance of this is millions to one.

We are just trying to point out to you that the image programs can perform a type of validate without the original partition(s) that were imaged being available. If you realize that but believe that type of validate is not good enough, fine.



rustleg wrote:
> TAC109 wrote:
> > ...
> > IFL, IFD, and IFW all have the facility to validate an image, whether it consist
> of
> > one file or many files. For example, if you start any of the GUI's (IFL, IFD, or
> IFW)
> > it is right there on the front window, the 'Validate' option.
> >
> > ...
> This validation is possible because at the time of validating the partition you are
> imaging is available and the file that IFL/IFW/IFD has created is also available. So
> the file can be read and checked back against the partition. This protects against
> things like disc errors.
>
> When the image file has been copied on to another PC, the original partition isn't
> available so there's no way to compare the contents of the file against the
> partition.
>
> The only way to perform a validation in this instance is for a checksum (also called
> message digest or hash) such as MD5 to be calculated from the validated image file
> and saved on a text file. MD5 checksums have the property that any change to a byte
> of the source file results in a change in the checksum. The chance of accidentally
> finding a different source file with the same MD5 checksum is millions to one. MD5
> isn't perfect for security purposes but is widely used for file integrity checking -
> see Wikipedia.
>
> If you have a MD5 checksum you don't need the original partition to be available to
> check the integrity of the file. You can store the file on a different computer and
> calculate its MD5. If you get the same answer as the MD5 originally calculated when
> the file was on the original computer you can be sure the file hasn't been changed
> in the process of copying to the other computer. This is not just a theoretical
> requirement. USB sticks can be unreliable and can introduce errors. It has happened
> to me.
>
> Of course its possible that Terabyte can create an image, hash it, then put both
> image and hash into one .tbi file. This would allow them to remotely validate a tbi
> file. But I doubt they have done this or they would have said so and also provided a
> feature in their programs to verify a .tbi file without the original partition being
> there.

[rustleg]

Thanks Bob and apologies to TAC109. I was entirely unaware of the facility to validate an image file by doing internal consistency checks. I never really properly examined all the options in the program.

I think maybe Terabyte do add some form of checksum into their image files to allow this consistency check. It would be nice to know for sure that a file created with the validate byte-for-byte and later passing a consistency check will ensure the image is entirely reliable. I am inclined to agree with you Bob that it is most likely knowing how good TB's programs are.

So I withdraw my suggestion for MD5's to be added to outputs as this appears unnecessary now.

At least this has aired this valuable property and I have learnt something which I will find is very useful to me.

Many thanks to TAC109 and Bob for educating me.

[Bob Coleman]

I've been thinking more about this and I have to say that I don't know exactly what "internally consistent and complete" means. Could a byte in an image file be somehow changed and the image file still pass the "internally consistent and complete" check? I really don't know.