MD5 checksum within IFL

[rustleg]

I want to suggest adding MD5 checksums to the log file in IFL. Maybe it's already in the log file, in which case apologies - I have never used the log file and when I tried to find it I couldn't.

Now I see from another post here where the log file is and I'd have to mount a partition then write the file. So as another suggestion, it would be much simpler if the log file were to be just added to the folder that the .tbi file was being created in.

My reason for this request is that I copy .tbi files from a Windows machine to my home Linux machine for back up using a USB stick. A while ago I experienced file corruption due to a faulty stick which wasn't found until a restore was attempted, which was sadly too late. So until recently I have used MD5 checksums on the Windows machine to generate a small text file of checksums which I then re-check in the Linux machine to make sure the backed up copy is not corrupted. This worked fine in Windows 7 using the certutil program. However now using Windows 10, I have not been able to get any file checksum program to read the .tbi file. Windows 10 refuses to give permission. I have tried various googled solutions including a Terabyte article without success. Windows 10 is such a pain with file permissions, I thought it would be much easier to get the checksum from IFL while the archive is being created, if it is possible. Linux already has a md5sum utility so I would think this ought not to be too difficult to add.

[TeraByte Support]

you could use validation feature. or TBScript has the ability to calc hashes of files. Log files are saved to the OS drive so direct written won't have option to save there, if you use a command line and file (os), you can give a custom name.

[rustleg]

Thanks for responding.

I don't see how the validation feature will help me. Ok it validates that the tbi file is a faithful copy of the partition (although there are some warnings somewhere about this not being 100% reliable). However, as I explained, I need a checksum to validate the file has not changed after copying it to another PC via USB stick which can be unreliable. If my backup copy is not a 100% faithful copy, the image is useless, and you only find this when you try to restore it when you really need it.

I didn't understand when you said "so direct written won't have option to save there" - please explain.

I still don't see why you can't save the log file to the same medium as the tbi file is saved. Mounting a partition, finding and copying a log file isn't easy, so as I see it, adding a couple of options to your list of tick options would be valuable:
1. save log file to same medium as tbi file
2. include MD5 checksum in log file

One alternative I've come across to solve my issue is to boot a Linux live CD after doing the backups and use this to create MD5s. I used Knoppix which worked quite well as mounting a partition is a gui operation. You only need the command line to calculate the MD5s. But of course if this were part of IFL all this would be unnecessary.

You may think that few users would value this. Obviously it's your call to decide what enhances your product. This is just my opinion.

[TAC109]

Creating a MD5 checksum would involve reading the complete image file after
copying. So why not just run a verification on the copied image file with
IFL? It should take a similar amount of time and provide the certainty you
require.

[TeraByte Support]

rustleg wrote:
>(although there are some warnings somewhere about this not being 100% reliable).

It's reliable. What you're thinking of is probably https://www.terabyteunlimited.com/kb/article.php?id=350

Byte-For-Byte can only be done during the backup itself.

[rustleg]

TAC109 wrote:
> Creating a MD5 checksum would involve reading the complete image file
> after
> copying. So why not just run a verification on the copied image file with
> IFL? It should take a similar amount of time and provide the certainty you
> require.

I can't tell if you've understood my issue. I *am* asking for a verification on the copied image file, but in a way I can export something so I can re-run a verification on another PC where the backed up image file is stored. From Support's original reply I could use scripting but I've never got into scripting with TB products and I expect most others also don't use it. My original post was a suggestion that this would enhance the IFL product for the average IFL user. It' s up to TBU whether they think it is a useful feature for their users. Hopefully they welcome suggestions from users even if they don't agree with them.

In case you've misunderstood I'll try to explain once more. The problem is that I have a tbi file (and .1, .2, etc) on a backup medium somewhere. It got there via copying from the file(s) created by IFL on the original source PC. The backup medium is elsewhere on another PC or a server, etc. How do I know that the process(es) which copied the orginal .tbi file from the original PC to the backup medium have created a faithful exact copy?

The point is that this has happened to me. I tried to use a backup file to re-create a partition and IFL (or maybe IFD) said it was corrupt. I use both validation methods when creating, validate and validate-byte-for-byte so the original file on the original PC was almost certainly 100% correct. I strongly suspect the USB stick used for copying across machines was faulty. Having a MD5 checksum created by IFL would be a way of validating the backup copy once it had been stored on the the backup medium. It's similar to using MD5 checksums to validate files downloaded from the internet which is a common practice.

As far as I'm concerned this is not a "nice to have" feature but an essential requirement of an image backup. I want certainty. If you can't be sure the image is recoverable you take the risk of the practice of imaging letting you down, so you'd then need to reinstall the OS on the partition from scratch along with all its software and settings which can take days. In such a scenario, using IFL would have been entirely useless.

Actually I suspect that the IFL validate option uses some sort of checksum, but I may be wrong.

[DrTeeth]

On Thu, 19 Jul 2018 02:37:56 EDT, just as I was about to take a herb,
rustleg disturbed my reverie and wrote:

>I can't tell if you've understood my issue.

I think this would be a very useful addition to the whole "image
for..." suite.

While validation is useful, it does not seem to confirm that it is a
faithful copy of what is on the hard drive. I can validate any old
copy and it will pass even though my hard drive's contents have of
course changed.

Would it be possible for you to use a checksum making and validating
utility in Windows or other OS? I have come across a brilliant one for
Windows. I had never bothered with this before finding the utility. If
you want detail, just ask here.
--
Cheers,

DrT

"If you want to find out what is wrong
with democracy, spend five minutes with
the average voter." - Winston Churchill

[TAC109]

rustleg wrote:
> TAC109 wrote:
> > Creating a MD5 checksum would involve reading the complete image file
> > after
> > copying. So why not just run a verification on the copied image file with
> > IFL? It should take a similar amount of time and provide the certainty you
> > require.
>
> I can't tell if you've understood my issue. I *am* asking for a verification on the
> copied image file, but in a way I can export something so I can re-run a verification
> on another PC where the backed up image file is stored. From Support's original reply
> I could use scripting but I've never got into scripting with TB products and I expect
> most others also don't use it. My original post was a suggestion that this would
> enhance the IFL product for the average IFL user. It' s up to TBU whether they think
> it is a useful feature for their users. Hopefully they welcome suggestions from users
> even if they don't agree with them.
>
> In case you've misunderstood I'll try to explain once more. The problem is that I
> have a tbi file (and .1, .2, etc) on a backup medium somewhere. It got there via
> copying from the file(s) created by IFL on the original source PC. The backup medium
> is elsewhere on another PC or a server, etc. How do I know that the process(es) which
> copied the orginal .tbi file from the original PC to the backup medium have created a
> faithful exact copy?

You run IFL, Validate mode, on the COPIED image file(s) I.e the ones on the 'other PC or server, etc'. This will check that the COPIED image file(s) are internally consistent, and complete. That's the point I'm trying to make.

[rustleg]

DrTeeth wrote:
>
> Would it be possible for you to use a checksum making and validating
> utility in Windows or other OS? I have come across a brilliant one for
> Windows. I had never bothered with this before finding the utility. If
> you want detail, just ask here.
> --
It should be possible. However one of the points in my original post mentioned that I have problems running any validation program in Windows 10. It comes back with permission denied. I have bent over backwards trying to change the security settings of the folder but still can't get any application to read these tbi files. I have successfully used certutil in Windows 7 and also another checksum utility, but Windows 10's security model is so finicky. I think it might be because the files were created outside Windows so presumably don't have the permission bits that Windows 10 expects (and presumably it's not possible for Terabyte to fix these when creating the file otherwise they'd have done it already. They have an article about modifying the folder security settings but when I tried to follow it, it didn't work).

I have successfully booted a Linux live CD and created MD5 checksums, but it's another fiddly operation to do involving mounting partitions and running terminal commands. Whereas IFL could simply have an option to add the MD5's to some extra file on the same medium as the .tbi files were being stored on. These extra files could then be copied along with the image files. IFL is a Linux system and Linux has a simple utility md5sum which calculates MD5 checksums. So I figured it wouldn't be much of a stretch to add it to IFL's features. Once Terabyte had developed this, the users could just tick a box and get something to validate the backup with.

[rustleg]

TAC109 wrote:
>
> You run IFL, Validate mode, on the COPIED image file(s) I.e the ones on the 'other PC
> or server, etc'. This will check that the COPIED image file(s) are internally
> consistent, and complete. That's the point I'm trying to make.

I don't think there's such a facility in IFL. Hopefully support will confirm this.