Multi-threaded / Parallel version?

User discussion and information resource forum for Image products.

Multi-threaded / Parallel version?

Postby Dr Arst » Thu Sep 10, 2015 4:21 am

I want to move away from Acronis TIH to be able to backup a Win/Linux triple boot machine. IFW seems to be excellent, apart from the dismal compression speed. I've tried all the compression types from SpeedA->Standard->SizeC (boot ISO and Win host) and would really like to use SizeB. But (and this is a show-stopper for me) looking at the task manager, I can see IFW is running the compression in as a 'single-threaded' process, for all compression types. Acronis uses all 8 'cores'. At the moment I'm using Acronis for NTFS and Clonezilla (which uses all cores) for ext4. I would really like to use a single program to do everything in one go (and can mount images for single file extraction).

What's holding me back from using IFW is the way higher backup time:
Proc: i7-4810MQ (4 physical cores at 3Ghz throughout the process)
Disk: Total 300Gb NTFS partitions (out of 480Gb SSD - 200Mb/sec sustained) containing mostly apps and virtual machine images:
Dest: 1TB external 7200rpm USB3 disk at 90Mb/sec sustained throughput.
Acronis on High compression: 0:50 hrs approx (using 8 'cores').
IFW on Enhanced SizeB : 2:20 hrs approx.
Both produce 180Gb approx target file.

It takes a LOT longer if I try to backup everything on the laptop (linux ext4 partition + 1TB NTFS spinning disk containing games). Clonezilla is twice as fast approx on ext4 compared to IFW (with similar target sizes) at the moment.

I realise that in the past, multicore was not so much an issue since most machines were dual cores throttled by the backup target disk/network. These days personal systems have gigabit LAN and SSDs, so multi-threaded apps processing large amounts of data makes a huge difference for more recent set-ups.

So, my question is:
will there be a Version3 of IFW/IFL which uses all available processing power?

(If so, I would buy the whole shebang in a heartbeat ;) )
Dr Arst
Posts: 2
Joined: Thu Sep 10, 2015 3:39 am

Re: Multi-threaded / Parallel version?

Postby TeraByte Support » Thu Sep 10, 2015 12:58 pm

I'd suggest using enhanced speed A. it's already faster than the drives
(waiting for data reading/writing) already running multiple threads/cores.

The other compression methods don't save much space and take a lot longer
and more appropriate for slower devices or where pressed for space.
FWIW, the ABC on those only exist for historic reasons, only the A version
is worth trying. B/C don't add much of anything but take ALOT longer.

While it probably wouldn't have any affect on Enhanced Speed A, it may
be worth investigating again for the others. It was looked in to in the
past and I believe there was a downside that offset any gains on the other.

Also, ensure you're not having a IO issue with the drivers being used,
that can happen. Ensure you're using a drive letter/windows path in
IFW for max speed, also VSS will be faster. Same for IFL, if you use
the IFL file path, it will be faster than file (direct).
TeraByte Support
Posts: 2478
Joined: Thu May 05, 2011 3:37 pm

Re: Multi-threaded / Parallel version?

Postby Dr Arst » Fri Sep 11, 2015 2:53 am

Oh well, that's a pity. I ran IFW in Windows7x64 using VSS. Such an excellent product otherwise. If you want to see how multi-threading helps speed things up on the linux side on my box, just try this out for starters:
If I run the script from below I get:
Single-thread time: 100.061589463
Multi-thread time: 24.382164286

$ ls -l
-rw-r--r-- 1 root root 439261265 Sep 11 10:26 home-mt-20150911@102427.tar.bz2
-rw-r--r-- 1 root root 438895375 Sep 11 10:26 home-st-20150911@102427.tar.bz2

Granted bz2 is a bit archaic, but testing the other compression types give similar speed advantages when running as parallel threads.
Also, when backing up 1.5 Tb, a 20% extra reduction in size is VERY important to me. If it would take the same time to run SizeA in multi-threaded as it does SpeedA in single-thread and produce a 20% smaller file (on a 4-core) I want that! (I'm thumb-sucking here if you haven't guessed).

I strongly believe there is a valid case for a multi-threaded version of IFW/IFL. Source code of multi-threaded compression implementations are freely available for perusal .
I truly hope IFW/IFL takes this suggestion seriously. Apart from that it is a sterling product!
Arst (retired embedded developer).

stamp=`date +%Y%m%d@%H%M%S`

START=$(date +%s.%N)
sudo tar cjf $targetFolder/home-st-$stamp.tar.bz2 /home
END=$(date +%s.%N)
echo Single-thread time: $(echo "$END - $START" | bc)

START=$(date +%s.%N)
sudo tar cf $targetFolder/home-mt-$stamp.tar.bz2 --use-compress-program=pbzip2 /home
END=$(date +%s.%N)
echo Multi-thread time: $(echo "$END - $START" | bc)
Dr Arst
Posts: 2
Joined: Thu Sep 10, 2015 3:39 am

Re: Multi-threaded / Parallel version?

Postby TeraByte Support » Fri Sep 11, 2015 10:43 am

as mentioned it will be checked again (was looked in to years ago), there is
no problem adding it in general (fairly simple), but I recall there was
something (I know standard wouldn't support it without modification, the
speed versions would, just don't recall the size options, or it was related
to some type of thrashing).
TeraByte Support
Posts: 2478
Joined: Thu May 05, 2011 3:37 pm

Re: Multi-threaded / Parallel version?

Postby timg11 » Tue Jul 05, 2016 11:43 am

I was just looking at the CPU core usage while running IFW on my i7-6700K. Using IFW 2.99, it appears that one core is fully occupied, while the rest are idle.
I add my "vote" for a multi-threaded compression function for future versions of IFW and IFL.
Posts: 171
Joined: Sun Oct 02, 2011 9:31 am

Return to Image for DOS/Linux/Windows