Вычислить md5sum каждой части потока в 1 МБ из канала

split from coreutils (the default on most Linux distributions) has a --filter option which you can use:

7z e -so image.7z | split -b 1000000 --filter=md5sum 

Something simple like this Perl script probably would suffice.

$amount = 1_000_000; while (read(STDIN, $buffer, $amount) > 0) { open MD5, "|md5"; print MD5 $buffer; close MD5; } 

Put this in foo.pl and invoke it as perl foo.pl at the end of your pipeline.

It seems to me that you're looking for this kind of a tool.

From the Readme file of BigSync:

Bigsync is a tool to incrementally backup a single large file to a slow destination (think network media or a cheap NAS). The most common cases for bigsync are disk images, virtual OSes, encrypted volumes and raw devices.

Bigsync will read the source file in chunks calculating checksums for each one. It will compare them with previously stored values for the destination file and overwrite changed chunks if checksums differ.

This way we minimize the access to a slow target media which is the whole point of bigsync's existence.

It was easy to write small 1MB hasher using rhash tools (librhash library). There is simple perl script which creates checksums of each 1MB part of standard input stream. It needs Crypt::Rhash bindings from cpan:

$ cpan (cpan) install Crypt::Rhash $ cat rhash1M.pl #!/usr/bin/perl # Compute md5 and sha1 sum of every 1 MB part of stream use strict; use local::lib; use Crypt::Rhash; my ($buf, $len, $i); my $r=Crypt::Rhash->new(RHASH_MD5|RHASH_SHA1); # we can add more hashes, like RHASH_TIGER etc binmode STDIN; $i=0; while($len= read STDIN,$buf,1024*1024){ print "$i+$len: \t"; # print offset $r->update($buf); print "md5:",$r->hash(RHASH_MD5), " sha1:", $r->hash(RHASH_SHA1),"\n"; $r->reset(); # reset hash calculator $i+=$len; } 

This public domain script will output decimal offset, then +, then block size, then md5 and sha1 sums of input.

For example 2 MB of zeroes has sums:

$ dd if=/dev/zero of=zerofile bs=1M count=2 $ ./rhash1M.pl < zerofile 0+1048576: md5:b6d81b360a5672d80c27430f39153e2c sha1:3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3 1048576+1048576: md5:b6d81b360a5672d80c27430f39153e2c sha1:3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3 

rsync works like this, computing a checksum to see if there are differences in parts of the file before sending anything.

I'm not sure how well it would function with files this large, although I have never heard of it having any file size limitation.

Pipe the output to this Python 2 script, for example 7z e -so image.7z | python md5sum.py:

import sys, hashlib CHUNK_SIZE = 1000 * 1000 for chunk in iter(lambda: sys.stdin.read(CHUNK_SIZE), ''): print hashlib.new('md5', chunk).hexdigest()