Насколько легко взломать следующую защиту от копирования?

How easy it would be to break this setup (to decrypt the root filesystem and make it boot from any sd card)?

How hard it is to "break" your setup depends on the number of bits of entropy in whatever method you're using to sign/encrypt the filesystem itself (as this determines the total number of unique combinations that can be used to brute-force the password).

What are the most weakest parts?

Without a doubt, using a predefined CID as a password, as well as using a custom pseudo-random number generation function.

The CID of an SD card is only supposed to be read-only, but it's not uncommon to find non-compliant flash memory devices in this day and age. Some people have even demonstrated the ability to overwrite the CID with certain SD cards. This would make it easier to brute-force the password, especially if one is just emulating an SD card after cloning yours (which is something else you might want to consider).

Finally, using any kind of pseudo-random number generator already has some intrinsic flaws, precisely because it's not random - there is a reason it's called pseudo-random. It might be better to use a pre-made encrypted bootloader (like TrueCrypt or LUKS, which both work on the Raspberry Pi) and avoid having to make any manual kernel modifications.

How easy is to decompile the kernel and find those custom encrypting functions?

It's very difficult to decompile anything. Conversely, de-assembly of a compiled application is often trivial, and there are many tools which can be used to assist with reverse engineering assembly back into another higher-level language. If an attacker has access even to a compiled kernel, analyzing something like a pseudo-random number generator is probably trivial unless the code is obfuscated on purpose.

TL,DR: Don't re-invent the wheel when it comes to encryption and security, stick with the tried and true. There are several full-disk encryption options that are already available and have been demonstrated to work just fine on the Raspberry Pi. I would avoid using the CID of the SD card as a kind of "password" - even if it cannot be changed, there are ways to spoof this value.

Copy protection is already included in the SD card specification as CPRM.

Someone skilled wouldn't have much trouble cracking this. It'd be relatively easy to boot the SD card under an emulator, and then just read the keys out of RAM. Then they post a version without the copy protection to the Pirate Bay (etc.), and that's that.

Alternatively, use the emulator to inject shellcode into the running emulated system. Then use the running system to copy the decrypted rootfs off (or read the keys using dmsetup table --showkeys, etc.)

A quick search turns up the existence of Raspberry Pi emulators, so part of the work has already been done.

You've got another problem, in particular this:

Kernel is modified to have the same encrypting function, which works as decryptor.

Anyone you distribute this to is entitled to the kernel source code, under the terms of the GPL. So you wouldn't need to disassemble it, you could just use diff to find the extra function.

(Not that finding it through disassembly would be that hard, as you can e.g., check vs. a stock kernel)

I'm not completely familiar with the Raspberry Pi boot code, but if you can reflash the bootloader with an embedded crypto key (that is then passed to the kernel), that'd at least not be on the SD card, so it'd foil an attempt to get it to boot in an emulator.