Ypou asked three questions. Lets start with this one:
Will PCI-e 2.0 work with either pci-e x8 and pci-e x4?
Yes, it will work. But you are mixing up two different things:
PCI express comes in several versions.
PCI-e version 1 supports up to 2.5 Gbit/sec per lane.
PCI-e version 2 supports up to 5.0 Gbit/sec per lane.
PCI-e version 3 supports up to 8 Gbit/sec per lane.
Each of those can use single lanes, or combine several of them. That is what the x8 and x4 stand for.
Does it matter if the PCI-e SATA expansion cards use SATA 6 Gb/s while the onboard uses SATA 3 Gb/s in a raid 10 setting?
No. Both 3.0 Gbit/sec and 6.0 Gbit/sec SATA connections are a lot faster then any rotating disk. (Best speed on spinning HDDs is about 200 MB/sec, while SATA II / SATA 3.0 allows for transfers up to 270 MB/sec. The only reason to buy anything with a higher speed is if you use a SSD, if you want to future proof your purchase, or is the faster version is cheaper.
If I am using the 6 onboard SATA ports, does it make sense to buy RAID cards to support the other 6 drives? Is it either using raid cards for all 12 of the SATA drives or none?
That depends on how you want to use it. If you use software RAID then it does not matter how you access the extra drives. If you want a hardware RAID card to manage several drives than all those drives need to be connected to that hardware RAID card.
For these, first consider how you want to use the drives. Read this post on our sister site SF and make a choice.
Then decide on software or hardware RAID. Software RAID tend to work well for most situations, but HW RAID might be a better choice when doing RAID 6 or RAID 60.
Only then decide if you want to buy a (16, or 12+ port) HW RAID card, or if you want another solution. Other solutions include plain host adaptors ('aka SATA/SAS cards'), or port multipliers.
(Since you were considering RAID 10, software RAID might be the cheapest solution.)