Your speed is quite similar to speeds I've seen on other 100Mbps networks with SSH. The transfer speed seems to depend alot on OS you are using, and not that much on the available CPU speed. On one network all transfers between PCs running Debian or Ubuntu Linux had 3MB/s speeds, but when I was using SCP on a Mac OS X server, I always got over 6MB/s. Also file transfers from a Xen server (which is running CentOS variant I think) also gave me higher speeds. Only time I've gotten faster transfer speeds with Debian Linux and scp are when I've copied data from one Xen virtual machine to another on the same Xen server, then I would get 20+MB/s speeds.
With Samba shares, you usually can get to 50Mbps or so, and If you want speed easily, I would recommend setting up a FTP server, with that I usually can get 90Mbps easily, even from a Windows machine.