So the fix was really simple: Drop Virtual Box. Instead I am using VMWare and literally everything was solved. I have no idea why Virtual box didn't like my network card but it didn't. Here are my new synthetic benchmarks using iPerf 3:
Cat5e LAN
- Desktop --> Server : 30 Mbits/sec
- Server --> Desktop: 767 Mbits/sec
10GBe Fiber LAN
- Desktop --> Server : 1.24 Gbits/sec
- Server --> Desktop: 1.96 Mbits/sec