Recording the network traffic data takes some time, which results in a slowdown of the page load. Though normally this slowdown isn't that big.
Note that the 20ms you measured only refers to the time needed to generate the page script. It doesn't take all the other requests into account involved into the page load, e.g. CSS, JavaScripts, images, etc.
I suggest you compare Firebug's Net panel output with the one of the Firefox DevTools' Network panel. If they differ much, disable the Console and Script panel and check if that makes any difference. If not, you may file a bug against Firebug.