The reason traceroute won't work on local network is, because it all goes via 2. layer, not 3. network one, so there are no hops, ping however will work as the machine will answer.
The firewall rules on the other network router are port specific, so they will work different as your setting. Your firewall seems to accept only established connections and I guess the rule #3 is dropping everything that tries to go inside your network except of established connection. Everything on local network should work just fine.
Therefore, I would try to troubleshoot problems with OSX server on server level not network level, that comes if you can ping it's interface. You have probably not setup web server on it properly.