You'd have to specify the term hostname more closely. If you are looking for the name as it is known to the DNS server that you don't want to use - there is no way of knowing except for actually asking that DNS server.
If you are simply guessing, nmap can be of good use. For example, if there is a smbd running on that server, the output of nmap -A
might look like this:
Host script results: | smb-security-mode: | Account that was used for smb scripts: guest | User-level authentication | SMB Security: Challenge/response passwords supported |_ Message signing disabled (dangerous, but default) |_smbv2-enabled: Server doesn't support SMBv2 protocol |_nbstat: NetBIOS name: FOOBAR, NetBIOS user: <unknown>, NetBIOS MAC: <unknown> | smb-os-discovery: | OS: Unix (Samba 3.5.6) | Computer name: foobar | Domain name: lan | FQDN: foobar.lan | NetBIOS computer name: |_ System time: 2013-03-14 17:02:27 UTC+1
Other services might give hints about the name of the machine as well.