When you're using the cable, all they have to go on is public records like the ones you can find on IP2Location. When you're using the wireless network, they can pick up a lot of information from the air. It's not just, as the other answer is implying, a matter of a public access point already being known.
The geolocation service also takes into account other networks in your vicinity (that you are not connected to) based on their SSID, MAC address and strength. If both you and your neighbor are using the geolocation service, and both of you see networks x, y and z, Google's service can draw the conclusion that you are located physically close to each other. It does this with a lot of data collected from users, which enables them to correlate your location rather well. I'm sure Google Street View vans are helping collect data from access points for this purpose as well.
Google explains how the service works here. However this page now has a less detailed description than it used to, but there's a copy of the old text in this StackOverflow answer.
And consequently, if you are, for privacy reasons, not comfortable with the service sending a list of access points in your neighborhood to Google, you should obviously not use this service.