There are several different ways that you can make this selection neater. If you had a tablet to work with it would improve your precision and you would be able to use the pen tool to replicate your shape completely from scratch, making the result flawless.
Since you are specifically asking about making a selection, here's a way to improve it.
Select > Color Range
Above selection has the artifacts that you describe in your question.
- Select > Color Range + Fuzziness (200)
Above selection is much smoother. When making your color range selection you can drag the fuzziness to max (or where you think is appropriate).
- Select > Color Range + Fuzziness
- Select > Modify > Feather
Above selection where I added some feather (3px) was way too much and didn't turn out quite well.
- Select > Color Range + Fuzziness
- Select > Modify > Smooth
Above selection is quite neat, I simply modified the selection to add 2 pixles of smoothness. It will bend the edges a bit but the overall result seems much nicer. If you are concerned about that you can change that to 1px smoothness and also add 1 px feather which will give you a similar result.
This way you can tamper with your selection before you remove the red background - change some settings until you feel happy with it.