Stop using pulseaudio because it breaks on its own all the time.
The sound card works fine once I added myself to the audio group and rebooted. I removed pulseaudio completely as well.
Not the best solution, but I figure it's just easier to use Alsa which works.