WPA2 Enterprise is based on parts of 802.11i that are based on 802.1X. 802.1X does NOT require a RADIUS server, but that's how it's commonly done for legacy reasons.
The RADIUS server's role is only at the beginning of the connection, but it does do one little thing more than you mentioned. As part of the authentication mechanism, keying material is securely generated on the RADIUS server (and the same keying material is also generated on the WPA2 client). After the RADIUS server tells the AP to accept that connection request, the RADIUS server sends that keying material in a RADIUS "key" message (they reused a RADIUS MPPE-KEY message/attribute that Microsoft had pioneered) to the AP, so the AP knows what per-user-per-session keys (including the Pairwise Temporal Key or PTK) to use for that session. That ends the RADIUS server's involvement.
You're absolutely right that it really doesn't take much server horsepower to run a RADIUS server. Just like a DHCP server or a DNS server for a small network or domain, you really don't need "server class" hardware to run it on. Probably any little low-power embedded network box will do. There are a lot of protocols in modern networking where the "server" end doesn't require much horsepower by todays standards. Just because you hear the term "server", don't assume it requires heavy-duty server hardware.
Backstory
You see, RADIUS was originally a way to move authentication out of your dial-up modem PPP servers, and into a centralized server. That's why it stands for "Remote Authentication Dial-In User Service" (it should be "Dial-In User Remote Authentication Service", but DIURAS doesn't sound as good as RADIUS). When PPP started being used for DSL authentication (PPPoE, PPPoA) and VPN authentication (PPTP and L2TP-over-IPSec are both "PPP inside an encrypted tunnel"), it was natural to continue to use the same RADIUS servers for centralized authentication for all your enterprise's "Remote Access Servers".PPP's original authentication mechanisms were lacking, and took a lot of standards-body involvement to create new ones, so eventually, the Extensible Authentication Protocol (EAP) was created to be a auth-type plug-in system for PPP-like authentication. Naturally, RADIUS servers and PPP clients were the first places that needed to support EAP. You could, of course, have your dial-in modem/PPP server, or your VPN server, or your PPPoE/PPPoA (really, L2TP PPP) server, or whatever, implement EAP locally, but by now, RADIUS was so widely deployed that it was mostly RADIUS servers that implemented it.
Eventually someone wanted a way to require authentication whenever someone plugs into an unguarded Ethernet port in the lobby or a conference room, so "EAP over LANs" was created for this. "EAPoL" as it was known, was standardized as 802.1X. 802.1X was later applied to 802.11 networks in IEEE 802.11i. And the Wi-Fi Alliance created an interoperability certification /branding/marketing program around 802.11i, and called it Wi-Fi Protected Access 2 (WPA2).
So, while your 802.11 AP itself could fulfill the entire 802.1X (WPA2-Enterprise) "Authenticator" role all by itself (without the help of a RADIUS server), it's just not commonly done. In fact, in some APs that are able to do 802.1X standalone, they actually built and open source RADIUS server into their firmware, and do 802.1X authentication via RADIUS via loopback, because it's easier to hook it up that way rather than trying to implement your own EAP authenticator code, or copy of the code out of some open source RADIUS server software and try to directly integrate it into your AP firmware's 802.11-related daemons.
Given that backstory, and depending on how old your proposed RADIUS server is, the important question is whether it implements the EAP type(s) you want to use for authentication on your network. PEAP? TTLS?
Also, note that RADIUS traditionally uses a "Shared Secret" known to the RADIUS client (the RADIUS client is the "Network Access Server": the AP in this case, or a VPN or PPP server or other "Remote Access Server" in other cases) and the RADIUS server, to both authenticate the RADIUS client and server to each other, and to encrypt their communication. Most RADIUS servers let you specify different Shared Secrets for each AP, based on the IP address of the AP. So an attacker on your network would have to be able to take over that IP address, and guess that shared secret, in order to have the RADIUS server talk to it. If the attacker wasn't on the network yet, the attacker would only be able to try to send specially-crafted/corrupted EAP messages that the AP would relay via RADIUS to the RADIUS server. If the security issue you're concerned about could be exploited via malformed EAP messages, then you could still have a problem.