Suppose you have a "go to the head of the line" card for the grocery store. You go to the store, fill your cart, go to the checkout counters, and find there's no one in line. Does your card help you get checked out faster? Nope.
Priorities don't affect processing speed, in that a higher priority process doesn't get to run faster or even to use more CPU time... not if it's the only thing that wants to use the CPU.
To really talk about this we need to mention threads. Processes don't "run" in Windows. Threads, which are parts of processes, are what run. (Although if a process has only one thread, the distinction is pretty fuzzy from the outside.)
Each thread is always in one of several scheduling states. The most commonly seen states are: Waiting (*nix calls this "blocked"; in both OSs this means waiting for I/O or similar, uses no CPU time and doesn't want any); Ready (wants to use CPU time but no CPUs are available for it right now); and Running. Only Running threads consume CPU time; ie if a process has no Running threads, it will be seen to use zero% CPU time in tools like Task Manager.
A thread can only run on one core (or, if hyperthreading is enabled, "logical processor") at a time, so a process can only use as many CPU cores (or LPs) as it has threads that want to run at the moment. (The same statement can be made of the system as a whole.)
Most threads on most systems spend most of their time in the Wait state. Exceptions would be the "working" threads of things like video or 3d rendering, gaming, etc.
The setting you make in Task Manager actually establishes the "base priority" for all of the threads in the process. The actual, or "current," priority of the thread may be higher (but never lower than the base). More on that in a moment. Scheduling decisions ("who gets to run, and on what CPU") are always done using the thread's current priority. Priority is only meaningful for Ready and Running threads (i.e. it isn't meaningful for Waiting threads).
Windows uses a pre-emptive scheduling algorithm. If only one thread in the system wants to use CPU time, then it doesn't matter in the slightest what its priority is; it gets 100% of the CPU. It is not as if the scheduler "holds back" a portion of the CPU's capability when a low priority thread is running, just in case something of higher priority comes along.
If two threads want to use a CPU, and they're of the same priority, then they are scheduled via what is called "time-slicing" and over time, each gets about 50% of the CPU's time. Whereas if they're of different priorities then the higher-priority thread gets 100% and the lower- gets nothing.
(In practice it will not quite get nothing, because it will experience a periodic "starvation avoidance priority boost" that may give it a few tens of msec every 4 or 5 seconds or so. But this is not really an exception to "higher priority wins", because it's done by adjusting the priority of the starved thread.)
If you have more than one CPU core, things get more interesting, and priorities in general have less effect. Suppose you have two threads that want to run. And suppose you have two or more CPU cores that aren't doing anything else of equal or higher priority than those threads. Then your two threads will each get 100% of a core, regardless of their respective priorities.
(Two people show up at the supermarket, and there are two checkers free. One of the customers has a "go to the head of the line" card. Doesn't matter.)
tl;dr version (so far): Priorities are not about "who gets what proportion of CPU time", but rather "who gets to run first".
I'm not going to get much into hyperthreading here, except to say that Windows treats each of the two "logical processors" in a core pretty much the same way it would treat a core if HT was turned off. i.e. they're treated as "real" CPUs, with this exception: Windows will try very hard to not use more than one LP in a core at a time. i.e. you don't normally start seeing both LPs in a core being used until you have more than number-of-cores threads trying to run all at the same time. This is because the two "logical processors" don't give you anything like twice the performance of a single non-hyperthreaded core.
About the "base priority": Windows will adjust ("boost" and "decay") the current priority of threads based on what they've done recently. Threads that recently completed I/O operations will normally be a notch or two above the base; UI threads (threads that are running a window) will often be considerably higher; CPU-bound threads will usually be at their base. The purpose of this is to maintain responsiveness in the program's UI, and also to keep IO requests flowing to things like disks.
A program (process) can also change the base priority of each of its threads, within a range determined by the process's priority (the thing you set in Task Manager). But the vast majority of programs don't bother. (More of them should.)
There are other things going on. Because of priority boost/decay, and because multiprocessing systems (either multicore, or hyperthreaded, or both) are so common these days, and because there's always stuff running in background in Windows (but, we hope, not using much CPU time), and because of the effects of both hard and soft "affinity", it's difficult to run test cases and get the exact results that would be predicted here. But this should give you close to a correct picture.
In conclusion...
It is reasonable to leave most things at "Normal". If you don't, you can easily end up starving something that you'd really like to have working (even though you might not know that it exists), like the OS's disk cache flushing functions. Indeed, many of the OS's processes will be at other than Normal, and they should be left wherever Windows put them.
A reasonable case for using Task Manager for fiddling with priorities is if you have some CPU-hogging task (like video or 3D rendering) and it's slowing down your use of the system while it's running. The right thing is, believe it or not, to lower its priority by a notch or two. It will happily use all of the CPU cycles nothing else wants but will stay out of the way of your interactive use of the system. It may take a little longer to get its work done, but it will get its work done with minimal interference to your interactive use of other programs. If you don't like that tradeoff, don't do it! But set it to a high priority in an attempt "to make it go faster" and it may hang your entire UI until it's finished.
Never set anything to the so-called Realtime priority class.
n.b.: Any video player app should be opting into the "Multimedia class scheduling" feature in Vista and later. This will automatically give it up to 80% of a CPU, reckoned over relatively short intervals. If you can't get glitch-free playback with that something is very wrong.
For more details, see the chapters on threads and scheduling in Windows Internals 6th Edition by Solomon, Russinovich, and Ionescu.
See also my answer here for info on how process and thread priorities are set, and the meaning of the "Priority" column in Task Manager.