Does an operating system itself take up resources (aside from the space it is installed on)?
Yes.
I just finished a course on operating systems and the definition of an operating system is still unclear to me.
On a low-level, an operating system is no different than any other program; ultimately, it is nothing more than a bunch of CPU instructions. The only difference is that the instructions of an OS do something (more or less) different than those of a program.
The best way to describe an OS is as a meta-program. That is, instead of doing something, the program makes it possible for other programs to do something.
Does any operating system itself take up resources such as CPU and memory?
Yes, of course.
For example the scheduling algorithm must take some processing power to compare tasks to see which goes first and whatever data structure the tasks are held in takes up space.
Exactly. The OS’s functions are just piles of CPU instructions, and like functions in a user-program, they take up space and must run on the CPU.
Also the act of knowing what to do with virtual memory must take calculations which take up resources, right?
Naturally.
Hypothetically speaking, if a computer with one program and no operating system was running the program, the program would run to completion faster than on the same computer but with an operating system. Is that true?
Slightly.
Remember that CPUs are very fast and can perform a lot instructions in a small amount of time. Moreover, operating systems have been designed and tweaked extensively to be as optimal as possible, so they use as little CPU and memory as they can (at least theoretically). As such, the algorithms they use for things like scheduling, memory management, task-switching, hardware driving, etc. use relatively little resources. You can see this in a task-manager; when there are no programs running and the OS is trimmed down to the minimum, then there will be very little memory in use and the CPU will run at “0%”. (Again, we’re talking theoretically; Windows for example has been “bloating” as of late, so it may not apply to that anymore.)