Oh, the wings of history.
In Windows history, this dates back to the original PC-DOS which ran on the IBM PC of 1981. (Its lineage, in turn, can be traced back to CP/M of the second half of the 1970s.)
The IBM PC originally did not support a hard disk drive. Besides hard disks being freakishly expensive, the PC did not come with a strong enough power supply to support a hard disk. You could retrofit a HDD, which required replacing the power supply, adding a disk controller card, and installing the hard disk itself, but very few people did so.
As I recall, PC-DOS 1.x didn't even support hard disks. This was introduced in PC-DOS 2.0, which also added support for directories in the file system. In PC-DOS 1.x, the way you categorized content was by using different floppy disks for your love letters, business spreadsheets, recipes collection, and so on. Most likely this remained a common way of categorizing content well after PC-DOS 2.0, until hard disks became a common PC accessory some time in the second half of the 1980s.
The original IBM PC did however natively and readily support up to two floppy disk drives. Some systems even supported up to four floppy disk drives, in which case the extra floppy drives were labelled C: and D: on early versions of MS-DOS, with the first hard disk's first partition referred to as E:, but this was an uncommon configuration. Early BIOSes did at the very least allow for up to four floppy disk drives, by using two bits (possible values: 0, 1, 2, 3) to return the number of floppy disk drives less one (thus 1, 2, 3 or 4) installed in the BIOS get equipment list function; this was soon expanded by the get disk type function which could in principle support up to 128 floppy disk drives, but required a full 256 calls for a full storage device enumeration.
On DOS, the first floppy disk remained referred to as A: and the second floppy disk was referred to as B:.
Since one or two floppy disk drives was a very common configuration, it made sense to use the next available letter for the first partition of the first hard disk, rather than simply the first available drive letter. Thus, C: was born. However, early versions of MS-DOS could push the first hard disk partition to a drive letter later than C: if additional floppy disk drives were installed. (For a more in-depth discussion on how the assignment was done, see my question How did MS-DOS assign drive letters in the case of more than two floppy disk drives? on the Retrocomputing SE.) As DOS could not boot from a non-primary partition (extended partitions was yet another later invention) and the IBM BIOS had no real ability to boot from anything but the first readable drive, most likely a decision was made to refer to the first partition of the first hard disk as C: even if there was no A: or B:. Having a fixed designation for that which was common across most systems was useful, as it allowed software to offer sensible defaults and documentation to offer sensible examples without lots of "ifs" and "buts". Eventually, multiple-floppy-disk-drive configurations fell out of favor.
Back in those days, also note that hard disks were small enough that it wasn't generally reasonable to subpartition them for anything but specialist workloads. A hard disk of 20 megabytes was big, and 40 MB was utterly huge. IIRC, it wasn't until PC-DOS 3.1 that FAT16 was introduced, which allowed for larger than 32 MB partitions, and formatting even a 32 MB partition using FAT12 was already hugely wasteful as it resulted in a minimum allocation size of 8 KB, a good 0.024% of the entire partition capacity, and allowed for storing a maximum of 4,068 files on the partition.
These days, we very rarely have any floppy disk drives at all installed, so A: and B: have fallen out of general use. However, the decision to refer to the first partition on the first hard disk as C: still stands, and thus on Windows we talk about C: instead of A:.
Note that C: referred to the first partition of the first physical hard disk drive; not to the drive itself. DOS offered no user-friendly way of addressing the physical drive itself, but the BIOS did, calling it drive number 128+n where n is the 0-indexed number of the drive. On the other hand, the BIOS had no concept of partitions. (Floppy drives are numbered n, again where n is 0-indexed. Hence, in a common configuration at the time, A: is drive 0, B: is drive 1, and C: is the first partition on drive 128. In a scheme where physical drives are addressed using an 8-bit number, which the designers of the IBM PC chose in 1980-1981, having 128 as the boundary means setting or clearing a single bit, which makes it trivial to test in software including the BIOS and OS.)
And of course, other operating systems have different ways of addressing partitions and drives. Linux commonly uses /dev/sdxy where x is a letter and y is a number, to address a specific partition (and the mount point of the file system on that partition for accessing the files on it); other Unix-like systems have similar but different device naming schemes.