On page 32 it says that both Momiji and Kouyo are written the same?
Both are written as 紅葉, yes.
How does that work, both in terms of how are the same characters making such different sounds
Pretty much all kanji have several readings, with some being similar (like 下 being ge/ke/ka/ga) and some being very different (like 下 being shita, moto, the o in oriru, the kuda in kudaru (from kudasai fame) and the sa from sagaru). 紅葉 as kouyou uses the onyomi readings of each kanji, whereas momiji is just a reading this compound has that doesn't even have anything to do with the regular readings of the separate characters. Why? Because Japan.
how would someone who's reading it know which set to use?
You don't.
Well, in this case it's a name, so you know because they tell you. Usually context also tells you which reading is correct for a character. Sometimes the reading is written seperately in so-called furigana, small characters next to/above the kanji. And sometimes you have to guess.
