I'm reading the manga and Shirou is always used for comedy, even his friend Issei shows interest in him, the "joke" is that basically almost everyone would be part of his harem, but only Rin and Luvia seens to have real interest in him, Maus, the first manga is indeed not very good, it's basically a parody of mahou shoujo, but in 2wei things start to get serious and have a real plot and in 3rei it becomes even more serious like a Type-Moon main story inteast of just another useless spin-off, but still have some comedy, not going to give spoiler but it seens they're connecting it with Fate/Extra, so it might be a good idea to play it before reading 3rei.
In short, if you can endure the first manga things get better in 2wei and 3rei, but I'm not sure if the anime will get better like the manga.