Yamada has tried to ask things in the past but she is usually interrupted by Mikawa. Even though we see the signs that she is Kase's first girlfriend, I think it is very reasonable that Yamada sees Kase as so popular and desirable that it is totally plausible someone else came before her. After all, when Yamada says that she has never dated anyone before, Kase doesn't say anything back to her about her also being her first.
Kase is very popular and gets awards and recognition for her talent. Yamada is very kind and great at gardening, but does not receive awards or recognition outside her supervisor and Kase. It makes total sense to me why Yamada's perception of her desirability ended up being like this. It also makes sense that Kase likes her and sees a lot in her.
This isn't the first time I've seen it referenced in a manga that being in a sports club elevates you in the social structure of a high school, and it works that way in plenty of other countries too. When Yamada goes to college and finds "her people" I expect her internal self worth to go up a lot because she is being judged on something she is good at.
As a side note, Inoue seems to put a lot of stock into the social hierarchy of the sports clubs, love rival or no. She's being a jerk to the poor little gardener.