i think you're right, the umbrellas actually have been swapped by the end. glasses puts her in and we clearly see it is the only white-handled one in there and then immediately after that the dude takes it. the next one we see is itsuki leaving and taking a white handled one, which she presumably put there on her arrival (off screen). it seems like the guy only knows itsuki, so presumably he took glasses' umbrella, thinking he recognised it as itsuki's, and at the end gives it to itsuki telling her it's hers.
anyway, it was a fun lil story