^It can also suck a lot for foreigners.. Especially women, who are considered as "easy" by average japanese men and routinely harassed with little to no reaction from authorities. The mere fact of being a foreigner who stands out is "asking for it".
In my experience when I visited Japan over the last few years, I have found that most of the time I am a foreigner first; and only then, as a far distant second, a woman. Admittedly, I am just north of 5.9 (1.76), and while my build is quite slender by European standards, I guess may seem a bit intimidating to your average Japanese person. Basically never had any issues with harassment in Japan, but realise that my experience is far from the typical daily interaction of Japanese women in particular.
The HRW report is quite interesting, and raises the question of how a highly conforming society can adapt to accommodate minorities, which are by definition alien to its enforced harmony. The bitter irony is that Japan has historically had a relatively open relationship with LGBT matters, but the more they opened themselves to the west, the more restrictive their perspective has become (India is another quite sad example in this space).
Having spent some time in Korea as well, I cannot help but suddenly see Japan as downright progressive in comparison. :p