The most common interpretation is that Kasen is an oni posing as a hermit for some reason.
It makes sense, but it seems too easy. I'm hoping there's more to it than that.
I know this is so three years ago, but I am a bit late to the party. One of my favorite moments while reading this series was the point where all the pieces fell into place and I realized Kasen was an Oni. They actually foreshadow it long before the actual big reveal during the drinking contest in chapter 32.
For me the Ahaa! moment was at the start of chapter 21 when they first mention the practice of bean throwing. Kasen's reaction was just so odd and then all of a sudden the broken chain on her wrist, her godly tolerance to booze, the way she always avoids the other oni, the title of the manga and the fact that she never appears without the two buns in her hair suddenly made sense.
I felt like an idiot for not noticing in sooner. It's hilarious the way the author hid it in plain sight starting from the very first chapter and how it doesn't reach an actual reveal until 32 chapters in. After you realize she's an Oni bent on taking over the shrine all of sudden all sorts of weird little things she said and did since the start of the story are recontextualized in a totally new light and you realize she's really a 4 dimensional chess master whose been subtly playing people the whole time.
I wonder how many other fun little Easter eggs and secrets are hiding in plain site like this. I am sure this series is packed full of them.