[He can't hide a small look of disappointment at the suspicion that statement brings. He likes his privacy.

A monastery...

It's possible you're at Orabonne Monastery on Pinnacle Rocks. It's a short distance away from Lindblum. I woke there myself...
And it has been where I was staying.