Thanks for all the interesting replies! Given the response, I decided to make a whole community around this, hope you’ll consider joining!
If you liked this thread, you might like: !likethismaylike@lemm.ee
Remember, if you’re on a different instance you may have to search the url first: https://lemm.ee/c/likethismaylike
If an unreliable shifting house would work, House of Leaves by Mark Danielewski. The writing is very much love it or hate it for a lot of people, but the idea fits.
Edit: Oh! And House of Windows by John Langan. No relation despite the similar titles.
I did love House of Leaves!