who lived in castles