Largely, so long as you don't cause harm on purpose, you'll be fine. The Nine Lords are protectors of those weaker first and foremost. The most dangerous might be those in the Marsh. Many there are outcasts. You'll find many who resemble poisonous plants or the dead, for example, and they'll be just wanting to live in peace. Rafflesimon is very protective of those under her wing to the point of being distrustful of outsiders.