Well I normally always carry my Daedric artifacts, showing my devotion to the Daedra. I for sure keep Molag Bal's mace and Clavicus Vile's masque beause they are the ones I am mainly devoted to.
I also always find the alive butterfly in a jar in some shack. It's my pet. Since it's beautiful, it resembles what my character wish she could be inside. She knows inside she's ugly, hateful, angry and immoral, but when she sees this beautiful butterfly, she wish she could be good inside.
I normally always keep my DB armor (even if I'm not wearing it) as this resembles my strongest connection to a faction, the Dark Brotherhood, especially once I become Listener.
I also keep Astrid's Blade of Woe as this represents my power to take life, and to take power through murder. She betrayed me greatly, and I killed her, so this weapon symbolizes that I won and you can't get away with betrayal.
I also keep Arentino's loom (not sure how to spell his name, but it's the little boy in the beginning of DB) because it symbolizes how the innocence of children always gets corrupted, and this is how my character felt as a child, full of hatred.
Last, but not least, I typically carry a diamond or sapphire around always. The gems remind my Character of when she used to be poor as a child. She had nothing and she was constantly starving, left to rot in the mud with the pigs. Her stealing/finding/selling gems is her way of making money mainly, and it helps her remember that she is no longer in her childhood, that she made something of herself.
Oh and Azura's star, because it represents her hunger to kill. Since the star consumes endless souls, she feels this is the only way to truly kill someone all the way, by stealing their soul and using it.