This is a little hint for you guys who hate getting caught by the guards and then having all your beloved swag swiped by those pesky legionnaires.
Once you've payed the fine or done your sentence, just go back into the prison barracks or dungeon, wherever the jailor is (the stolen goods are always in an evidence chest next to him). Just ask to visit a prisoner, and then he'll grumble and walk towards the door to let you into the jails.
While he's doing that, you can just sneak behind him, pick the very easy lock while he's opening the door and get your goods back. Then you can just walk out with a smug look on your face

while the jailor's still opening the door for you. Problem solved.
Of course this doesn't work when there's two jailors there.