The cheapest house in the game is only 2000 Gold. You can buy it from the woman at the Imperial Census & Excise Office in the Market District. (Thats not the correct name for the building but I'ts close) First you have to raise her disposition to at least 60 for her to mention the house in conversation. You can do that using the Speechcraft mini-game or by bribing her. The house is located right outside the city in the Waterfront District.
Once you have bought the house, you can buy furnishings at the Three Brothers Trade Goods in the Market District. The only furnishings I have bought was the Storage Room kit. It adds a chest to the foot of the bed, a Dresser beside the fireplace and another chest to the left side of the fireplace.
Its pretty cozy and functional for a house that only costs 2000 Gold. Plus it is right there outside the Imperial City so the location is good.