I've tried a game session without music too now, and indeed, the atmosphere got much better! At least when out in the wilderness and in caves (that music gives me the creeps
). You hear all sounds like birds, trees swaying, water running, anilmals etc. much better.
It also gets much more exciting, since you can't hear when enemies have spotted you. I was walking merrily along a path collecting plants when BAM! a bear jumped on me from behind. I almost wet my pants!
I still like to keep music on when in the cities though.