MDPs out in the cold