Goldsmith rejects calls from 'fusionists and defeatists'