The prime minister has accused his opponents of launching a smear campaign