Fords drove voters away from Conservatives more than Duffy: Poll