A version of the permutation argument in the philosophy of mathematics leads to the thesis that mathematical terms, contrary to appearances, are not genuine singular terms referring to individual objects; they are purely schematic or variables. By postulating 'ante-rem structures', the ante-rem structuralist aims to defuse the permutation argument and retain the referentiality of mathematical terms. This paper presents two semantic problems for the ante-rem view: (1) ante-rem structures are themselves subject to the permutation argument; (2) the ante-rem structuralist fails to explain reference in a way that makes her account different to, and privileged over, that of her eliminativist rivals. Both problems undercut the motivation behind ante-rem structuralism.