Remember that during PS and even throughout CoS (possibly the entire time before reincarnation) the Malfoys were officially neutral: while publicly disdaining Dumbledore, Lucius would even tell Draco off in CoS for being to mean to the Boy-Who-Lived in public. Presumably, aligning with the Malfoys would be more anti-Dumbledore (Harry can DEFINITELY fit that bill) than pro-Voldemort!

William//Slytherin