I don't think Hufflepuff is wishy-washy or anything. I actually think they are almost as misunderstood as Slytherins though. I feel like Ravenclaw had more large characters. While Hufflepuff does have the 'numbers' you list, the majority of their actions seem to be in book 4 where they united against Harry.