You generally don't need to stack speed on one-handed, because with a few cards/pieces of gear, you'll be able to swing so fast anyways that you'll only really get higher outputs of damage if you can button mash extremely fast in the first place.
A little bit helps, but if you have too much than you can actually button mash, then the speed is pretty much wasted by that point. Generally it's better to go damage, with a little bit of speed from talents, cards, or the weapon your using itself.
Basically, get attack speed as much as you can comfortable actually handle, and then dump the rest into attack damage.