Think again. While youll compile kernels and encode Lame files ever so slightly faster, youll have no palpable performance increase for average tasks, and even take a hit with some games:
http://www.phoronix.com/scan.php?page=article&item=616&num=2
I have a feeling that the repositories are a bit sparer for x64 as well.
/me sez with a 1.0ghz boxen and a PIII lappie :P