Think again. While youll compile kernels and encode Lame files ever so slightly faster, youll have no palpable performance increase for average tasks, and even take a hit with some games:

http://www.phoronix.com/scan.php?page=article&item=616&num=2

I have a feeling that the repositories are a bit sparer for x64 as well.

/me sez with a 1.0ghz boxen and a PIII lappie