Even if you dont think you can lip read, youd be amazed how much being able to see a face changes things - conversation in posh restaurants is very muttery with the odd muffled laugh generally.
Down a phone youre clearer and louder to overcome not being able to see a face, which means everyone else can hear you clearly and succinctly too. And of course they can only hear half the conversation so it doesnt flow and can be very distracting.
EDIT: I couldnt have a long-ish conversation on a phone when around a lot of captive people anyway - I dont want them knowing my business fir a start and I know how irritating it is for me when someones talking and laughing loudly down a phone.
Oh, and I like to pace about when on my mobile.