Ubuntu in VM, install kernel LTS 3.2 at installation and stay there?

I want to use Ubuntu or Xubuntu in a Virtual Machine.
I just istalled Ubuntu 12.04 twice: One with downloading updates, the other one without downloading updates while installation.
Both were on Kernel 3.8.
The long term kernel is 3.2 (or so), how do I install kernel 3.2 and stay there, without installing 3.8?