Copy and Paste between Ubuntu desktop to Ubuntu Server running on VirtualBox

I set the clipboard to bidirectional in my VM settings running Ubuntu Server 14.04. I know you can use Ctrl + Shift + V to paste and copy is just right click on the command I want and copy. But when I try copying the command I want from my desktop Ctrl + Shift + V does not work.

Is there something I am missing or this is not something that can be done?