[server] Help me understand file permissions in and out of my containers.

I'm learning as I go about LXD unprivileged containers in 16.04 and I have some questions.

I have a directory on my host that I want several containers to have write access to the directory.

Using LXD when i create an unprivileged container the root user of the that container shows on the host as uid:gid 100000:100000 and the default first user (ubuntu) is 101000:101000 so far this seems consistent, no matter how many containers i setup each has a root user with id's on the host of 100000:100000 and ubuntu user with id's on the host of 101000:101000

I've been playing around with permissions on the directory on the host (lets call it /vm/Downloads) to make it available for writing to the ubuntu user in the container. To compound the issue various processes installed on the container will have uid:gid different than the ubuntu user, and those processes I want to have write access as well. The processes do some reading, organizing, renaming processing etc on the files and I want them all to have read/write access to the files that processes from other containers may create.

My files are shared from the host to each container in this way:
One way i got it to work is make the permissions on the directory very open (on the host)
This is the easiest way, but I don't know this is the best way, or what I want. I'm worried about having this folders permissions wide open like this.
Another option would be to use NFS and do it over the network, controlling permissions via NFS. I'm hoping to keep the access to these files as fast as possible and would rather not send this all over the network unless that was the only way.
I THINK the right way is to configure groups on the host that include the uid's of the accounts in the containers and give the proper permissions to that group. This is what I would love to figure out how to do, but when i tried I couldn't add uid 100000 to a group on the host... because it doesn't exist.

Am I on the right track with this? Is there a way to add the mapped uid's to groups on the host?

I'm newish to this unprivileged container business, so if i'm way off track please let me know. :)

Thanks for any help anyone can provide.