How do I control permissions for people invited to a given portal?

You control these permissions from within that specific Portal

You can either click on an individuals name in the "Current Portal Users" list or click on "Manage Permissions" both options will open a panel where you can fine tune each person's permissions individually.