Applying smart groups to a group set?

Hi all — I have been trying to set up a few smart groups that would function within a group set (in other words, only search against a subset of my references); however, the smart groups seem to only be searching against the full “all references” data. Can someone please point me in the right direction? How can I limit the smart group search area to only the references that are part of a smaller group set? Thanks in advance.