So, you are saying that the order has to be by appearance, but within a citation group, by year, but it doesn’t work when they are being used again, later in the document? There is no way to do both. The sort applies to all the citations, not just those in a group. If they are in order, then used later, they should stay as ordered by appearance. Anything else is going to apply to the whole reference order in the document, and not just within the group of citations…
So, if you ask that the citations be ordered by Year, then if citations are used a second time in a new group, they will be listed by year and not by number - and you will get [7, 1a, 5] if 7 was published before 1a and both were published before ref 5.
I don’t think endnote has the ability to accomplish what you want. The only way I see you getting this output, would be to ensure you insert the citations the first time, in a group, is to put them in order in the group.