George, Another country can't board your U.S. flagged vessel in international waters. If you're in their territorial waters you're subject to their jurisdiction and they can board you as much as their law allows. They can also obtain a statement of no objection to board you from your flag state, which often is granted if they have the equivalent of probable cause that you're smuggling or polluting or some other major violation. Once your flag state gives permission, which the U.S. does, they could board you. Finally, you need to be sure you really are U.S. flagged. You can't just trot out a literal piece of cloth with the stars and stripes on it and instantly be U.S. flagged. The country can check with your flag country and if they deny you're one of theirs you're stateless and fair game to be boarded by any country anywhere.
All that said, it sounds like you were in he territorial seas of another country and that is what gave them authorization to board you.

I just would like to clarify something you wrote about Coast Guard boardings. As a US-flagged and documented sailboat, the USCG can board me anywhere, but can other natioinalities board my US-flagged vessel when it is in international waters? I am just asking as I am sailing in Europe, I paid VAT to the Dutch and had a Post-Construction Assessment survey and the boat is CE marked. My US flag attracts various law enforcement agencies in Europe like a red flag to a bull. I am not really concerned as I have all my documents up to date and am not engaging in any illegal activities.

The French Douane once when looking through my ship's papers and saw my purchase price, they told me that they didn't believe that I paid so little for the boat and that they could reassess the boat and charge me additional VAT. I told them they didn't see the boat when I bought it and I have been refitting her for 12 years...