My native language isn't non-Alemannic German, but I've still discerned that sometimes an attempt to invoke a Swiss-German word or phrase can be misunderstood that way. It hasn't happened in a long time, but there were a few instances where the reaction was awkward at best. Lately, especially among (Swiss) friends and relatives, there seems to be a (sometimes pleasantly surprised) appreciation of my efforts at linguistic assimilation (though I remain domiciled in Texas for the time being).

At a dinner in Winterthur last October, our host identified a particularly tasty dessert cheese as having come from Graubünden, and I won a round of chuckles with "Aso, es Bündnerchaas, oder?"