Probably 10 or more years ago now, I heard a talk by Larry Weaner. At the time, I didn’t realize how influential it was going to become in my gardening style.
What he said was that it’s important to manage invasive plants and then to let the land show you what wants to be there.
I am not sure that I will ever be “done” managing invasive plants, particularly with the number of birds on my property. Fine.
But the land certainly has shown me what wants to be here and it’s goldenrod. But it’s also lots of other things as well.
This is a plant called boneset (eupatorium perfoliatum). I have quite a bit of it. Pollinators love it because of the tiny, multi-part flowers. It’s native for me.
This huge patch of asters is just one of several types–all natives– that appeared here by chance and grow beautifully in my heavy wet clay. They’re great for pollinators and go nicely with the goldenrod in this bed.
This aster prefers more shade so it grows on the edge of our woods.
If I were weed-averse–or less likely to let something bloom to see if it is a weed or a wildflower–chances are I wouldn’t have half these plants. My garden and my pollinators would be poorer for it.
Gardendaze 
In town, I weeded meticulously. I sort of did the same in parts of the less refined garden higher in the mountains, but also found that I could promote a few natives that I really liked by just pulling out the larger specimens of competing species that were less desirable. I did that with some iris that were not very flashy, but looked better with other plants removed from within their colonies. (They were West Coast iris, but we know them as San Francisco iris – which is not the bearded iris ‘San Francisco’.)
It’s kind of amazing that nature will give you nice native plants if you just let it. Of course, as you say, it also gives you other plants. I have a lot of Virginia creeper. Lovely native with awesome fall color and great berries for the birds. Of course, great berries means that I have more Virginia creeper than I want–& I need to be vigilant about that.
And we won’t even talk about the invasive vines!
Karla