Firstly, your captive fish may be carrying certain diseases that it has immunity to, but the native fish population does not. Entire native fish populations can be wiped out if you introduce a novel disease into their environment. The fish you release may seem perfectly healthy, but that doesn't mean it isn't carrying something that it is immune to. This really does happen. For example, there is a pathogen called Chytridiomycosis (Chytrid for short) that is decimating wild amphibian species across the world, and there is significant evidence suggesting that it was originally spread into the United States by captive African Clawed Frogs that were used for research.
As if that wasn't enough of a reason not to release captive fish, there's more! Certain captive fish, when released into a favorable natural environment, can become extremely invasive and out-compete other fish for resources. This can also result in native populations of fish being wiped out, which has a ripple effect on the entire ecosystem of the area.
This issue is a danger to our hobby of fish-keeping, and especially goldfish-keeping. Did you know that some states and countries have been creating more restrictive laws about moving fish between states and between countries because they're concerned about these fish either spreading disease to native fish or becoming invasive in native waters? If you release captive fish into the wild, you are endangering not only the native ecosystem, but also the hobby we love so dearly! So please, think twice before releasing that fish. If you don't want it anymore, bring it to your local fish store for rehoming. Every pet store I've ever been to has accepted fish from people and sometimes will even give you store credit in return for the fish.