Picking up trash under water is not the same as picking up trash on land. The small piece OP collected was probably ok, but the damage you can do to the ecosystem by removing a bottle or other big chunks that have been there for years can be really bad. I hope all divers have that in mind when they explore the underwater world.
Edit: I don’t get the downvotes, I’m not making this up. It’s one of the first things you learn as a CMAS diver.
There’s just a different “quality” of trash. Something deeply embedded in the ecosystem will become part of it. If you plug a bottle or large piece of metal from the floor you will destroy some form of habitat. Floating plastic has to go of course, as well as strings or nets.