That's a very good question and one that gets raised in the US all the time.
I've heard from numerous people "Ordnance is owned by the US Gov't" but no one has ever shown me where the law says that. AR 75-15 states that the Army is responsible for identification and remediation of ordnance recovered above the high water line and the Navy is responsible for items below it. Responsibility does not equate to ownership. One can take that even farther by questioning the issue of if they have any claim if there is no explosive hazard to the item.
The FBI recently published a guide to Civil War ordnance that states "Civil War Ordnance is the property of the United States Government" which in my mind says "Since you have it and it's ours, we can come take it when we want" and that is just not the case.
The classic argument I use with squads is "How can the US Gov't claim ownership to a WWI German hand grenade that I purchased from someone in Poland?". They can't.
Every squad that I encounter I tell them the same thing, if you are removing a piece of ordnance from someone's house or property because of a call you received, you better have a piece of paper that abandons that item to you or a warrant that allows you to seize it. Without one or the other you risk opening yourself up to a law suit.
Oh and that reminds me, if you are a collector do us and everyone else a favor...let those around you know who to call immediately if something bad happens to you. There is nothing worse than seeing someones collection get destroyed because someone in the family doesn't know that the items are inert, doesn't know who to call and turns to the local bomb squad for help. 95% of the time that scenario ends with most if not all of the items being destroyed in a field by a big counter charge.
The US government has, in the past released substantial amounts of ordnance to the public. Back in the 70s and 80s large quanties was sold through the Property Management Office, right along with sleeping bags and mess kits. One of the last ordnance type items I remember seeing sold was the old M1 chemical mine ((1 gallon gas can). They were sold in lots of 20 or more. I used to also pick up the mortar subcal kits, with barrel, projectiles, barrel inserts.
For years ordnance scrap was sold from many facilities to scrap dealers with no requirements. You could go into scrap yards and buy whatever was there for scrap prices, by the ton. This began to tighten up in the mid 1980s after some accidents, again after the death in Fontana, Californa in 1996.
In addition there have been many items built by the military industrial complex, to be put forth as proposals offered fro sale to the government. Once rejected (or even accepted) the proposal pieces are dealt with as the company sees fit. One recent example of this is the General Dynamics mini grenade. Reportedly only 500 were made, the US govt didn't accept them, now collectors are scrambling for them.
I had it explained to me one time by an old Criminal Investigations officer (CID) who said, so much has been released over the years, unless it violates a law by either being live (>20mm) or classified, we have to prove theft. If its inert, unclass and there has been no theft reported, there is no crime for us to pursue.
Obviously not everyone has to agree, but it would seem an uphill fight by law enforcement to prove otherwise.