hazardous and toxic materials

When the stuff is in the ground, it is generally in minute quantities and is chemically bonded with other materials making it inert. For example, you don't find pure aluminum in the ground - you find bauxite. Gold is chemically extracted with various acids. Lead is found primarily in galena, mixed with zinc, silver, and copper. And so on.

Putting refined pure metals back in the ground is not natural, and it would take thousands, if not tens of thousands of years for it to approach the "safe" levels found in nature. If you put it back in the ground the same way you took it from the ground (as bauxite, galena, etc.) it would be just as safe as if you had never taken it out in the first place. That's not what's happening though.