Absolutely not! All you need is something to dissolve the glue and a non marring tool to push it off. The plastic spatula idea is good, I prefer a cheap plastic 1.5" putty knife from Home Depot (less than a buck). Spray some brake cleaner on it and pop it off, then some more brake cleaner and a rag to remove the residue.
Don't do anything stupid like trying to use a razor blade or metal scraper.
You can use a heat gun set on low or a hair dryer to soften the adhesive on the weights. Then use a nylon scraper to remove them. WD40 will finish cleaning the adhesive off.