Over time you will fuck them up. Simply put the weight that is on the bar will actually only be supported by that weight that touches the floor. Putting aside technique bumper plates, a standard bumper plate is only designed to support its own weight, hence the 25kg 20kg 15kg 10kg and 5kg plates are all the same diameter.
You may not break immediately if you start putting on some plates with a shorter diameter, but the lifespan of that bumper will shorten dramatically.