Django CMS Pages is built like so it has multiple versions of pages. Each Page model that is changed is stored under new revision. And so plugin is copied and relinked to that page too. So plugin revisions multiply to.
For e.g. in case you have a Draft page with two plugins that have model primary key number (pk in future) 12 and 14 accordingly. Say you hit publish changes. Not only PK of Page model rises and data are copypasted into new empty Page model instance. Those plugins are copied to. So mentioned plugins PK would change to 15 and 16 accordingly, assuming 14 is the latest plugin pk. New copied plugins will be linked to new page. Thats how things work in Django CMS.
There is also a line in the CMS code stating in its comment that you need to override m2m storing by yourself. Somehow its omitted from basic Django CMS tutorials and I can only find it in the old versions documentation for now. SO THE SOLUTION TO ALL THIS:
Say you have CMSPlugin a model for your custom plugin:
Here method copy_relations overrides default empty method.
For foreign keys it is described another behaviour and copy_relations mappings. However Foreign keys were working like intended by themselves in my plugins. SO I believe info about them is outdated a bit.
Otherwise it may be useful to read about it in official docs.
Thats it. Hope it will help someone like myself to solve a problem like this in a timely manner.