Option to sync insert only

[fabioespinosa]

Awesome project, works amazingly well.

We are using it in my company to transition to postgres, and during the syncing phase it would be pretty useful to have a "just append whatever is new" instead of deleting the old.

This way we can migrate and perform one last "sync" so that whichever rows in the previous DB get added without deleting the new ones inserted by new users.

[willbryant]

Hmm interesting. What would happen if there's any conflicting unique keys?

[denji]

@willbryant Anything can happen, as the function and trigger can be used on either side.

[fabioespinosa]

yeah, that would certainly be a problem for incremental primary keys and so FKs. We could assume the schema is full of uuids or we could do:

on conflict do nothing

So we give priority to the data that is already there.

If someone were to use this with incremental PKs they could perhaps sum a fixed number to the all the old PKs (on the from database) and then perform the sync, then reset the sequence.

[PekopT]

It's data warehouse case. Will be very useful for us too.

[PekopT]

@willbryant any updates on this?

[willbryant]

Honestly I think I'm missing something, I don't really understand why you would Kitchen Sync for this, it feels so far away from the current behavior and scope of the tool. I've always seen KS's job as being to scan the data and fix up any mismatches.

There's lots of ETL/ELT tools that do a "start from the last row" methodology if you just want to quickly pick up new stuff and there's no changes in history at the source end and new data at the receiving end. In the 'one last "sync" so that whichever rows in the previous DB get added without deleting the new ones inserted by new users' use case, I have trouble imagining that there are many schema where that could produce a valid result. If there's new rows at both ends, for example, any conflicting FKs are going to end up with mis-attributed associations.

If your app's schema does support this, then you probably want multi-master replication?

[willbryant]

OK team, in the spirit of Christmas, I've drafted https://github.com/willbryant/kitchen_sync/tree/insert_only so you can try it and find out.

Not gonna merge it at the moment as I strongly suspect it isn't all that useful, but let me know how you find it.