Data synchronization

Data formats tend to grow more complex with time as the organization grows and evolves. This results not only in building simple interfaces between the two applications (source and target), but also in a need to transform the data while passing them to the target application. ETL (extraction transformation loading) tools can be helpful at this stage for managing data format complexities.

In real-time systems, customers want to see the current status of their order in e-shop, the current status of a parcel delivery—a real time parcel tracking—, the current balance on their account, etc. This shows the need of a real-time system, which is being updated as well to enable smooth manufacturing process in real-time, e.g., ordering material when enterprise is running out stock, synchronizing customer orders with manufacturing process, etc. From real life, there exist so many examples where real-time processing gives successful and competitive advantage.

There are no fixed rules and policies to enforce data security. It may vary depending on the system which you are using. Even though the security is maintained correctly in the source system which captures the data, the security and information access privileges must be enforced on the target systems as well to prevent any potential misuse of the information. This is a serious issue and particularly when it comes for handling secret, confidential and personal information. So because of the sensitivity and confidentiality, data transfer and all in-between information must be encrypted.

Data quality is another serious constraint. For better management and to maintain good quality of data, the common practice is to store the data at one location and share with different people and different systems and/or applications from different locations. It helps in preventing inconsistencies in the data.

There are five different phases involved in the data synchronization process:

• data extraction from the source (or master, or main) system;
• data transfer;
• data transformation;
• data load to the target system.
• data updation

Each of these steps is critical. In case of large amounts of data, the synchronization process needs to be carefully planned and executed to avoid any negative impact on performance.

The problem of synchronizing unordered data (also known as the set reconciliation problem) is modeled as an attempt to compute the symmetric difference ${\displaystyle S_{A}\oplus S_{B}=(S_{A}-S_{B})\cup (S_{B}-S_{A})}$ between two remote sets ${\displaystyle S_{A}}$ and ${\displaystyle S_{B}}$ of b-bit numbers.[3] Some solutions to this problem are typified by:

Wholesale transfer

In this case all data is transferred to one host for a local comparison.

Timestamp synchronization

In this case all changes to the data are marked with timestamps. Synchronization proceeds by transferring all data with a timestamp later than the previous synchronization.[4]

Mathematical synchronization

In this case data are treated as mathematical objects and synchronization corresponds to a mathematical process.[3][5][6]

In this case, two remote strings ${\displaystyle \sigma _{A}}$ and ${\displaystyle \sigma _{B}}$ need to be reconciled. Typically, it is assumed that these strings differ by up to a fixed number of edits (i.e. character insertions, deletions, or modifications). Then data synchronization is the process of reducing edit distance between ${\displaystyle \sigma _{A}}$ and ${\displaystyle \sigma _{B}}$, up to the ideal distance of zero. This is applied in all filesystem based synchronizations (where the data is ordered). Many practical applications of this are discussed or referenced above.

It is sometimes possible to transform the problem to one of unordered data through a process known as shingling (splitting the strings into shingles).[7]