database replication tagged posts

Recap for Concurrency: Tracking Database Changes

We talked how to track database changes with one auditing table last month. If we want to replicate these changes into another database, then we might use multi-threading to gain the high performance in data ingestion. However, based on our previous design, we track changes to column level, which generates a serious concurrency control issue. Let’s think about, we have a data table, with 10 columns updates at once, then all these changes are captured and inserted into my_audit_table. If at the same time, another table gets updated, then there might have some cross inserting in my_audit_table, which makes the multi-threading read even harder.

Is there a better approach that we should go? In this case, it’s worth to re-design the audit_table. Let’s see how it looks like (in Oracle PL/SQL):

c...
Read More

How to Improve Data Replication Performance Significantly

Recently we achieved a project regarding large data replication from Oracle database to PostgreSQL. We are facing the challenge that we have very limited window to complete the large volume data loading on daily basis.

Initially we tried PostgreSQL loading tool psql, it took 4.5 hours to complete transferring approx. 250,000 records. database-master-masterBut regular business operations require a lot shorter time to transfer more data, e.g. complete 1 million records replication in 30 minutes. Based on the performance of psql (which is very likely single thread basis), to complete 1 million records replication needs 18 hours (1,080 minutes)! – which is not feasible from business perspectives.

We built up an application based on Quartz Scheduler which does table to table asynchronous replication in nearly rea...

Read More