PolarDB for PostgreSQL (PolarDB for short) is an open source database system based on PostgreSQL.

PolarDB Banner

What is PolarDB for PostgreSQL?

PolarDB for PostgreSQL (PolarDB for short) is an open source database system based on PostgreSQL. It extends PostgreSQL to become a share-nothing distributed database, which supports global data consistency and ACID across database nodes, distributed SQL processing, and data redundancy and high availability through Paxos based replication. PolarDB is designed to add values and new features to PostgreSQL in dimensions of high performance, scalability, high availability, and elasticity. At the same time, PolarDB remains SQL compatibility to single-node PostgreSQL with best effort.

PolarDB will evolve and offer its functions and features in two major parts: an extension and a patch to Postgres. The extension part includes components implemented outside PostgreSQL kernel, such as distributed transaction management, global or distributed time service, distributed SQL processing, additional metadata and internal functions, and tools to manage database clusters and conduct fault tolerance or recovery. Having most of its functions in a Postgres extension, PolarDB targets easy upgrading, easy migration, and fast adoption. The patch part includes the changes necessary to the kernel, such as distributed MVCC for different isolation levels. We expect functions and codes in the patch part is limited. As a result, PolarDB can be easily upgraded with newer PostgreSQL versions and maintain full compatible to PostgreSQL.

Quick start with PolarDB

Three approaches are offered to quickly try out PolarDB: Alibaba Cloud service, deployment using Docker images, and deployment from source codes.

Alibaba Cloud Service

TBD

Deployment Using Docker Images

TBD

One-Key Deployment

onekey.sh can be used to build, configure, deploy, start, and init a cluster of PolarDB.

  • before executing onekey.sh

    • set up environment variables (LD_LIBRARY_PATH and PATH)
    • install dependent packages
    • set up ssh login without password
  • run onekey.sh script

./onekey.sh all
  • check running processes (leader, follower, logger), their replica roles and status:
ps -ef|grep polardb
psql -p 10001 -d postgres -c "select * from pg_stat_replication;"
psql -p 10001 -d postgres -c "select * from polar_dma_cluster_status;"

Deployment from Source Code

We extend a tool named as pgxc_ctl from PG-XC/PG-XL open source project to support cluster management, such as configuration generation, configuration modification, cluster initialization, starting/stopping nodes, and switchover, etc. Its detail usage can be found deployment.

  • download source code
  • install dependent packages (use Centos as an example)
sudo yum install libzstd-devel libzstd zstd cmake openssl-devel protobuf-devel readline-devel libxml2-devel libxslt-devel zlib-devel bzip2-devel lz4-devel snappy-devel
  • set up ssh login without password Call ssh-copy-id command to configure ssh so that no password is needed when using pgxc_ctl.
ssh-copy-id [email protected]
  • build and install binary

you can just call build script to build. If you get errors, please reference deployment for detail reasons.

./build.sh
  • set up environment variables
vi ~/.bashrc
export PATH="$HOME/polardb/polardbhome/bin:$PATH"
export LD_LIBRARY_PATH="$HOME/polardb/polardbhome/lib:$LD_LIBRARY_PATH"ß
  • generate default configure file
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf prepare standalone
  • deploy binary file
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf deploy all
  • clean residual installation and init cluster
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf clean all
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf init all
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf monitor all
  • install dependent packages for cluster management
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf deploy cm
  • start cluster or node
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf start all
  • stop cluster or node
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf stop all
  • failover datanode

datanode_1 is node name configured in polardb_paxos.conf.

pgxc_ctl -c $HOME/polardb/polardb_paxos.conf failover datanode datanode_1
  • cluster health check

check cluster status and start failed node.

pgxc_ctl -c $HOME/polardb/polardb_paxos.conf healthcheck all
  • example for other command
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf kill all
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf log var datanodeNames
pgxc_ctl -c $HOME/polardb/polardb_paxos.conf show configuration all
  • check and test
ps -ef | grep postgres
psql -p 10001 -d postgres -c "create table t1(a int primary key, b int);"
createdb test -p 10001
psql -p 10001 -d test -c "select version();"

reference deployment for detail instructions.

Regress and other test details can be found here. Some benchmarking example is here

Architecture & Roadmap

PolarDB uses a share-nothing architecture. Each node stores data and also executes queries, and they coordinate with each other through message passing. The architecture allows the database to be scaled by adding more nodes to the cluster.

PolarDB slices a table into shards by hashing its primary key. The number of shards is configurable. Shards are stored in PolarDB nodes. When a query accesses shards in multiple nodes, a distributed transaction and a transaction coordinator are used to maintain ACID across nodes.

Each shard is replicated to three nodes with each replica stored on different node. In order to save costs, we can deploy two of the replicas to store complete data. The third replica only stores write ahead log (WAL), which participates in the election but cannot be chosen as the leader.

See architecture design for more information

Documentation

Contributing

PolarDB is built on open source projects, and extends open source PostgreSQL. Your contribution is welcome and appreciated. Please refer contributing for how to start coding and submit a PR.

Licensing

PolarDB code is released under the Apache Version 2.0 License and the Licenses with PostgreSQL code.

The relevant licenses can be found in the comments at the top of each file.

Reference License and NOTICE for details.

Acknowledgements

Some codes and design ideas were from other open source projects, such as PG-XC/XL(pgxc_ctl), TBase (part of timestamp-based vacuum and MVCC), and Citus (pg_cron). Thanks for their contributions.

Communications


Copyright © Alibaba Group, Inc.

Owner
Comments
  • Should we force full English communicate for all reply?

    Should we force full English communicate for all reply?

    If you open-source a project, then ensure that everyone is able to participate. English is spoken by everyone in the open-source communities. Either accept this fact and ensure that every communication is in English AND NOT in Chinese.

    如果你開源一個項目,那麼確保每個人都能參與。 開源社區中的每個人都說英語。 要么接受這個事實,並確保每一次交流都是用英語而不是中文,要么把你的項目移到不同的地方。

  • FATAL:  could not access file

    FATAL: could not access file "pg_cron": No such file or directory

    When initializing the database, starting the database reported an error that the file could not be accessed.But the file does exist, and the user is authorized. image

    如下是路径所在目录: image

  • [Bug] Illegal instruction when pfs mkfs

    [Bug] Illegal instruction when pfs mkfs

    Describe the bug 按照进阶部署文档(https://apsaradb.github.io/PolarDB-for-PostgreSQL/deploying/deploy.html)中的步骤部署,我们采用的是Ceph 共享存储+PFS,读写节点启动了,但是在编译部署PolarDB内核>只读节点部署>启动只读节点 步骤,出现问题如下。

    [[email protected]_34 ~]$ $HOME/tmp_basedir_polardb_pg_1100_bld/bin/pg_ctl start -D $HOME/replica1 waiting for server to start....119 2022-08-28 11:59:41.982 UTC LOG: polar shmem file polar_shmem (key 5433002, ID 1) already exists, try to reuse it. 119 2022-08-28 11:59:41.982 UTC WARNING: polardb skip readControlfile 119 2022-08-28 11:59:41.983 UTC LOG: polar_replica = 'on' 119 2022-08-28 11:59:41.983 UTC LOG: read polar_replica = on, polardb in replica mode, use ro mode mount pfs 119 2022-08-28 11:59:41.984 UTC LOG: polar_vfs loaded in postmaster 119 119 2022-08-28 11:59:41.986 UTC WARNING: could not access file "polar_resource_group": No such file or directory 119 2022-08-28 11:59:41.989 UTC LOG: Auto detecting polar_stat_sql.linux_hz parameter... 119 2022-08-28 11:59:41.989 UTC LOG: polar_stat_sql.linux_hz is set to 1000000 119 2022-08-28 11:59:41.991 UTC LOG: listening on IPv4 address "0.0.0.0", port 5433 119 2022-08-28 11:59:41.991 UTC LOG: listening on IPv6 address "::", port 5433 119 2022-08-28 11:59:41.991 UTC LOG: listening on Unix socket "/tmp/.s.PGSQL.5433" 119 2022-08-28 11:59:41.992 UTC LOG: clog buffer max slot size is 256, and it will be set to 256 119 2022-08-28 11:59:41.992 UTC LOG: commit timestamp buffer max slot size is 128, and it will set to 16 119 2022-08-28 11:59:41.992 UTC LOG: The total log index memory table size is 402525528 119 2022-08-28 11:59:41.992 UTC LOG: The total log index memory table size is 133987128 119 2022-08-28 11:59:41.992 UTC LOG: reuse persisted shared memory, key:5433002, id:1 119 2022-08-28 11:59:41.992 UTC LOG: init vfs_mount_state done 119 2022-08-28 11:59:41.992 UTC LOG: Database will be in readonly mode 119 2022-08-28 11:59:41.992 UTC LOG: init pangu cluster disk 119 2022-08-28 11:59:41.992 UTC LOG: begin mount pfs name sdz1 id 2 pid 119 backendid -1 [PFSD_SDK INF Aug 28 11:59:41.993050][119]pfs_mount_prepare 103: begin prepare mount cluster(disk), PBD(sdz1), hostid(2),flags(0x11) Wrong conn id -5 Wrong conn id -5 [PFSD_SDK INF Aug 28 11:59:41.994208][119]chnl_connection_poll_shm 1238: ack data update s_mount_epoch 1 [PFSD_SDK INF Aug 28 11:59:41.994218][119]chnl_connection_poll_shm 1266: connect and got ack data from svr, err = -5, mntid 0 [PFSD_SDK ERR Aug 28 11:59:41.994221][119]chnl_connection_poll_shm 1296: Connect failed err -5 : Input/output error [PFSD_SDK INF Aug 28 11:59:41.995287][119]pfsd_sdk_init 191: pfsd_chnl_connect failed 119 2022-08-28 11:59:41.996 UTC FATAL: can't mount PBD sdz1, id 2 119 2022-08-28 11:59:41.996 UTC BACKTRACE: /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/postgres(elog_finish+0x1b3) [0x9c0c63] /home/postgres/tmp_basedir_polardb_pg_1100_bld/lib/polar_vfs.so(+0x809c) [0x7ff2460be09c] /home/postgres/tmp_basedir_polardb_pg_1100_bld/lib/polar_monitor_preload.so(allocShmem+0x14) [0x7ff23827bc54] /home/postgres/tmp_basedir_polardb_pg_1100_bld/lib/pg_stat_statements.so(+0x409f) [0x7ff23826a09f] /home/postgres/tmp_basedir_polardb_pg_1100_bld/lib/polar_stat_sql.so(+0x38e9) [0x7ff2460618e9] /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/postgres(CreateSharedMemoryAndSemaphores+0x461) [0x84f361] /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/postgres(PostmasterMain+0xa63) [0x7e4dc3] /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/postgres(main+0x6a5) [0x4f1795] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff243cda555] /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/postgres() [0x4f1892] 119 2022-08-28 11:59:41.996 UTC LOG: database system is shut down stopped waiting pg_ctl: could not start server Examine the log output.

    我的环境是 [[email protected]_34 etc]$ cat redhat-release CentOS Linux release 7.8.2003 (Core) [[email protected]_34 ~]$ uname -a Linux 6_34 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [[email protected]_34 ~]$ sudo docker version Client: Version: 1.13.1 API version: 1.26 Package version: docker-1.13.1-208.git7d71120.el7_9.x86_64 Go version: go1.10.3 Git commit: 7d71120/1.13.1 Built: Mon Jun 7 15:36:09 2021 OS/Arch: linux/amd64 Server: Version: 1.13.1 API version: 1.26 (minimum version 1.12) Package version: docker-1.13.1-208.git7d71120.el7_9.x86_64 Go version: go1.10.3 Git commit: 7d71120/1.13.1 Built: Mon Jun 7 15:36:09 2021 OS/Arch: linux/amd64 Experimental: false

    当时的ceph状态如下 [[email protected]_135 ~]$ docker exec mon01 ceph -s cluster: id: 3ebca9a3-a751-4640-99f5-39345132aaa0 health: HEALTH_OK

    services: mon: 3 daemons, quorum node6_135,6_34,6_33 (age 4h) mgr: node6_135(active, since 4h) mds: 1/1 daemons up osd: 3 osds: 3 up (since 4h), 3 in (since 4h) rgw: 1 daemon active (1 hosts, 1 zones)

    data: volumes: 1/1 healthy pools: 8 pools, 177 pgs objects: 219 objects, 7.2 KiB usage: 55 MiB used, 270 GiB / 270 GiB avail pgs: 177 active+clean

    (采用3个节点,其中磁盘分区后的大小相同,名称不同,在软链接之后名称相同)

  • The configuration file used to build the three-node HA

    The configuration file used to build the three-node HA

    I want to build an HA cluster with three physical servers. Is the configuration correct?DDB1 dDB2 DDB3 is the host map configured to /etc/hosts. polardb_paxos.txt

  • Cannot use pgxc_ ctl

    Cannot use pgxc_ ctl

    I followed the README, but pgxc_ctl binary was not generated. So I took advice from others, run "make install to generate pgxc_ctl binary file in PGXC_CTL_HOME".

    It does generate pgxc_ctl binary. But if you use it, you will get an error:

    /usr/bin/bash /home/postgres/polardb/polardbhome/bin/pgxc_ctl is not directory. Check your configurfation

  • run ./onekey.sh all occur error: ./onekey.sh: line 39: pgxc_ctl: command not found

    run ./onekey.sh all occur error: ./onekey.sh: line 39: pgxc_ctl: command not found

    os:centos 7.6.1810

    run ./onekey.sh all occur error: ./onekey.sh: line 39: pgxc_ctl: command not found

    原因: build.sh 脚本中export $PATH的环境变量,不会在onekey.sh脚本中生效,导致pgxc_ctl命令找不到,需要在onekey.sh中重新export $PATH

  • fix: count of regress tests

    fix: count of regress tests

    Cases in "polar_ignore" has not been tested, so they should not be count in success_count. Success cases in "ignore" has been count in success_count, so count of all test case should contain fail_count and fail_ignore_count, not ignore_count.

  • docs: verify quick start in windows

    docs: verify quick start in windows

    I have verified the quick start operation under docker in windows. It works in single instance, multiple instance and HTAP instance. Therefore, I make some change to quick-start.md.

  • [Question]运行HTAP测试的时候出现ERROR: could not resize shared memory segment

    [Question]运行HTAP测试的时候出现ERROR: could not resize shared memory segment

    运行HTAP测试的时候出现ERROR: could not resize shared memory segment

    首先,我们使用oltpbench运行polardb的测试前,测试了读写节点和所有只读节点前的连通性,可以通过psql -h <ip_address> -p -Upostgres postgres登录 然后我们在读写节点上运行TP事务,在其中一个读写节点上运行AP查询(猜测只读节点之间会自动平衡负载),TP和AP中数据库的连接均填写了读写节点的ip,运行过程中,读写节点没有出现错误,而AP执行中出现以下错误

    07:26:41,776 (Worker.java:501) WARN  - The DBMS rejected the transaction without an error code
    org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:309)
            at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:295)
            at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:272)
            at org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:246)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.Q15.run(Q15.java:71)
            at com.oltpbenchmark.benchmarks.chbenchmark.CHBenCHmarkWorker.executeWork(CHBenCHmarkWorker.java:37)
            at com.oltpbenchmark.api.Worker.doWork(Worker.java:388)
            at com.oltpbenchmark.api.Worker.run(Worker.java:296)
            at java.lang.Thread.run(Thread.java:750)
    Caused by: org.postgresql.util.PSQLException: ERROR: could not resize shared memory segment "/PostgreSQL.1522265545" to 8388608 bytes: No space left on device
      Where: parallel worker
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:158)
            at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:108)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.GenericQuery.run(GenericQuery.java:77)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.Q15.run(Q15.java:69)
            ... 4 more
    07:28:32,885 (Worker.java:501) WARN  - The DBMS rejected the transaction without an error code
    org.postgresql.util.PSQLException: ERROR: could not resize shared memory segment "/PostgreSQL.1509574628" to 8388608 bytes: No space left on device
      Where: parallel worker
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:158)
            at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:108)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.GenericQuery.run(GenericQuery.java:77)
            at com.oltpbenchmark.benchmarks.chbenchmark.CHBenCHmarkWorker.executeWork(CHBenCHmarkWorker.java:37)
            at com.oltpbenchmark.api.Worker.doWork(Worker.java:388)
            at com.oltpbenchmark.api.Worker.run(Worker.java:296)
            at java.lang.Thread.run(Thread.java:750)
    07:38:58,125 (Worker.java:501) WARN  - The DBMS rejected the transaction without an error code
    org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:309)
            at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:295)
            at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:272)
            at org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:246)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.Q15.run(Q15.java:71)
            at com.oltpbenchmark.benchmarks.chbenchmark.CHBenCHmarkWorker.executeWork(CHBenCHmarkWorker.java:37)
            at com.oltpbenchmark.api.Worker.doWork(Worker.java:388)
            at com.oltpbenchmark.api.Worker.run(Worker.java:296)
            at java.lang.Thread.run(Thread.java:750)
    Caused by: org.postgresql.util.PSQLException: ERROR: could not resize shared memory segment "/PostgreSQL.888046644" to 8388608 bytes: No space left on device
      Where: parallel worker
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:158)
            at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:108)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.GenericQuery.run(GenericQuery.java:77)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.Q15.run(Q15.java:69)
            ... 4 more
    07:52:39,608 (Worker.java:501) WARN  - The DBMS rejected the transaction without an error code
    org.postgresql.util.PSQLException: ERROR: could not resize shared memory segment "/PostgreSQL.1289792276" to 8388608 bytes: No space left on device
      Where: parallel worker
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:158)
            at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:108)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.GenericQuery.run(GenericQuery.java:77)
            at com.oltpbenchmark.benchmarks.chbenchmark.CHBenCHmarkWorker.executeWork(CHBenCHmarkWorker.java:37)
            at com.oltpbenchmark.api.Worker.doWork(Worker.java:388)
            at com.oltpbenchmark.api.Worker.run(Worker.java:296)
            at java.lang.Thread.run(Thread.java:750)
    07:53:24,679 (Worker.java:501) WARN  - The DBMS rejected the transaction without an error code
    org.postgresql.util.PSQLException: ERROR: could not resize shared memory segment "/PostgreSQL.335896825" to 8388608 bytes: No space left on device
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:158)
            at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:108)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.GenericQuery.run(GenericQuery.java:77)
            at com.oltpbenchmark.benchmarks.chbenchmark.CHBenCHmarkWorker.executeWork(CHBenCHmarkWorker.java:37)
            at com.oltpbenchmark.api.Worker.doWork(Worker.java:388)
            at com.oltpbenchmark.api.Worker.run(Worker.java:296)
            at java.lang.Thread.run(Thread.java:750)
    07:53:37,134 (Worker.java:501) WARN  - The DBMS rejected the transaction without an error code
    org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:309)
            at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:295)
            at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:272)
            at org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:246)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.Q15.run(Q15.java:71)
            at com.oltpbenchmark.benchmarks.chbenchmark.CHBenCHmarkWorker.executeWork(CHBenCHmarkWorker.java:37)
            at com.oltpbenchmark.api.Worker.doWork(Worker.java:388)
            at com.oltpbenchmark.api.Worker.run(Worker.java:296)
            at java.lang.Thread.run(Thread.java:750)
    Caused by: org.postgresql.util.PSQLException: ERROR: could not resize shared memory segment "/PostgreSQL.759095010" to 8388608 bytes: No space left on device
      Where: parallel worker
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:158)
            at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:108)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.GenericQuery.run(GenericQuery.java:77)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.Q15.run(Q15.java:69)
            ... 4 more
    07:55:02,147 (Worker.java:501) WARN  - The DBMS rejected the transaction without an error code
    org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:309)
            at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:295)
            at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:272)
            at org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:246)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.Q15.run(Q15.java:71)
            at com.oltpbenchmark.benchmarks.chbenchmark.CHBenCHmarkWorker.executeWork(CHBenCHmarkWorker.java:37)
            at com.oltpbenchmark.api.Worker.doWork(Worker.java:388)
            at com.oltpbenchmark.api.Worker.run(Worker.java:296)
            at java.lang.Thread.run(Thread.java:750)
    Caused by: org.postgresql.util.PSQLException: ERROR: could not resize shared memory segment "/PostgreSQL.67511109" to 8388608 bytes: No space left on device
      Where: parallel worker
            at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
            at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2241)
            at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
            at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
            at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
            at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:158)
            at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:108)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.GenericQuery.run(GenericQuery.java:77)
            at com.oltpbenchmark.benchmarks.chbenchmark.queries.Q15.run(Q15.java:69)
            ... 4 more
    

    根据这个问题,我们尝试了修改docker容器的shm_size,停下容器修改配置文件和重启容器时附带配置两种方式均没有能够成功修改这个值,想问下是我运行的方式不正确吗,还是需要修改某个特定参数呢

  • fix: polar_px extension does not specify schema

    fix: polar_px extension does not specify schema

    If we do not specify the SCHEMA of polar_px extension, PX may fail to create plan after SCHEMA change. That is because we can not get px_workerid_funcid after SCHEMA change.

  • Fail to run PolarDB[Question]

    Fail to run PolarDB[Question]

    Describe the problem

    ... When I tried to rerun ./polardb_build.sh --initpx in docker polardb/polardb_pg_devel , I met this error.

    2022-07-06 11:31:09.833 UTC [88204] WARNING:  polardb skip readControlfile
    2022-07-06 11:31:09.833 UTC [88204] LOG:  polar_replica = 'on'
    2022-07-06 11:31:09.833 UTC [88204] LOG:  read polar_replica = on, polardb in replica mode, use ro mode mount pfs
    2022-07-06 11:31:09.834 UTC [88204] LOG:  polar_vfs loaded in postmaster 88204
    2022-07-06 11:31:09.836 UTC [88204] WARNING:  could not access file "polar_resource_group": No such file or directory
    2022-07-06 11:31:09.838 UTC [88204] LOG:  Auto detecting polar_stat_sql.linux_hz parameter...
    2022-07-06 11:31:09.838 UTC [88204] LOG:  polar_stat_sql.linux_hz is set to 1000000
    2022-07-06 11:31:09.843 UTC [88204] LOG:  listening on IPv4 address "0.0.0.0", port 5433
    2022-07-06 11:31:09.843 UTC [88204] LOG:  listening on IPv6 address "::", port 5433
    2022-07-06 11:31:09.844 UTC [88204] LOG:  listening on Unix socket "./.s.PGSQL.5433"
    2022-07-06 11:31:09.845 UTC [88204] LOG:  clog buffer max slot size is 256, and it will be set to 256
    2022-07-06 11:31:09.845 UTC [88204] LOG:  commit timestamp buffer max slot size is 128, and it will set to 16
    2022-07-06 11:31:09.845 UTC [88204] LOG:  The total log index memory table size is 402525528
    2022-07-06 11:31:09.845 UTC [88204] LOG:  The total log index memory table size is 133987128
    2022-07-06 11:31:09.845 UTC [88204] LOG:  The total log index memory table size is 66993864
    2022-07-06 11:31:09.847 UTC [88204] FATAL:  could not map anonymous shared memory: Cannot allocate memory
    2022-07-06 11:31:09.847 UTC [88204] HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 2644295680 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
    2022-07-06 11:31:09.847 UTC [88204] BACKTRACE:  
            /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/postgres(PGSharedMemoryCreate+0x726) [0xa12fe6]
            /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/postgres(CreateSharedMemoryAndSemaphores+0x3c0) [0xa95a80]
            /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/postgres(PostmasterMain+0xa6d) [0xa2b36d]
            /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/postgres(main+0x781) [0x72a0b1]
            /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f54218e2555]
            /home/postgres/tmp_basedir_polardb_pg_1100_bld/bin/postgres() [0x735f8c]
    2022-07-06 11:31:09.847 UTC [88204] LOG:  database system is shut down
     stopped waiting
    pg_ctl: could not start server
    Examine the log output.
    

    I did not do anything except running that command.

  • [Question][Roadmap]超融合数据库

    [Question][Roadmap]超融合数据库

    增强方向(一切皆SQL): 1)实时ETL/流数据处理:仅靠SQL就能实现实时ETL/流数据处理; 2)机器学习:库内机器学习,通过类SQL实现常用机器学习(可离线可实时); 3)时序数据库。

    背景:目前大数据的组件越来越多,使用及维护变得越来越麻烦,并且会造成数据冗余在各个组件里,形成数据孤岛。超融合数据库性价比高,能大大简化技术栈,降低系统复杂度,降低运维复杂度,提升开发效率。分久必合,超融合数据库有很好的发展前景。

  • WIP: parallel acceleration of materialized view creation

    WIP: parallel acceleration of materialized view creation

    Use PolarDB PG's HTAP ability to accelerate materialized view creation.

    Optimizer in PolarDB PG can generate MPP execution plan for qualifying SQLs, enabling concurrent execution of such a plan across multiple PolarDB PG instances. Utilize this ability to generate a MPP execution plan for materialized view creation.

    relate to #247

  • WIP: supports distributed acceleration of COPY [skip ci]

    WIP: supports distributed acceleration of COPY [skip ci]

    What?

    This pr is for #246

    This feature will assist PolarDB for PostgreSQL in implementing distributed acceleration of COPY so that PolarDB PG can significantly improve performance when executing COPY statements in the case of large data volumes and fully utilise the HTAP framework.

    The implementation includes

    • For the COPY ... TO ... statement, multiple machines should be read in parallel for the read operation,
    • For the COPY ... FROM ... statement should be written by multiple processes on a single read/write node.

    How?

    • [x] global variable for the user to turn on this feature set polar_px_enable_copy=on
    • [x] In Copy query to File, make query use the PX optimizer
    • [x] Copy ... To ...
    • [ ] Copy ... From ...

    Benchmark

    • Copy query to File image

    • Copy ... To ... image

    Screenshots (optional)

  • 2022 Alibaba Summer of Code

    2022 Alibaba Summer of Code

    Welcome to the open source world! If you haven't planned how to spend this summer, come to the Alibaba Summer of Code and code with us! 💻 Alibaba Summer of Code is a global program focused on engaging students directly in open source software development. Under the guidance of the mentor in the Alibaba open source project, students can experience software development in the real world. Alibaba Summer of code will begin from May 30th to September 1st. Students can use the summertime to participate in the open source project and work with the core members of the project.

    What you can get?

    On this exclusive developer journey, students will have the opportunity to: ● Participate in the top projects of the International Open Source Foundation; ● Get a scholarship from Alibaba; ● Obtain an open source contributor certificate; ● Get a fast pass of Alibaba Internship ● Get your code adopted and used by the open source project!

    Our Mentor

    He kewen,@HBKO ,ASoC Mentor, PolarDB-for-PostgreSQL Committer. Zhang Jingtang,@mrdrivingduck ,ASoC Mentor, PolarDB-for-PostgreSQL Committer. Sun Mengshi,ASoC Mentor, PolarDB-X Committer. Wu Di,@vettalwu,ASoC Mentor, PolarDB-X Committer.

    Timeline

    image

    Apply Now!

    1. Browse open idea list here: [ASoC] PolarDB HTAP 支持 COPY 导入数据的分布式加速 [ASoC] PolarDB HTAP 支持并行加速创建物化视图 [ASoC] PolarDB-X Operator 混沌测试实践 [ASoC] PolarDB-X 支持 DBLink

    2. Upload your CV and project proposal via ASOC 2022 official website https://opensource.alibaba.com/asoc2022 Contact the Organizer If you have any questions, visit the event website:https://opensource.alibaba.com/asoc2022 Email address:
      Email address: [email protected]

    PolarDB-PG 开源交流钉钉群: image

    PolarDB-X 开源交流钉钉群 image

  • [ASoC] PolarDB HTAP 支持并行加速创建物化视图

    [ASoC] PolarDB HTAP 支持并行加速创建物化视图

    题目描述

    在 CREATE MATERIALIZED VIEW(创建物化视图)时需要按需对表中的元组进行扫描。 PostgreSQL: Documentation: 14: 41.3. Materialized Views

    项目目标

    通过 PolarDB 的 HTAP 能力并行加速这个过程。

    项目技术要求

    熟悉C语言及Linux上的相关调试技术,对 PostgreSQL 内核有一定了解。

    题目难度

    中等

    导师信息

    Zhang Jingtang,@mrdrivingduck , ASoC Mentor, PolarDB-for-PostgreSQL maintainer. [email protected](联系导师请附上个人简历)

  • [ASoC]PolarDB HTAP支持COPY导入数据的分布式加速

    [ASoC]PolarDB HTAP支持COPY导入数据的分布式加速

    题目要求:

    为了使PolarDB PG具备更强的导入导出数据的能力,可以利用PolarDB PX并行执行来加速COPY命令。实现的效果为,COPY命令在读取表导出csv的时候,可以多个读节点同时并发读取表来进行加速;

    在COPY写入表的时候,可以在单个写节点起多个写进程并发写入。

    任务目标:

    实现并行读并行写的COPY功能

    题目难度

    中等

    导师信息

    He kewen,@HBKO ,ASoC Mentor, PolarDB-for-PostgreSQLCommitter. [email protected](联系导师请附上个人简历)

YugabyteDB is a high-performance, cloud-native distributed SQL database that aims to support all PostgreSQL features
YugabyteDB is a high-performance, cloud-native distributed SQL database that aims to support all PostgreSQL features

YugabyteDB is a high-performance, cloud-native distributed SQL database that aims to support all PostgreSQL features. It is best to fit for cloud-native OLTP (i.e. real-time, business-critical) applications that need absolute data correctness and require at least one of the following: scalability, high tolerance to failures, or globally-distributed deployments.

Nov 24, 2022
A friendly and lightweight C++ database library for MySQL, PostgreSQL, SQLite and ODBC.

QTL QTL is a C ++ library for accessing SQL databases and currently supports MySQL, SQLite, PostgreSQL and ODBC. QTL is a lightweight library that con

Nov 22, 2022
upstream module that allows nginx to communicate directly with PostgreSQL database.

About ngx_postgres is an upstream module that allows nginx to communicate directly with PostgreSQL database. Configuration directives postgres_server

Apr 29, 2022
Open Source Oracle Compatible PostgreSQL.

IvorySQL is advanced, fully featured, open source Oracle compatible PostgreSQL with a firm commitment to always remain 100% compatible and a Drop-in r

Nov 25, 2022
A mini database for learning database

A mini database for learning database

Nov 14, 2022
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability
Nebula Graph is a distributed, fast open-source graph database featuring horizontal scalability and high availability

Nebula Graph is an open-source graph database capable of hosting super large scale graphs with dozens of billions of vertices (nodes) and trillions of edges, with milliseconds of latency.

Nov 26, 2022
TimescaleDB is an open-source database designed to make SQL scalable for time-series data.

An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.

Nov 22, 2022
DB Browser for SQLite (DB4S) is a high quality, visual, open source tool to create, design, and edit database files compatible with SQLite.
DB Browser for SQLite (DB4S) is a high quality, visual, open source tool to create, design, and edit database files compatible with SQLite.

DB Browser for SQLite What it is DB Browser for SQLite (DB4S) is a high quality, visual, open source tool to create, design, and edit database files c

Dec 2, 2022
The open-source database for the realtime web.
The open-source database for the realtime web.

RethinkDB What is RethinkDB? Open-source database for building realtime web applications NoSQL database that stores schemaless JSON documents Distribu

Nov 24, 2022
GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.
GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.

Overview GridDB is Database for IoT with both NoSQL interface and SQL Interface. Please refer to GridDB Features Reference for functionality. This rep

Nov 24, 2022
Database system project based on CMU 15-445/645 (FALL 2020)

Database system project based on CMU 15-445/645 (FALL 2020)

Nov 10, 2022
DuckDB is an in-process SQL OLAP Database Management System
DuckDB is an in-process SQL OLAP Database Management System

DuckDB is an in-process SQL OLAP Database Management System

Nov 23, 2022
GalaxyEngine is a MySQL branch originated from Alibaba Group, especially supports large-scale distributed database system.

GalaxyEngine is a MySQL branch originated from Alibaba Group, especially supports large-scale distributed database system.

Nov 29, 2022
MillenniumDB is a graph oriented database management system

Millennium DB MillenniumDB is a graph oriented database management system developed by the Millennium Institute for Foundational Research on Data (IMF

Oct 23, 2022
Kunlun distributed DBMS is a NewSQL OLTP relational distributed database management system

Kunlun distributed DBMS is a NewSQL OLTP relational distributed database management system. Application developers can use Kunlun to build IT systems that handles terabytes of data, without any effort on their part to implement data sharding, distributed transaction processing, distributed query processing, crash safety, high availability, strong consistency, horizontal scalability. All these powerful features are provided by Kunlun.

Nov 23, 2022
ESE is an embedded / ISAM-based database engine, that provides rudimentary table and indexed access.

Extensible-Storage-Engine A Non-SQL Database Engine The Extensible Storage Engine (ESE) is one of those rare codebases having proven to have a more th

Nov 20, 2022
A MariaDB-based command line tool to connect to OceanBase Database.

什么是 OceanBase Client OceanBase Client(简称 OBClient) 是一个基于 MariaDB 开发的客户端工具。您可以使用 OBClient 访问 OceanBase 数据库的集群。OBClient 采用 GPL 协议。 OBClient 依赖 libobclie

Nov 9, 2022
The official C++ client API for PostgreSQL.

libpqxx Welcome to libpqxx, the C++ API to the PostgreSQL database management system. Home page: http://pqxx.org/development/libpqxx/ Find libpqxx on

Nov 21, 2022
A PostgreSQL extension providing an async networking interface accessible via SQL using a background worker and curl.

pg_net is a PostgreSQL extension exposing a SQL interface for async networking with a focus on scalability and UX.

Nov 25, 2022