5.3.1. Latency or Relative Latency Display

Tungsten Cluster can operate using either absolute or relative latency. The two are distinguished according to how the difference between transaction commit times are handled:

  • Absolute latency — (default) is the difference between when a transaction was applied to a Replica and when the transaction was originally applied to the Primary.

  • Relative latency — is the difference between now and when the last transaction was written to the Replica.

Absolute latency indicates the difference between transaction times, but, may also provide a misleading impression of the cluster state if there are large transactions being applied, or if the Replica has stopped or become 'stuck' due to a transient failure. This is because absolute latency shows the time difference between transactions. If a transaction takes 5 or 10 seconds to apply, the absolute latency will only display the difference between when the transaction was written, and only after this has occurred on both the Primary and the Replica. The actual time difference between these may be less than a second, even though the transaction took 10 seconds to succeed.

Relative latency shows the time difference between the last transaction committed and the current time, hence if the transaction takes a considerable time to be applied, the relative latency will increase up until the transaction has finally been committed. If the relative latency increases and continues to increase, it may indicate a lagging or even failed Replica.

To enable relative latency, the cluster must have been deployed, or updated, using the --use-relative-latency=true option to tpm. Once enabled, the following operational activities change:

  • The output of show slave status when connected to MySQL through a connector will be updated so that the Seconds_Behind_Master field shows the relative, rather than absolute, latency. For example, in a cluster where relative latency is enabled, but no transactions are occurring, the output will show an increasing value:

    mysql> show slave status\G
    *************************** 1. row ***************************
    ...
    Seconds_Behind_Master: 0
    ...
    1 row in set (0.01 sec)
    mysql> show slave status\G
    *************************** 1. row ***************************
    ...
    Seconds_Behind_Master: 7
    ...
    1 row in set (0.01 sec)
    mysql> show slave status\G
    *************************** 1. row ***************************
    ...
    Seconds_Behind_Master: 38
    ...
    1 row in set (0.01 sec)
  • cctrl will output an additional field, relative, showing the relative latency value against the standard latency value. This can be seen in the example below:

    [LOGICAL] /alpha > ls
    
    COORDINATOR[host1:AUTOMATIC:ONLINE]
    
    ROUTERS:
    +----------------------------------------------------------------------------+
    |connector@host1[6189](ONLINE, created=1, active=0)                          |
    |connector@host2[14253](ONLINE, created=3, active=2)                         |
    |connector@host3[2419](ONLINE, created=1, active=0)                          |
    +----------------------------------------------------------------------------+
    
    DATASOURCES:
    +----------------------------------------------------------------------------+
    |host1(master:ONLINE, progress=5, THL latency=1.008, relative=144.636)       |
    |STATUS [OK] [2014/09/07 02:28:44 PM BST]                                    |
    +----------------------------------------------------------------------------+
    |  MANAGER(state=ONLINE)                                                     |
    |  REPLICATOR(role=master, state=ONLINE)                                     |
    |  DATASERVER(state=ONLINE)                                                  |
    |  CONNECTIONS(created=4, active=1)                                          |
    +----------------------------------------------------------------------------+
    
    +----------------------------------------------------------------------------+
    |host2(slave:ONLINE, progress=5, latency=0.000, relative=144.638)            |
    |STATUS [OK] [2014/09/07 02:28:58 PM BST]                                    |
    +----------------------------------------------------------------------------+
    |  MANAGER(state=ONLINE)                                                     |
    |  REPLICATOR(role=slave, master=host1, state=ONLINE)                        |
    |  DATASERVER(state=ONLINE)                                                  |
    |  CONNECTIONS(created=1, active=1)                                          |
    +----------------------------------------------------------------------------+
    
    +----------------------------------------------------------------------------+
    |host3(slave:ONLINE, progress=5, latency=5.938, relative=144.620)            |
    |STATUS [OK] [2014/09/07 02:29:13 PM BST]                                    |
    +----------------------------------------------------------------------------+
    |  MANAGER(state=ONLINE)                                                     |
    |  REPLICATOR(role=slave, master=host1, state=ONLINE)                        |
    |  DATASERVER(state=ONLINE)                                                  |
    |  CONNECTIONS(created=0, active=0)                                          |
    +----------------------------------------------------------------------------+
  • The Tungsten Connector will use the value when the maxAppliedLatency option is used in the connection string to determine whether to route a connection to a Primary or a Replica.

    For example, when running a scrip that sends a heartbeat, and then connects through a connector, the connection will be routed first to the Replica, and then to the Primary:

    echo "cluster heartbeat" | cctrl
    sleep 1
    mysql -utungsten_testing -pprivate --port=9999 --host=`hostname` \
    mysql@maxAppliedLatency=20?qos=RO_RELAXED -e"select 1;tungsten connection status;"
    
    sleep 21
    
    mysql -utungsten_testing -pprivate --port=9999 --host=`hostname` \
    mysql@maxAppliedLatency=20?qos=RO_RELAXED -e"select 1;tungsten connection status;"

    The output of the execution of the script shows the Replica and then Primary connections:

    [LOGICAL] /alpha > cluster heartbeat
    HEARTBEAT 'DEFAULT' INSERTED
    [LOGICAL] /alpha > 
    Exiting...
    +---+
    | 1 |
    +---+
    | 1 |
    +---+
    +---------------------------------------------------------------------------------+
    | Message                                                                         |
    +---------------------------------------------------------------------------------+
    | host1@alpha(slave:ONLINE) STATUS(OK), QOS=RO_RELAXED SSL.IN=false SSL.OUT=false |
    +---------------------------------------------------------------------------------+
    +---+
    | 1 |
    +---+
    | 1 |
    +---+
    +---------------------------------------------------------------------------------+
    | Message                                                                         |
    +---------------------------------------------------------------------------------+
    | host1@alpha(master:ONLINE) STATUS(OK), QOS=RO_RELAXED SSL.IN=false SSL.OUT=false |
    +---------------------------------------------------------------------------------+