5.3.1. Latency or Relative Latency Display

Tungsten Clustering can operate using either absolute or relative latency. The two are distinguished according to how the difference between transaction commit times are handled:

  • Absolute latency — (default) is the difference between when a transaction was applied to a slave and when the transaction was originally applied to the master.

  • Relative latency — is the difference between now and when the last transaction was written to the slave.

Absolute latency indicates the difference between transaction times, but, may also provide a misleading impression of the cluster state if there are large transactions being applied, or if the slave has stopped or become 'stuck' due to a transient failure. This is because absolute latency shows the time difference between transactions. If a transaction takes 5 or 10 seconds to apply, the absolute latency will only display the difference between when the transaction was written, and only after this has occurred on both the master and the slave. The actual time difference between these may be less than a second, even though the transaction took 10 seconds to succeed.

Relative latency shows the time difference between the last transaction committed and the current time, hence if the transaction takes a considerable time to be applied, the relative latency will increase up until the transaction has finally been committed. If the relative latency increases and continues to increase, it may indicate a lagging or even failed slave.

To enable relative latency, the cluster must have been deployed, or updated, using the --use-relative-latency=true option to tpm. Once enabled, the following operational activities change:

  • The output of tungsten show slave status when connected to MySQL through a connector will be updated so that the Seconds_Behind_Master field shows the relative, rather than absolute, latency. For example, in a cluster where relative latency is enabled, but no transactions are occurring, the output will show an increasing value:

    mysql> show slave status\G
    *************************** 1. row ***************************
    ...
    Seconds_Behind_Master: 0
    ...
    1 row in set (0.01 sec)
    mysql> show slave status\G
    *************************** 1. row ***************************
    ...
    Seconds_Behind_Master: 7
    ...
    1 row in set (0.01 sec)
    mysql> show slave status\G
    *************************** 1. row ***************************
    ...
    Seconds_Behind_Master: 38
    ...
    1 row in set (0.01 sec)
  • cctrl will output an additional field, relative, showing the relative latency value against the standard latency value. This can be seen in the example below:

    [LOGICAL] /alpha > ls
    
    COORDINATOR[host1:AUTOMATIC:ONLINE]
    
    ROUTERS:
    +----------------------------------------------------------------------------+
    |connector@host1[6189](ONLINE, created=1, active=0)                          |
    |connector@host2[14253](ONLINE, created=3, active=2)                         |
    |connector@host3[2419](ONLINE, created=1, active=0)                          |
    +----------------------------------------------------------------------------+
    
    DATASOURCES:
    +----------------------------------------------------------------------------+
    |host1(master:ONLINE, progress=5, THL latency=1.008, relative=144.636)       |
    |STATUS [OK] [2014/09/07 02:28:44 PM BST]                                    |
    +----------------------------------------------------------------------------+
    |  MANAGER(state=ONLINE)                                                     |
    |  REPLICATOR(role=master, state=ONLINE)                                     |
    |  DATASERVER(state=ONLINE)                                                  |
    |  CONNECTIONS(created=4, active=1)                                          |
    +----------------------------------------------------------------------------+
    
    +----------------------------------------------------------------------------+
    |host2(slave:ONLINE, progress=5, latency=0.000, relative=144.638)            |
    |STATUS [OK] [2014/09/07 02:28:58 PM BST]                                    |
    +----------------------------------------------------------------------------+
    |  MANAGER(state=ONLINE)                                                     |
    |  REPLICATOR(role=slave, master=host1, state=ONLINE)                        |
    |  DATASERVER(state=ONLINE)                                                  |
    |  CONNECTIONS(created=1, active=1)                                          |
    +----------------------------------------------------------------------------+
    
    +----------------------------------------------------------------------------+
    |host3(slave:ONLINE, progress=5, latency=5.938, relative=144.620)            |
    |STATUS [OK] [2014/09/07 02:29:13 PM BST]                                    |
    +----------------------------------------------------------------------------+
    |  MANAGER(state=ONLINE)                                                     |
    |  REPLICATOR(role=slave, master=host1, state=ONLINE)                        |
    |  DATASERVER(state=ONLINE)                                                  |
    |  CONNECTIONS(created=0, active=0)                                          |
    +----------------------------------------------------------------------------+
  • The Tungsten Connector will use the value when the maxAppliedLatency option is used in the connection string to determine whether to route a connection to a master or a slave.

    For example, when running a scrip that sends a heartbeat, and then connects through a connector, the connection will be routed first to the slave, and then to the master:

    echo "cluster heartbeat" | cctrl
    sleep 1
    mysql -utungsten_testing -pprivate --port=9999 --host=`hostname` \
    mysql@maxAppliedLatency=20?qos=RO_RELAXED -e"select 1;tungsten connection status;"
    
    sleep 21
    
    mysql -utungsten_testing -pprivate --port=9999 --host=`hostname` \
    mysql@maxAppliedLatency=20?qos=RO_RELAXED -e"select 1;tungsten connection status;"

    The outp[ut of the execution of the script shows the slave and then master connections:

    [LOGICAL] /alpha > cluster heartbeat
    HEARTBEAT 'DEFAULT' INSERTED
    [LOGICAL] /alpha > 
    Exiting...
    +---+
    | 1 |
    +---+
    | 1 |
    +---+
    +---------------------------------------------------------------------------------+
    | Message                                                                         |
    +---------------------------------------------------------------------------------+
    | host1@alpha(slave:ONLINE) STATUS(OK), QOS=RO_RELAXED SSL.IN=false SSL.OUT=false |
    +---------------------------------------------------------------------------------+
    +---+
    | 1 |
    +---+
    | 1 |
    +---+
    +---------------------------------------------------------------------------------+
    | Message                                                                         |
    +---------------------------------------------------------------------------------+
    | host1@alpha(master:ONLINE) STATUS(OK), QOS=RO_RELAXED SSL.IN=false SSL.OUT=false |
    +---------------------------------------------------------------------------------+