The key fact that must be understood in order that the operation of the rules can be grasped is that the rules are primarily driven by notification events which are, in turn, generated by the cluster monitoring subsystem or, in some cases, by threads internal to the manager that act as monitors for specific conditions.
As a general rule, notifications are generated in the monitoring subsystem, on each host, independently of the other hosts, and sent, by each manager, to all of the managers in the group, including itself, via group communications and the notifications are then routed to the rules engine in each manager.
These notifications are sent via a messaging protocol that guarantees that the monitoring events are received by all managers in the group in exactly the same order. This ordering is a critical part of correct rules operation for if notifications are received in a different order and used to drive the rules evaluation, the rules on each manager could arrive at a different state/conclusion based on the ordering.
The cluster monitoring subsystem driven by a set of checker threads that are configured from files found in tungsten-manager/conf and are named checker*.properties. There are currently four of these checkers, with their corresponding checker configurations stored in the following tungsten-manager/conf/checker*.properties files:
checker.heartbeat.properties - runs a thread the generates a manager heartbeat notification for the manager. If a manager goes away from the group, either by crashing, being stopped, or having a network outage, other managers won't 'see' that manager's heartbeat and will use that as a clue that there may be something amiss on that manager's host.
checker.instrumentation.properties - TBD
checker.mysqlserver.properties - polls the local mysql server for
liveness/state. This checker attempts to establish a connection with the
local MySQL server and to execute the query that is found in
tungsten-manager/conf/mysql_checker_query.sql. The
checker then evaluates the success or failure of the connect attempt and
subsequent query execution and establishes the state of the MySQL server
based on that evaluation. Because database server state is so critical
to the operation of the cluster, particularly when it comes to
availability of database resources, this particular checker uses a very
fine-grained and configurable process for evaluating the state.
checker.tungstenreplicator.properties - polls the local Tungsten replicator for liveness/state. This checker connects to the local replicator via the JMX interfaces and queries the replicator for its current state. If, in the process of connecting to the replicator, a connection cannot be established because the replicator is not running, the checker simply returns the STOPPED state.
Let's look at the contents of one of these files - checker.mysqlserver.properties:
# AUTO-GENERATED: 2017-03-22T10:43:25-07:00 ##################################### # CHECKER.MYSQLSERVER.PROPERTIES # ##################################### requiresProxy=true name=mysql_response class=com.continuent.tungsten.monitor.checkers.JDBCMySQLDatabaseServerChecker # delay between each monitoring run - default 3000ms frequency=3000 # connection will be renewed after this period reconnectAfter=30000 serverName=viveka host=viveka vendor=mysql port=3306 driver=org.drizzle.jdbc.DrizzleDriver url=jdbc:mysql:thin://viveka:3306/tungsten username=tungsten password=passw0rd query=select 1 queryTimeout=5 queryFileName=mysql_checker_query.sql
The key thing to understand about this configuration file is that the value for the class property indicates which Java class the manager will load and that class, when alive, will then be configured with the additional properties that can be seen in this file.
Every checker will have a property frequency which indicates how often the checker thread will become active.
Then, since this checker is for MySQL server, you can see that the configuration file indicates which jdbc driver to use, which port to use to connect directly to MySQL, username, password etc. for the connection as well as the name of a file that determines what SQL the checker will run to check for the MySQL server liveness.