C.2. Support Tools

C.2.1. Generating Diagnostic Information

To aid in the diagnosis of issues, a copy of the logs and diagnostic information will help the support team to identify and trace the problem. There are two methods of providing this information:

  • Using tpm diag

    The tpm diag command will collect the logs and configuration information from the active installation and generate a Zip file with the diagnostic information for all hosts within it. The command should be executed from the staging directory. Use tpm query staging to determine this directory:

    shell> tpm query staging
    tungsten@host1:/home/tungsten/tungsten-replicator-7.0.1-96
    shell> cd /home/tungsten/tungsten-replicator-7.0.1-96
    shell> ./tools/tpm diag

    The process will create a file called tungsten-diag-2014-03-20-10-21-29.zip, with the corresponding date and time information replaced. This file should be included in the reported support issue as an attachment.

    For a staging directory installation, tpm diag will collect together all of the information from each of the configured hosts in the cluster. For an INI file based installation, tpm diag will connect to all configured hosts if ssh is available. If a warning that ssh is not available is generated, tpm diag must be run individually on each host in the cluster.

  • Manually Collecting Logs

    If tpm diag cannot be used, or fails to return all the information, the information can be collected manually:

    1. Run tpm reverse on all the hosts in the cluster:

      shell> tpm reverse
    2. Collect the logs from each host. Logs are available within the service_logs directory. This contains symbolic links to the actual log files. The original files can be included within a tar archive by using the -h option. For example:

      shell> cd /opt/continuent
      shell> tar zcfh host1-logs.tar.gz ./service_logs

      The tpm reverse and log archives can then be submitted as attachments with the support query.

C.2.2. Generating Advanced Diagnostic Information

To aid in the diagnosis of difficult issues, below are tools and procedures to assist in the data collection.

Warning

ONLY excute the below commands and procedures when requested by Continuent support staff.

  • Manager Memory Usage Script

    We have provided a script to easily tell us how much memory a given manager is consuming.

    Place the script on all of your manager hosts (i.e. into the tungsten OS user home directory).

    Note

    The script assumes that 'cctrl' is in the path. If not, then change the script to provide a full path for cctrl.

    shell> su - tungsten
    shell> vi tungsten_manager_memory
    #!/bin/bash
    memval=`echo gc | cctrl | grep used | tail -1 | awk -F: '{print $2}' | tr -d ' |'`
    megabytes=`expr $memval / 1000000`
    timestamp=`date +"%F %T" | tr '-' '/'`
    echo "$timestamp | `hostname` | $megabytes MB"
    
    shell> chmod 750 tungsten_manager_memory
    shell> ./tungsten_manager_memory

    This script is ideally run from cron and the output redirected to time-stamped log files for later correlation with manager issues.

  • Manager Thread Dump Procedure

    This procedure creates a Manager memory thread dump for detailed analysis.

    Run this command on manager hosts when requested by Continuent support.

    This will append the detailed thread dump information to the log file named tmsvc.log in the /opt/continuent/tungsten/tungsten-manager/log directory.

    shell> su - tungsten
    shell> manager dump
    shell> tungsten_send_diag -f /opt/continuent/tungsten/tungsten-manager/log/tmsvc.log -c {case_number}
  • Manager Heap Dump Procedure

    This procedure creates a Manager memory heap dump for detailed analysis.

    Run this command on manager hosts when requested by Continuent support.

    This will create a file named {hostname}.hprof in the directory where you run it.

    shell> su - tungsten
    shell> jmap -dump:format=b,file=`hostname`.hprof `ps aux | grep JANINO | grep -v grep | awk '{print $2}'`
    shell> tungsten_send_diag -f `hostname`.hprof -c {case_number}
  • Configuring Connector Debug Logging

    This procedure allows the Connector to be configured for debug logging.

    Perform this procedure on Connector hosts when requested by Continuent support.

    Warning

    Enabling Connector debug logging will decrease performance dramatically. Disk writes will increase as will disk space consumption. Do not use in production environments unless instructed to do so by Continuent support. In any case, run in this mode for as short a period of time as possible - just long enough to gather the needed debug information. After that is done, disable debug logging.

    To enable debug logging, edit the Connector configuration file tungsten-connector/conf/log4j.properties and uncomment two lines. For example:

    shell> su - tungsten
    shell> vi /opt/continuent/tungsten/tungsten-connector/conf/log4j.properties
    Uncomment these two lines:
    
    #log4j.logger.org.continuent.myosotis=debug, stdout
    #log4j.additivity.org.continuent.myosotis=false
    
    so they look like this:
    
    log4j.logger.org.continuent.myosotis=debug, stdout
    log4j.additivity.org.continuent.myosotis=false
    
    shell> connector reconfigure

    Warning

    When disabling debug logging, DO NOT comment the lines out! Instead replace debug with info.

    To disable debug logging, edit the Connector configuration file tungsten-connector/conf/log4j.properties and change the keyword debug to info on for the single key log4j.logger.org.continuent.myosotis.

    For example, this is how it should look when the edit to disable is completed:

    shell> vi /opt/continuent/tungsten/tungsten-connector/conf/log4j.properties
    ...
    log4j.logger.org.continuent.myosotis=info, stdout
    ...
    shell> connector reconfigure
  • Configuring Connector Verbose Debug Logging

    This procedure allows the Connector to be configured for verbose debug logging (and we mean VERBOSE).

    Perform this procedure on Connector hosts when requested by Continuent support.

    Warning

    Enabling Connector verbose debug logging will decrease performance dramatically. Disk writes will increase as will disk space consumption. Do not use in production environments unless instructed to do so by Continuent support. In any case, run in this mode for as short a period of time as possible - just long enough to gather the needed debug information. After that is done, disable debug logging.

    To enable verbose debug logging (and we mean VERBOSE), edit the Connector log configuration file tungsten-connector/conf/log4j.properties and ADD two lines. For example:

    shell> su - tungsten
    shell> vi /opt/continuent/tungsten/tungsten-connector/conf/log4j.properties
    
    log4j.logger.org.continuent.myosotis.protocol.mysql.MySQLProtocolHandler=trace, stdout 
    log4j.additivity.org.continuent.myosotis.protocol.mysql.MySQLProtocolHandler=false
    
    shell> connector reconfigure

    Warning

    When disabling verbose debug logging, DO NOT comment the lines out! Instead replace trace with info.

    To disable verbose debug logging, edit the Connector log configuration file tungsten-connector/conf/log4j.properties and change the keyword trace to info for the single key log4j.logger.org.continuent.myosotis.protocol.mysql.MySQLProtocolHandler.

    For example, this is how it should look when the edit to disable is completed:

    shell> vi /opt/continuent/tungsten/tungsten-connector/conf/log4j.properties
    ...
    log4j.logger.org.continuent.myosotis.protocol.mysql.MySQLProtocolHandler=info, stdout 
    ...
    shell> connector reconfigure
  • Configuring Connector Debug Logging via the Manager

    This procedure allows the Manager to be configured to enable Connector-specific debug logging in the manager logs.

    Perform this procedure on Manager hosts when requested by Continuent support.

    Warning

    Enabling Manager-based Connector debug logging will decrease performance. Disk writes will increase as will disk space consumption. Do not use in production environments unless instructed to do so by Continuent support. In any case, run in this mode for as short a period of time as possible - just long enough to gather the needed debug information. After that is done, disable debug logging.

    To enable Manager-based Connector debug logging, edit the Manager log configuration file tungsten-manager/conf/log4j.properties and ADD two lines. For example:

    shell> su - tungsten
    shell> vi /opt/continuent/tungsten/tungsten-manager/conf/log4j.properties
    
    log4j.logger.com.continuent.tungsten.manager.router.gateway.RouterGatewayProxy=DEBUG, stdout 
    log4j.additivity.com.continuent.tungsten.manager.router.gateway.RouterGatewayProxy=false
    
    shell> manager restart

    Warning

    When disabling Manager-based Connector debug logging, DO NOT comment the lines out! Instead replace DEBUG with info.

    To disable Manager-based Connector debug logging, edit the Manager log configuration file tungsten-manager/conf/log4j.properties and change the keyword DEBUG to info for the single key log4j.logger.com.continuent.tungsten.manager.router.gateway.RouterGatewayProxy.

    For example, this is how it should look when the edit to disable is completed:

    shell> vi /opt/continuent/tungsten/tungsten-manager/conf/log4j.properties
    ...
    log4j.logger.com.continuent.tungsten.manager.router.gateway.RouterGatewayProxy=info, stdout 
    ...
    shell> manager restart

C.2.3. Using tungsten_upgrade_manager

tungsten_upgrade_manager is used to correct a cctrl display bug in the Manager that causes the useSSL value shown via cctrl> ls -l to be false when it should be true after an upgrade from v6 to v7.

Warning

Only use the tungsten_upgrade_manager command when instructed to do so by Continuent Support!