To aid in the diagnosis of issues, a copy of the logs and diagnostic information will help the support team to identify and trace the problem. There are two methods of providing this information:
Using tpm diag
The tpm diag command will collect the logs and configuration information from the active installation and generate a Zip file with the diagnostic information for all hosts within it. The command should be executed from the staging directory. Use tpm query staging to determine this directory:
shell>tpm query stagingtungsten@host1:/home/tungsten/tungsten-clustering-5.4.1-41 shell>cd /home/tungsten/tungsten-clustering-5.4.1-41shell>./tools/tpm diag
The process will create a file called
tungsten-diag-2014-03-20-10-21-29.zip,
with the corresponding date and time information replaced. This file
should be included in the reported support issue as an attachment.
For a staging directory installation, tpm diag will collect together all of the information from each of the configured hosts in the cluster. For an INI file based installation, tpm diag will connect to all configured hosts if ssh is available. If a warning that ssh is not available is generated, tpm diag must be run individually on each host in the cluster.
Manually Collecting Logs
If tpm diag cannot be used, or fails to return all the information, the information can be collected manually:
Run tpm reverse on all the hosts in the cluster:
shell> tpm reverse
Collect the logs from each host. Logs are available within the
service_logs directory. This contains
symbolic links to the actual log files. The original files can
be included within a tar archive by using the
-h option. For example:
shell>cd /opt/continuentshell>tar zcfh host1-logs.tar.gz ./service_logs
The tpm reverse and log archives can then be submitted as attachments with the support query.
To aid in the diagnosis of difficult issues, below are tools and procedures to assist in the data collection.
ONLY excute the below commands and procedures when requested by Continuent support staff.
We have provided a script to easily tell us how much memory a given manager is consuming.
Place the script on all of your manager hosts (i.e. into the tungsten OS user home directory).
The script assumes that 'cctrl' is in the path. If not, then change the script to provide a full path for cctrl.
shell>su - tungstenshell>vi tungsten_manager_memory#!/bin/bash memval=`echo gc | cctrl | grep used | tail -1 | awk -F: '{print $2}' | tr -d ' |'` megabytes=`expr $memval / 1000000` timestamp=`date +"%F %T" | tr '-' '/'` echo "$timestamp | `hostname` | $megabytes MB"shell>chmod 750 tungsten_manager_memoryshell>./tungsten_manager_memory
This script is ideally run from cron and the output redirected to time-stamped log files for later correlation with manager issues.
This procedure creates a Manager memory thread dump for detailed analysis.
Run this command on manager hosts when requested by Continuent support.
This will append the detailed thread dump information to the log file
named tmsvc.log in the
/opt/continuent/tungsten/tungsten-manager/log
directory.
shell>su - tungstenshell>manager dumpshell>tungsten_send_diag -f /opt/continuent/tungsten/tungsten-manager/log/tmsvc.log -c {case_number}
This procedure creates a Manager memory heap dump for detailed analysis.
Run this command on manager hosts when requested by Continuent support.
This will create a file named {hostname}.hprof in
the directory where you run it.
shell>su - tungstenshell>jmap -dump:format=b,file=`hostname`.hprof `ps aux | grep JANINO | grep -v grep | awk '{print $2}'`shell>tungsten_send_diag -f `hostname`.hprof -c {case_number}
This procedure allows the Connector to be configured for debug logging.
Perform this procedure on Connector hosts when requested by Continuent support.
Enabling Connector debug logging will decrease performance dramatically. Disk writes will increase as will disk space consumption. Do not use in production environments unless instructed to do so by Continuent support. In any case, run in this mode for as short a period of time as possible - just long enough to gather the needed debug information. After that is done, disable debug logging.
To enable debug logging, edit the Connector configuration file
tungsten-connector/conf/log4j.properties and
uncomment two lines. For example:
shell>su - tungstenshell>vi /opt/continuent/tungsten/tungsten-connector/conf/log4j.propertiesUncomment these two lines:#log4j.logger.org.continuent.myosotis=debug, stdout #log4j.additivity.org.continuent.myosotis=falseso they look like this:log4j.logger.org.continuent.myosotis=debug, stdout log4j.additivity.org.continuent.myosotis=falseshell>connector reconfigure
When disabling debug logging, DO NOT comment the lines out! Instead
replace debug with info.
To disable debug logging, edit the Connector configuration file
tungsten-connector/conf/log4j.properties and change
the keyword debug to info on for
the single key log4j.logger.org.continuent.myosotis.
For example, this is how it should look when the edit to disable is completed:
shell>vi /opt/continuent/tungsten/tungsten-connector/conf/log4j.properties... log4j.logger.org.continuent.myosotis=info, stdout ... shell>connector reconfigure
This procedure allows the Connector to be configured for verbose debug logging (and we mean VERBOSE).
Perform this procedure on Connector hosts when requested by Continuent support.
Enabling Connector verbose debug logging will decrease performance dramatically. Disk writes will increase as will disk space consumption. Do not use in production environments unless instructed to do so by Continuent support. In any case, run in this mode for as short a period of time as possible - just long enough to gather the needed debug information. After that is done, disable debug logging.
To enable verbose debug logging (and we mean VERBOSE), edit the
Connector log configuration file
tungsten-connector/conf/log4j.properties and ADD
two lines. For example:
shell>su - tungstenshell>vi /opt/continuent/tungsten/tungsten-connector/conf/log4j.propertieslog4j.logger.org.continuent.myosotis.protocol.mysql.MySQLProtocolHandler=trace, stdout log4j.additivity.org.continuent.myosotis.protocol.mysql.MySQLProtocolHandler=falseshell>connector reconfigure
When disabling verbose debug logging, DO NOT comment the lines out!
Instead replace trace with info.
To disable verbose debug logging, edit the Connector log configuration file
tungsten-connector/conf/log4j.properties and change
the keyword trace to info for
the single key
log4j.logger.org.continuent.myosotis.protocol.mysql.MySQLProtocolHandler.
For example, this is how it should look when the edit to disable is completed:
shell>vi /opt/continuent/tungsten/tungsten-connector/conf/log4j.properties... log4j.logger.org.continuent.myosotis.protocol.mysql.MySQLProtocolHandler=info, stdout ... shell>connector reconfigure
This procedure allows the Manager to be configured to enable Connector-specific debug logging in the manager logs.
Perform this procedure on Manager hosts when requested by Continuent support.
Enabling Manager-based Connector debug logging will decrease performance. Disk writes will increase as will disk space consumption. Do not use in production environments unless instructed to do so by Continuent support. In any case, run in this mode for as short a period of time as possible - just long enough to gather the needed debug information. After that is done, disable debug logging.
To enable Manager-based Connector debug logging, edit the
Manager log configuration file
tungsten-manager/conf/log4j.properties and ADD
two lines. For example:
shell>su - tungstenshell>vi /opt/continuent/tungsten/tungsten-manager/conf/log4j.propertieslog4j.logger.com.continuent.tungsten.manager.router.gateway.RouterGatewayProxy=DEBUG, stdout log4j.additivity.com.continuent.tungsten.manager.router.gateway.RouterGatewayProxy=falseshell>manager restart
When disabling Manager-based Connector debug logging, DO NOT comment the lines out!
Instead replace DEBUG with info.
To disable Manager-based Connector debug logging, edit the Manager log configuration file
tungsten-manager/conf/log4j.properties and change
the keyword DEBUG to info for
the single key
log4j.logger.com.continuent.tungsten.manager.router.gateway.RouterGatewayProxy.
For example, this is how it should look when the edit to disable is completed:
shell>vi /opt/continuent/tungsten/tungsten-manager/conf/log4j.properties... log4j.logger.com.continuent.tungsten.manager.router.gateway.RouterGatewayProxy=info, stdout ... shell>manager restart
tungsten_upgrade_manager is used to correct a cctrl display bug in the Manager that causes the useSSL value shown
via cctrl> ls -l to be false when it should be true after an upgrade from v6 to v7.
Only use the tungsten_upgrade_manager command when instructed to do so by Continuent Support!