To aid in the diagnosis of issues, a copy of the logs and diagnostic information will help the support team to identify and trace the problem. There are two methods of providing this information:
Using tpm diag
The tpm diag command will collect the logs and configuration information from the active installation and generate a Zip file with the diagnostic information for all hosts within it. The command should be executed from the staging directory. Use tpm query staging to determine this directory:
shell>tpm query stagingtungsten@host1:/home/tungsten/tungsten-replicator-6.1.25-6 shell>cd /home/tungsten/tungsten-replicator-6.1.25-6shell>./tools/tpm diag
The process will create a file called
tungsten-diag-2014-03-20-10-21-29.zip,
with the corresponding date and time information replaced. This file
should be included in the reported support issue as an attachment.
For a staging directory installation, tpm diag will collect together all of the information from each of the configured hosts in the cluster. For an INI file based installation, tpm diag will connect to all configured hosts if ssh is available. If a warning that ssh is not available is generated, tpm diag must be run individually on each host in the cluster.
Manually Collecting Logs
If tpm diag cannot be used, or fails to return all the information, the information can be collected manually:
Run tpm reverse on all the hosts in the cluster:
shell> tpm reverse
Collect the logs from each host. Logs are available within the
service_logs directory. This contains
symbolic links to the actual log files. The original files can
be included within a tar archive by using the
-h option. For example:
shell>cd /opt/continuentshell>tar zcfh host1-logs.tar.gz ./service_logs
The tpm reverse and log archives can then be submitted as attachments with the support query.
To aid in the diagnosis of difficult issues, below are tools and procedures to assist in the data collection.
ONLY excute the below commands and procedures when requested by Continuent support staff.
We have provided a script to easily tell us how much memory a given manager is consuming.
Place the script on all of your manager hosts (i.e. into the tungsten OS user home directory).
The script assumes that 'cctrl' is in the path. If not, then change the script to provide a full path for cctrl.
shell>su - tungstenshell>vi tungsten_manager_memory#!/bin/bash memval=`echo gc | cctrl | grep used | tail -1 | awk -F: '{print $2}' | tr -d ' |'` megabytes=`expr $memval / 1000000` timestamp=`date +"%F %T" | tr '-' '/'` echo "$timestamp | `hostname` | $megabytes MB"shell>chmod 750 tungsten_manager_memoryshell>./tungsten_manager_memory
This script is ideally run from cron and the output redirected to time-stamped log files for later correlation with manager issues.
This procedure creates a Manager memory thread dump for detailed analysis.
Run this command on manager hosts when requested by Continuent support.
This will append the detailed thread dump information to the log file
named tmsvc.log in the
/opt/continuent/tungsten/tungsten-manager/log
directory.
shell>su - tungstenshell>manager dumpshell>tungsten_send_diag -f /opt/continuent/tungsten/tungsten-manager/log/tmsvc.log -c {case_number}
This procedure creates a Manager memory heap dump for detailed analysis.
Run this command on manager hosts when requested by Continuent support.
This will create a file named {hostname}.hprof in
the directory where you run it.
shell>su - tungstenshell>jmap -dump:format=b,file=`hostname`.hprof `ps aux | grep JANINO | grep -v grep | awk '{print $2}'`shell>tungsten_send_diag -f `hostname`.hprof -c {case_number}
This procedure allows the Connector to be configured for debug logging.
Perform this procedure on Connector hosts when requested by Continuent support.
Enabling Connector debug logging will decrease performance dramatically. Disk writes will increase as will disk space consumption. Do not use in production environments unless instructed to do so by Continuent support. In any case, run in this mode for as short a period of time as possible - just long enough to gather the needed debug information. After that is done, disable debug logging.
To enable debug logging, edit the Connector configuration file tungsten-connector/conf/log4j.properties
and turn Connector logging from INFO to DEBUG (or to TRACE for verbose logging):
shell>su - tungstenshell>vi /opt/continuent/tungsten/tungsten-connector/conf/log4j.properties# Enable debug for the connector only logger.Connector.name=com.continuent.tungsten.connector # WAS: logger.Connector.level=INFO logger.Connector.level=DEBUG logger.Connector.additivity=false shell>connector reconfigure
To disable debug logging, edit the Connector configuration file
tungsten-connector/conf/log4j.properties and revert the change
from DEBUG to INFO.
tungsten_upgrade_manager is used to correct a cctrl display bug in the Manager that causes the useSSL value shown
via cctrl> ls -l to be false when it should be true after an upgrade from v6 to v7.
Only use the tungsten_upgrade_manager command when instructed to do so by Continuent Support!