Jump to: navigation, search
MithiWiki Home > ConnectXf Home > ConnectXf Administration > Configuration > Logs in Monitoring


Troubleshooting Icon.png
Troubleshooting
Product ConnectXf
Version All
Applies to Administrators
Level Advanced



Logs

Overview

Connect Xf and Linux track all happenings in various log files, which enable audits and troubleshooting when the need arises. Essentially Connect Xf keeps track of the following in logs:

  • Transactions: These include mail flow, logins, access by users, modification operations done by users, etc.
  • Administration Trails: These include administration access, operations performed by administrators via the Application Manager or via the command line using sudo.
  • Monitoring trails: These include regular probes to check on the health of several parameters like CPU utilisation, memory utilisation, swap, storage, backups etc.

All the logs for the entire server working are stored under the /var/log folder.

What all logs are available

The table below identifies the critical logs of Connect Xf and their purpose

Transaction logging

/var/log/messages This has a record of firewall events, mail flow and access events, and several server related events.
/var/log/maillog This has a record of mail delivery events (post being queued)
/var/log/tomcat/catalina.out This has a record of all user access via Baya and administrator access via Application Manager.
/var/log/httpd/access_log This has a record of all access via the HTTPD service
/var/log/jabberd/jabber.log This has a record of all chat events by users.
/var/log/openfire/info*.log If openfire is used instead of Jabber, then this has a record of all chat events by users.
/var/log/mithi/mcs/prequeue.cpp.log This has a record of all mail flow stages before the mail is queued e.g. archiving, quota check, footer addition, etc.
/var/log/mithi/mcs/mithiMDC.cpp.log This has a record of all mail delivery events like filters, forwarding, routing, etc.

Administration Trails

/var/log/mithi/mcs/*.log This folder has a collection of logs for each command line utility available to the administrator (/mithi/mcs/bin/*.sh).
/var/log/mithi/mcs/sudolog.log This has a record of all access by sudo administrators and their activities.
/var/log/tomcat/catalina.out This has a record of all administrator access via Application Manager.
.bash_history This has a record of all commands run by the administrator during his command line login.

Monitoring Trails

/var/log/mithi/mcs/mcs_monitoring.log
/var/log/mithi/mcs/databackup*.log


Viewing logs

In MCS, all logs created are rotated daily and rotated logs are maintained for the last 30 days. The time of log rotation is configurable (by the system administrator).Therefore, today’s log will be available in <log>, yesterdays in <log>.1 etc., where <log> is the complete path of the log file. When debugging a problem, you will have to access the log of the day for which the problem has been reported.

  • To access the current log <log>, use an editor such as “vi” to access the contents.
  • To access older logs (logs in files <log>.1, <log>.2 etc), the log has to be first un-archived. This is because all the rotated logs are archived.
  • The steps involved in accessing an archived log are as follows:
  • Copy the archived log file to a temporary work folder (e.g. /root/mithiwork).
Warning
  • Un-ziping the log file in the logs folder may cause the log rotation to fail.
  • Unzip the log file using the following command
gunzip <log>
  • The contents of this file can now be accessed using an editor such as “vi”.
Warning
  • If you try to open a large log file on a live server, it can cause the server to slow down or freeze. If the file is large, it is better to take it offline and then open it.

Maintaining Logs

Using the Command line interface

Command sh setlogstorageduration.sh <no.of days>
Description For setting the log storage duration in terms of day
Command sh getlogstorageduration.sh
Description To get the current log storage duration

Log rotation

Different services and components of Mithi Connect Server create logs to track the exact behaviour of the system. They are useful in case of problems for analysis.These logs can take up a large space when the server is continuously up and running and viewing them will be difficult. Hence Mithi Connect Server performs 'Log Rotation'i.e. whenever the log file exceeds certain size limit, it is backed up with the filename having the date of backup.

The log files being rotated are as below:

System Logs

Mail log ( /var/log/maillog ) 
Mail Scanner ( /var/spool/qmailscan/qmail-queue.log ) 
Fresh ClamAV ( /var/log/clamav/freshclam.log ) 
Tomcat ( /var/log/tomcat/catalina.out )
Messages ( /var/log/messages ) 
IP-Tables ( /var/log/iptables.log ) 
Cron ( /var/log/cron ) 
PostgreSQL ( /var/log/pgsql.log ) 
Apache access log ( /var/log/httpd/access_log, /var/log/httpd/access.log )  
Apache error log ( /var/log/httpd/error_log, /var/log/httpd/error.log ) 
Mail scanner (/var/spool/qmailscan/qmail-queue.log)
OpenLDAP ( /var/log/openldap.log )
Qmail ( /var/qmail/qmailanalog/err/*.err ) 
/var/spool/qmailscan/qmail-queue.log 
/var/spool/qmailscan/quarantine.log 
/var/spool/qmailscan/qmail-mithi-queue.log 
/var/spool/qmailscan/mailstats.csv
/var/log/maillog
/var/log/messages
CourierIMAP ( /var/log/maillog )
Webcal ( Apache logs )
SpamAssassin ( /var/log/maillog )
MCS logs (The command line programs used in Mithi Connect Server, each log to their own log file,  
which are located in the following folders)
Event logs (/mithi/mcs/ modules /mithi-system/report/*.log)
Command program logs (/var/log/mithi/mcs/*)
  • These are also rotated on a daily basis:

The logs are rotated daily and maintained for 30 days. This interface sets a daily schedule to archive all the Mithi Connect Server related system logs. The daily archives are maintained for the last 30 days. Every day the oldest archive beyond 30 days is deleted. The logs are truncated, zipped and archived. The naming convention followed is .n.gz for logs which are compressed and archived. E.g. catalina.out.1.gz, catalina.2.gz, etc. For logs which are not compressed, the naming convention is similar except that the '.gz' is missing. The 'n' denotes a number going from 1-30 with '1' denoting the most recent.

Note

  • If at the scheduled time, the server is not running, the log rotation will not be done for that day. If you realize that the set schedule clashes with the time when the server is typically switched off, please adjust this schedule to match the time when the server is running so that the logs are archived daily. If the log files are not properly archived, they will grow too large and may even slow down the performance of certain parts of the server.

Using the Command line interface

Command sh getlogrotateschedule.sh
Description Get the log rotate schedule
Log /var/log/mithi/mcs/getlogrotateschedule.sh.log


Command sh setlogrotateschedule.sh [Start time (hours)]
Description Set the log rotate schedule
Log /var/log/mithi/mcs/setlogrotateschedule.sh.log

Log Density

Baya and Application Manager applications

The Baya application and the Application Manager generate logs in the /var/log/tomcat/catalina.out file. The level of logging to this file can be controlled to view dense logs with maximum information to help troubleshoot. Using this you can control log level for individual packages, classes or for the entire application. This can be controlled by using the following steps:

  • Sign into your server as the root user.
  • # cd /mithi/mcs/components/mithi-java-utils/conf/server/
  • # vi log4j.properties

The possible log levels are

DEBUG -> Most Dense with maximum information
INFO
WARN
ERROR
FATAL -> Least dense

This file enlists all the classes of the Mithi Web Mail application, which can be individually controlled to provide different levels of log outputs.

Example

to get the signature class to output dense logs, edit the following line:
log4j.logger.Signature=
to be 
log4j.logger.Signature=DEBUG
to change the log level of all the classes in the application, edit the following line:
log4j.rootLogger=INFO, A1
to be
log4j.rootLogger=DEBUG, A1
(INFO is the default log level for the application)
The possible log l
log4j.logger.Signature=DEBUG
To change the log level of all the classes in the application, edit the following line:
log4j.rootLogger=INFO, A1
to be 
log4j.rootLogger=DEBUG, A1
(INFO is the default log level for the application)
  • Save the file.
  • Wait for approximately a minute and monitor the tomcat log.

Note