JULI
Previous versions of Tomcat (till 5.x) use Apache common logging services for generating logs. A major disadvantage with this logging mechanism is that it can handle only single JVM configuration and makes it difficult to configure separate logging for each class loader for independent application. In order to resolve this issue, Tomcat developers have introduced a separate API for Tomcat 6 version, that comes with the capability of capturing each class loader activity in the Tomcat logs. It is based on java.util.logging framework.
By default, Tomcat 7 uses its own Java logging API to implement logging services. This is also called as JULI. This API can be found in TOMCAT_HOME/bin of the Tomcat 7 directory structures (tomcat-juli.jar). The following screenshot shows the directory structure of the bin directory where tomcat-juli.jar is placed. JULI also provides the feature for custom logging for each web application, and it also supports private per-application logging configurations. With the enhanced feature of separate class loader logging, it also helps in detecting memory issues while unloading the classes at runtime.
For more information on JULI and the class loading issue, please refer to http://tomcat.apache.org/tomcat-7.0-doc/logging.html and http://tomcat.apache.org/tomcat-7.0-doc/class-loader-howto.html respectively.
Loggers, appenders, and layouts
There are some important components of logging which we use at the time of implementing the logging mechanism for applications. Each term has its individual importance in tracking the events of the application. Let’s discuss each term individually to find out their usage:
- Loggers:It can be defined as the logical name for the log file. This logical name is written in the application code. We can configure an independent logger for each application.
- Appenders: The process of generation of logs are handled by appenders. There are many types of appenders, such as FileAppender, ConsoleAppender, SocketAppender, and so on, which are available in log4j. The following are some examples of appenders for log4j:
log4j.appender.CATALINA=org.apache.log4j.DailyRollingFileAppender log4j.appender.CATALINA.File=${catalina.base}/logs/catalina.out log4j.appender.CATALINA.Append=true log4j.appender.CATALINA.Encoding=UTF-8
The previous four lines of appenders define the DailyRollingFileAppender in log4j, where the filename is catalina.out. These logs will have UTF-8 encoding enabled.
If log4j.appender.CATALINA.append=false, then logs will not get updated in the log files.
# Roll-over the log once per day log4j.appender.CATALINA.DatePattern=’.’dd-MM-yyyy’.log’ log4j.appender.CATALINA.layout = org.apache.log4j.PatternLayout log4j.appender.CATALINA.layout.ConversionPattern = %d [%t] %-5p %c- %m%n
The previous three lines of code show the roll-over of log once per day.
- Layout: It is defined as the format of logs displayed in the log file. The appender uses layout to format the log files (also called as patterns). The highlighted code shows the pattern for access logs:
Loggers, appenders, and layouts together help the developer to capture the log message for the application event.
Types of logging in Tomcat 7
We can enable logging in Tomcat 7 in different ways based on the requirement. There are a total of five types of logging that we can configure in Tomcat, such as application, server, console, and so on. The following figure shows the different types of logging for Tomcat 7. These methods are used in combination with each other based on environment needs. For example, if you have issues where Tomcat services are not displayed, then console logs are very helpful to identify the issue, as we can verify the real-time boot sequence. Let’s discuss each logging method briefly.
Application log
These logs are used to capture the application event while running the application transaction. These logs are very useful in order to identify the application level issues. For example, suppose your application performance is slow on a particular transition, then the details of that transition can only be traced in application log. The biggest advantage of application logs is we can configure separate log levels and log files for each application, making it very easy for the administrators to troubleshoot the application.
Log4j is used in 90 percent of the cases for application log generation.
Server log
Server logs are identical to console logs. The only advantage of server logs is that they can be retrieved anytime but console logs are not available after we log out from the console.
Console log
This log gives you the complete information of Tomcat 7 startup and loader sequence. The log file is named as catalina.out and is found in TOMCAT_HOME/logs. This log file is very useful in checking the application deployment and server startup testing for any environment. This log is configured in the Tomcat file catalina.sh, which can be found in TOMCAT_HOME/bin.
The previous screenshot shows the definition for Tomcat logging. By default, the console logs are configured as INFO mode.
There are different levels of logging in Tomcat such as WARNING, INFORMATION, CONFIG, and FINE.
The previous screenshot shows the Tomcat log file location, after the start of Tomcat services.
The previous screenshot shows the output of the catalina.out file, where Tomcat services are started in 1903 ms.
Access log
Access logs are customized logs, which give information about the following:
- Who has accessed the application
- What components of the application are accessed
- Source IP and so on
These logs play a vital role in traffic analysis of many applications to analyze the bandwidth requirement and also helps in troubleshooting the application under heavy load. These logs are configured in server.xml in TOMCAT_HOME/conf. The following screenshot shows the definition of access logs. You can customize them according to the environment and your auditing requirement.
Let’s discuss the pattern format of the access logs and understand how we can customize the logging format:
- Class Name: This parameter defines the class name used for generation of logs. By default, Apache Tomcat 7 uses the org.apache.catalina.valves.AccessLogValve class for access logs.
- Directory: This parameter defines the directory location for the log file. All the log files are generated in the log directory—TOMCAT_HOME/logs—but we can customize the log location based on our environment setup and then update the directory path in the definition of access logs.
- Prefix: This parameter defines the prefix of the access log filename, that is, by default, access log files are generated by the name localhost_access_log.yy-mm-dd.txt.
- Suffix: This parameter defines the file extension of the log file. Currently it is in .txt format.
- Pattern: This parameter defines the format of the log file. The pattern is a combination of values defined by the administrator, for example, %h = remote host address. The following screenshot shows the default log format for Tomcat 7. Access logs show the remote host address, date/time of request, method used for response, URI mapping, and HTTP status code.
In case you have installed the web traffic analysis tool for application, then you have to change the access logs to a different format.
Host manager
These logs define the activity performed using Tomcat Manager, such as various tasks performed, status of application, deployment of application, and lifecycle of Tomcat. These configurations are done on the logging.properties, which can be found in TOMCAT_HOME/conf.
The previous screenshot shows the definition of host, manager, and host-manager details. If you see the definitions, it defines the log location, log level, and prefix of the filename.
In logging.properties, we are defining file handlers and appenders using JULI.
The log file for manager looks similar to the following:
I28 Jun, 2011 3:36:23 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host ‘localhost’ 28 Jun, 2011 3:37:13 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host ‘localhost’ 28 Jun, 2011 3:37:42 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: undeploy: Undeploying web application at ‘/sample’ 28 Jun, 2011 3:37:43 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host ‘localhost’ 28 Jun, 2011 3:42:59 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host ‘localhost’ 28 Jun, 2011 3:43:01 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host ‘localhost’ 28 Jun, 2011 3:53:44 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host ‘localhost’
Types of log levels in Tomcat 7
There are seven levels defined for Tomcat logging services (JULI). They can be set based on the application requirement. The following figure shows the sequence of log levels for JULI:
Every log level in JULI had its own functionality. The following table shows the functionality of each log level in JULI:
| Log level |
Description |
| SEVERE(highest) |
Captures exception and Error |
| WARNING |
Warning messages |
| INFO |
Informational message, related to
server activity |
| CONFIG |
Configuration message |
| FINE |
Detailed activity of server transaction
(similar to debug) |
| FINER |
More detailed logs than FINE |
| FINEST(least) |
Entire flow of events (similar to trace) |
For example, let’s take an appender from logging.properties and find out the log level used; the first log appender for localhost is using FINE as the log level, as shown in the following code snippet:
localhost.org.apache.juli.FileHandler.level = FINE localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs localhost.org.apache.juli.FileHandler.prefix = localhost.
The following code shows the default file handler configuration for logging in Tomcat 7 using JULI. The properties are mentioned and log levels are highlighted:
############################################################ # Facility specific properties. # Provides extra control for each logger. ############################################################ org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.FileHandler org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager] .level = INFO org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager] .handlers = 3manager.org.apache.juli.FileHandler org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host- manager].level = INFO org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host- manager].handlers = 4host-manager.org.apache.juli.FileHandler
(For more resources on Apache, see here.)
Log4j
Log4j is the project run by THe Apache Software Foundation. This project helps in enabling the logs at the various levels of the server and the application.
The major advantage of log4j is manageability. It provides the developer a freedom to change the log level at the configuration file level. Also, you can enable/disable logs at the configuration level, so there is no need to change the code. We can customize the log pattern based on the application, separately. Log4j has six log levels. The following figure shows the different types of log levels in log4j.
Log level for log4j
Every log level in log4j had its own functionality. The following table shows the functionality of each log level in log4j:
| Log level |
Description |
| OFF |
This level is set when you want logging to be set as false (Stopped logging). |
| FATAL |
This log level will print the severe errors that cause premature termination. |
| ERROR |
This log level is used to capture runtime errors or unexpected conditions.
Expect these to be immediately visible on a status console. |
| WARN |
This level is used in the previous version.
It gives you almost errors, other runtime situations that are
undesirable or unexpected, but not necessarily wrong.
Expect these to be immediately visible on a status console. |
| INFO |
This log level will define the interesting runtime events (startup/shutdown).
It is best practice to put the logs at INFO level. |
| DEBUG |
Detailed information on the flow through the system is defined in this level. |
| TRACE |
This log level captures all the events in the system and application,
you can say everything. |
How to use log4j
Following are the steps to be performed to use log4j:
- Download apache-log4j-1.2.X.tar.gz from its official URL http://logging.apache.org/log4j/1.2/download.html, where X is the minor version.
- Unzip the folder and place the log4j.jar in the lib for TOMCAT_HOME/lib and delete juli*.jar from lib.
- Delete logging.properties from TOMCAT_HOME/CONF.
- Create a file log4j.properties in TOMCAT_HOME/CONF and define the log appenders for the Tomcat instance. The following screenshot shows the appenders for catalina.out. Also, the highlighted code shows the roll-over of logs per day.
You can customize the log rotation based on size, day, hour, and so on, using the previous log4j appenders marked in bold.
- Restart Tomcat services.
Important tip for production environment
DEBUG and TRACE modes should only be enabled in production. In case of troubleshooting, the ideal mode is INFO (DEBUG and TRACE generate heavy logging and also affect the server performance).
Appenders should be enabled everyday in a production environment. This helps the administrator to perform log analysis very easily (file size is less).
Log level mapping
Till now, we have discussed about the various log levels for JULI and log4j. Let us do a quick log level mapping for JULI and log4j. The following table shows the one-to-one mapping for log4j and JULI:
| Log level in JULI |
Log level in log4j |
| SEVERE |
FATAL, ERROR |
| WARNING |
WARN |
| INFO |
INFO |
| CONFIG |
NA |
| FINE |
DEBUG |
| FINER |
DEBUG |
| FINEST |
TRACE |
Values for Tomcat 7
Values are defined as identifiers which change the pattern of the string in the log . Suppose you want to know the IP address of remote host, which has accessed the website, then you add the combination of the following values mentioned in the log appenders. For example, let’s customize the access logs for Tomcat 7. By default, access logs for Tomcat are defined as the follows:
We want to change the log pattern to show the time taken to process the request. We have to add the %T in the patterns. The changed code is shown as follows:
The following table shows the values used in Tomcat 7 for log pattern customization:
| Values |
Description |
| %a |
Remote IP address |
| %A |
Local IP address |
| %b |
Bytes sent, excluding HTTP headers, or ” if zero |
| %B |
Bytes sent, excluding HTTP headers |
| %h |
Remote hostname (or IP address if enableLookups for the connector is false) |
| %H |
Request protocol |
| %l |
Remote logical username from identd |
| %m |
Request method (GET, POST, and so on) |
| %p |
Local port on which this request was received |
| %q |
Query string (prepended with a ‘?’ if it exists) |
| %r |
First line of the request (method and request URI) |
| %s |
HTTP status code of the response |
| %S |
User session ID |
| %t |
Date and time, in Common Log format |
| %u |
Remote user that was authenticated (if any) |
| %U |
Requested URL path |
| %v |
Local server name |
| %D |
Time taken to process the request, in milliseconds |
| %T |
Time taken to process the request, in seconds |
| %I |
Current request thread name (can compare later with stack traces) |
Log analysis
Log analysis is a very important and tricky issue, which needs to be handled with a lot of care. If you overlook a few lines, then you will never be able to find the root cause of the issue. Some of the best practices which need to be kept in consideration while doing the log analysis are mentioned as follows:
- Check the logs of the last 1 hour from the issue
- Always go to the first exception in the logs when the error has started
- Always keep in mind that issues are not caused due to malfunction of Tomcat, also check the other infrastructure resources
In non-DOS operating systems (Linux, Unix, Ubuntu, and so on), there are two utilities which are very useful in log analysis, grep and awk. Let’s discuss grep and awk utilities briefly:
- grep: This utility prints the lines which match the string searched.grep Error catalina.logThe previous command is an example of the grep command for searching the word “error” in the file catalina.log and display the lines which contain the word “error”.
- awk: This command is used for pattern scanning. Suppose we want to print 10 columns in the entire data file, then this command is very useful. The following screenshot shows the output of the command when run for the /opt directory.find “location of directory “ -type f -size +10000k -exec ls -lh {} \; | awk ‘{ print $9 “: ” $5 }’ find “/opt“ -type f -size +10000k -exec ls -lh {} \; | awk ‘{ print $9 “: ” $5 }’
Helpful commands for log analysis
Administrators are looking for shortcut commands to do their work efficiently. The following are some useful commands that I have collected during log analysis:
The following commands are used for searching big log files. Sometimes in a production environment, we get alerts for disk out of space. The following commands can be used:
- Finding large files and directories in Linux:
find “location of directory “; type f -size +10000k -exec ls -lh {} \; | awk ‘{ print $9 “: ” $5 }’
- Finding directories with size over 100MB:
find / -type d -size +100000k
- Sort directories as per size using du:
du –max-depth=1 -m | sort -n -r
du -sh folder_name du -ch folder_name du -csh folder_name
- The following command is used for truncating huge log files on the live system (log rotation can be done without recycle of services):
cat /dev/null > file_name
The following mentioned commands are used for searching the string in different files:
grep ERROR log_file
- Last 200 lines in log file:
tail -200 log_file
- Current logs to be updated
tail -f log_file
balajiommudali 