Audit Logging in AstroGrid-D

AstroGrid-D uses the GRAM Audit module to achieve basic job audit logging. (Note: gsissh activity can not be monitored.)

Introduction

The job data is first stored in a PostgreSQL database and then uploaded to Stellaris by a database trigger. To enable audit logging, two steps have to be completed:

1. On every resource, the GRAM audit logging must be enabled (instructions follow below). GRAM Audit Logging is used for host monitoring of globusrun-ws jobs.

2. Once per site (institute) a PostgreSQL database dedicated to audit logging must be set up. See second part of this manual.

Requirements for audit logging are:

GT 4.0.5 or above (test: $GLOBUS_LOCATION/bin/globus-version)
A PostgreSQL database which is accessible for globus user (firewall!) and in which You can define Perl functions (test: createdb auditDatabase; createlang plperlu auditDatabase)

1. Configuration of grid resources for audit logging

Only Globus configuration files need be altered.

Edit (as user globus) the $GLOBUS_LOCATION/container-log4j.properties, which activates GRAM audit logging, and add these lines:


# GRAM				AUDIT
log4j.category.org.globus.exec.service.exec.StateMachine.audit=DEBUG,				AUDIT
log4j.appender.AUDIT=org.globus.exec.utils.audit.AuditDatabaseAppender
log4j.appender.AUDIT.layout=org.apache.log4j.PatternLayout
log4j.additivity.org.globus.exec.service.exec.StateMachine.audit=false

Edit (as user globus) the file $GLOBUS_LOCATION/etc/gram-service/jndi-config.xml which contains the information for connecting the audit database. In the <resource name="auditDatabaseConfiguration"> element change the host name after jdbc:postgresql:// and the password as shown below (the changes are italic).


<resource
...>
  <resourceParams> ...
    <parameter>

<name>url</name><value>jdbc:postgresql://servername/auditDatabase</value> </parameter>

<parameter><name>user</name><value>globus</value></parameter>

<parameter><name>password</name><value>foo</value></parameter> ... </resourceParams> </resource>

where servername is the machine that runs the central database (at AIP it is is.astrogrid-d.org, or astrodata01.gac-grid.org), user name and password are defined according to the first line in update_stellaris.sql. If the database is running locally, 'localhost' doesn't need to be changed.

2. Configuration of a central site database

One audit Database can be used for logging of one or more Grid containers, e.g. a whole cluster or institute.

Install the database table (following http://www.globus.org/toolkit/docs/4.0/execution/wsgram/WS_GRAM_Audit_Logging.html .

Download the file update_stellaris.sql. As user postgres:


createdb auditDatabase

createlang plperlu auditDatabase

psql -d auditDatabase -f update_stellaris.sql

If createlang fails, add the package "postgresql.pl" to your postgres installation.

If createlang still fails, reinstall at first PostgreSQL database version 8.2 from sources:

./configure --with-perl; make; sudo make install.

If PostgreSQL version 7 is used, replace $update_stellaris$ in update_stellaris.sql by single quotes "'".

The first line of Connection Settings in /var/lib/pgsql/data/postgresql.conf has to be changed:

# Connection - Settings -
listen_addresses = '*'
...

The /var/lib/pgsql/data/pg_hba.conf is updated

# TYPE DATABASE        USER      IP-ADDRESS      IP-MASK          METHOD
host   auditDatabase   globus   <IP-Range>      255.255.255.255   md5
...

where <IP-Range> represents all the resources that will connect to the server, e.g. a range 141.33.4.161/170.

Then start the database with


sudo /etc/init.d/postgresql restart

, and restart the Globus container with


sudo /etc/init.d/globus restart

.

Note that each time the PostgreSQL database is restarted, all the Globus containers of all the resources connecting to the database must be restarted, in order to re-establish the connection.

3. Pre WS GRAM resource configuration for audit logging

Pre WS GRAM Audit Logging is used for host monitoring of globus-job-run jobs.

Add to $GLOBUS_LOCATION/log4j.properties, these lines to activate GRAM audit logging:


# GRAM AUDIT

log4j.category.org.globus.exec.service.exec.StateMachine.audit=DEBUG, AUDIT

log4j.appender.AUDIT=org.globus.exec.utils.audit.AuditDatabaseAppender

log4j.appender.AUDIT.layout=org.apache.log4j.PatternLayout

log4j.additivity.org.globus.exec.service.exec.StateMachine.audit=false

Create a directory /tmp/globus, and edit the file $GLOBUS_LOCATION/etc/globus-job-manager.conf.


-home "/work1/globus/gt406"

-globus-gatekeeper-host dublin.aip.de

-globus-gatekeeper-port 2119

-globus-gatekeeper-subject "/C=DE/O=GermanGrid/OU=AIP/CN=host/dublin.aip.de"

-globus-host-cputype x86_64

-globus-host-manufacturer unknown

-globus-host-osname Linux

-globus-host-osversion 2.6.18-8.1.14.el5xen

-globus-toolkit-version 4.0.6

-save-logfile on_error

-state-file-dir /work1/globus/gt406/tmp/gram_job_state

-machine-type unknown

-audit-directory /tmp/globus

By default, You need only to edit the last line.

Then restart the Globus container with sudo /etc/init.d/globus restart.

To put the files from the /tmp/globus (permissions drws-wsrwx, e.g. chmod u+s /tmp/globus) directory to Stellaris, use the Perl script update_stellaris.cron (download here) as a cron job.

Test

From now on, the jobs are shown in the Grid timeline.