Page History

Lumeta Asset Manager logs are useful for the purposes of auditing, debugging, and notifying.

1. Auditing - Auditing logs help Lumeta customers with system security by providing a record of system access and changes to system configuration, among other things. The files can be sent to an external syslog server where they can to be monitored in near real-time. An administrator or security officer can be notified of any unexpected events. The files can be used after an event as forensic evidence of nefarious activity.
2. Debugging - Lumeta Asset Manager services produce debugging logs that Lumeta Support, quality engineers (QE) and developers can use to determine the root cause of bugs and anomalous behaviourbehavior.
3. Notification - The Lumeta CEF log contains notifications of scanning and configuration events in "Common Event Format." The log can be sent to an external syslog server where it can to be monitored and made actionable by third-party software. Lumeta users Users can select which notifications they want to send to the CEF log on the Settings > Lumeta GUI's Settings > Asset Manager    Systems page of the GUI. 

Files and Directories in /var/log

Log files specific to Lumeta Asset Manager are in bold type.

 Apache

General httpd logs in error_log and access_log. Security module has two log files, modsec_audit.log and modsec_debug.log.

...

Logging for mod_security is configured in /etc/httpd/conf.d/mod_security.conf. A reference manual can be found here: https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-%28v2.x%29

Crond

The cron daemon logs every command that it runs. However by default it sends output (stderr or stdout) from those commands to root via email. The daemon can be configured to send the output to syslog instead of email by putting the following line in /etc/sysconfig/crond:

CRONDARGS=-s -m off

Postgres

Postgres logs to /var/log/pg_log/postgresql.log. Logging is configured in /var/lib/pgsql/13/data/postgresql.conf. It also logs to the syslog facility LOCAL3. Postgres auditing can be turned on and off from the CLI. It should only be enabled when necessary because of the large amount of data generated.

Auditing

Linux system auditing can be turned on and off via the “system audit” command. Database auditing can be turned on via the “system database-auditing” command.

• All successful and unsuccessful login attempts
• All logoffs
• Additions, deletions, and modifications to user privileges, access rules, permissions, and user accounts including account attributes (such as type, identifier, and so on)

-  Changes to programs and parameters (system, network, and security settings)

• hostname/domain
• /etc/sysconfig/network
• failed access to/deletions of files and programs
• successful file system mounts
• changes to sudoers
• loading/unloading of kernel modules
• time-related changes
• Use of setgid/setuid programs
• /etc/selinux changes

-  Changes to critical hardware elements

• All mainframe and server system startups and shutdowns
• All password changes

-  Abnormal system events (for example, performance deterioration, files filling up, programs ending abnormally)

-  Attempts to perform unauthorized functions (that is, those events which create violations)

• approaching storage capacity & other storage issues

-  Privileged account activity

-  Audit log activity (initialization, interruption, unexpected stop, etc.)

• audit log volume capacity

Syslog

Out of the box, syslog-ng logs to the following files:

...

The syslog-ng admin guide has information about customizing the message format.

Syslog-ng Configuration

Syslog-ng uses three parameter types to configure logging: sources, filters and destinations. Sources can be programs that syslog-ng launches, log files from running programs, pipes, snmp traps, local mailboxes, the Linux kernel and the network, among others. A single logical source can be built from many real sources.

...

log { source(s_sys); filter(f_default); destination(d_mesg); }; log { source(s_sys); filter(f_warn); destination(d_lumeta); };

...

Asset Manager

Lumeta Asset Manager uses a combination of syslog-ng and log4j for logging our own software. CentOS and Linux utilities running on Lumeta Asset Manager also produce logs as described above and their output can be controlled via syslog-ng, although we don’t currently supply any way for users to do so.

 The CLI log command can change the logging level of the webapp (API + dojo UI) and discovery components. Those components use the java log4j library. The CLI log command can change the destination of the webapp, discovery and CEF log files. They can remain on the local system or be copied to a remote syslog server. If copied to a remote server, the originals remain on the Lumeta Asset Manager system.

 The gather_diagnostics utility returns the entire contents of /var/log.

Logging Configuration via the CLI

The CLI log command is used to set log levels, set a remote syslog server, enable CEF logging to a remote syslog server and show log contents of the lumeta-webapp.out log file.

...

The CLI calls the helper script log_config, which in turn calls the API to make all the logging changes.

The API, written in Java, calls the helper script observer_log to manipulate the syslog-ng configuration files and to read the log entries from lumeta-webapp.out.

Webapp and Discovery Logging

The lumeta-webapp includes the dojo GUI and the API and anything that calls the API (CLI and scripts, although that’s not distinguished in the log file). Both the webapp and the discovery agent (aka scan agent) use the log4j library. They also print to stdout and stderr.

...

The letter that precedes the thread name is one of {W, A, U, D} for Webapp, API, UI and Discovery-agent, respectively, indicating the .war file that generated the message. Currently, D messages will only appear in discovery-agent.out and W/A/U messages will only appear in lumeta-webapp.out.

Query Logging

By default, query logging is off, but query timing logging is on. Query timing data gets sent to its own log file, lumeta-queries.log. The queries themselves, if logging is enabled get sent to lumeta-webapp.out.

...

log level set TRACE API com.lumeta.api.query.EventDao.addNotification

CEF Logging

In addition to the logging for webapp and discovery defined above, we log Lumeta Asset Manager notifications in a “Common Event Format.” The log file can be sent to a remote syslog server for integration with third-party applications and tools. The CEF implementation is described in the Confluence document Common Event Format Support for LumetaAsset Manager.

We define a log4j appender in log4j.xml (see next section) which outputs the CEF log messages to syslog-ng using facility LOCAL1. The formatting of the messages is implemented in Java in the CEFLayout.java file.

...

• CEF version is always 0
• vendor is always LumetaAsset Manager
• product is always ESI
• priority is always 5
• the following fields are optional and only appear for notifications that contain the data:
• ○      cn1
• ○      cn1Label
• ○      suser
• ○      dhost
• ○      c6a3
• ○      Mac

The remote CEF log server can be defined in the CLI using the “log cefserver” command, or it can be configured in the GUI on the Lumeta Asset Manager Systems page under the CEF Notifications tab. Also on the CEF Notifications tab, an administrator can configure which device and system notifications get logged to the CEF log server.

More on log4j

On startup, the lumeta-webapp and discovery-agent programs read in their log4j.xml files. These files, in addition to setting up appenders, set the initial log levels for various components. They also set up appenders and levels for the query timings and CEF logging.

...

When we use the CLI or API to set log levels, the levels get recorded in the database (system.loglevel table) and get reapplied if the applications restart.

Syslog-ng

Although syslog-ng has the concept of filters and log levels, we only modify the syslog-ng configuration to change the remote syslog and CEF servers. Instead, we use log4j to set the log levels and everything we log that’s at or above those levels goes into the log files. If remote logging is set up, then syslog-ng reads the log files and send the messages to the remote server, where the administrator of that server may decide to apply additional filtering.

...

Errors or warnings in between those lines may indicate an issue even when the UI seems to befunctioning be functioning properly.

Once you see “Started,” the UI and API should be ready for new connections.

...

Here are portions of output from the ps command, annotated to indicate if duplicate lines areexpectedare expected. Some of the lines have been truncated and may appear that way when viewed withtop with top or ps.