Debugging a LAVA instance¶
Each of these components has a service which may need to be restarted when making changes. Each of these services are restarted when the relevant packages are installed.
lava-server - the frontend UI and admin interface. If using apache use
apache2ctl restartwhen changing any of the django files, device type templates or lava-server settings:
$ sudo apache2ctl restart
worker - each dispatcher worker connects to the server using HTTP and follows the instructions of the server, using configuration specified by the server. Restart is rarely needed, usually only when changing the dispatcher code related to HTTP or the loghandler:
$ sudo service lava-worker restart
The worker have dedicated singleton processe which should be put
DEBUG when investigating problems.
Restart the service after editing the service file.
/etc/init.d/lava-workercurrently defaults to DEBUG.
All log files use
logrotate, so the information you need may be in a
log.2.gz file - typically up to
zgrep for older log files.
django - by default
/var/log/lava-server/django.logcontains errors and warnings from django.
test jobs -
/var/lib/lava-server/default/media/job-output/individual files are in a directory named after the start time of the corresponding job
2018/08/10/1234. The directory includes the validation output
description.yamland the full log file
Command line debugging¶
sudo lava-server manage shell.
lava-dispatcher - The actions of
lava-workercan be replicated on the command line. The relevant device configuration can be obtained using
$ lavacli devices dict get --render DEVICE_HOSTNAME
This config can then be passed to
lava-run, in this example in a file named
$ sudo lava-run --device device.yaml --output-dir /tmp/debug/ job.yaml
Every job is validated before starting and the validate check can be run directly by adding the
$ sudo lava-run --job-id 0 --device device.yaml --validate --output-dir /tmp/debug/ job.yaml
The job will not start when
--validateis used - if validation passes, the complete pipeline will be described. If errors are found, these will be output.
lava-serverif this is changed. Holds details for django settings including the authentication methods and site customization settings.
jinja2 templates -
/etc/lava-server/dispatcher-config/device-typesThese files are updated from
lava_scheduler_app/tests/device-typesin the codebase. The syntax is YAML with jinja2 markup. Restart the
lava-masterafter changing the templates.
to validate changes to the templates, use:
$ /usr/share/lava-server/validate_devices.py --instance localhost
to validate the combination of the template with the device dictionary content, use:
$ lavacli devices dict get --render DEVICE_HOSTNAME
device dictionaries -
/etc/lava-server/dispatcher-config/devicesThese files are specific to each instance and need to be named according to the
hostname.jinja2 of the device as configured on the same instance.
Check LAVA instance¶
Once a LAVA instance is installed admins can check for errors and warnings on the deployed instance using:
$ sudo lava-server manage check --deploy
check --deploy command uses the Django system check framework which is a
set of static checks to detect common problems and provide hints for how to fix
https://docs.djangoproject.com/en/3.2/ref/checks/ to know more about Django system check framework.
If the JSON syntax of
/etc/lava-server/settings.conf is invalid,
this check will display a warning:
settings.conf is not a valid json
LAVA sets the following values by default:
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_BROWSER_XSS_FILTER = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
CSRF_COOKIE_HTTPONLY = True
X_FRAME_OPTIONS = 'DENY'
These values can be overridden in
The following checks are silenced and does not show any errors or warnings:
SECURE_HSTS_SECONDS i.e., security.W004 which is documented in https://docs.djangoproject.com/en/3.2/ref/settings/#secure-hsts-seconds
SECURE_SSL_REDIRECT i.e., ‘security.W008’ which is documented in https://docs.djangoproject.com/en/3.2/ref/settings/#secure-ssl-redirect
Admins should consult the respective Django documentation before changing these values to suit the requirements of each LAVA instance.
Displaying current settings¶
The django developer shell can be used to check the value which is currently set:
$ sudo lava-server manage shell
Python 3.6.7 (default, Oct 21 2018, 08:08:16)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from django.conf import settings
Avoid making changes to the defaults inside the LAVA code.
These will be replaced when you upgrade to future versions of the
lava-server package. If you need to make changes, instead edit
/etc/lava-server/settings.conf where they will be preserved.
Whenever you make changes, be sure to restart the LAVA daemons
before checking for the effects of the change:
$ sudo service lava-server-gunicorn restart
In some situations, you may also need to restart
lava-publisher in the same way.
Installing a new release or a set of Developer packages will also restart all LAVA daemons.
Django support in 1.11¶
If your local instance is using
buster, the new
support in 1.11 and later can be used to display the settings as well:
$ sudo lava-server manage diffsettings --all
Settings that don’t appear in the defaults are followed by “###”, e.g.:
BRANDING_MESSAGE = 'Master branch instance on Debian Buster.' ###
lava-server manage needs to be restricted
in the same way as read permissions on
/etc/lava-server/settings.conf as the current settings will
include the database secret_key, authentication passwords for
services like LDAP and other sensitive information.