Testing

Run unit tests

The unit tests (without selenium/ functional tests) can be run using pytest:

pytest --ignore tests/selenium

============================= test session starts =============================
platform win32 -- Python 3.6.5, pytest-3.7.1, py-1.5.4, pluggy-0.7.1
Django settings: dqmsite.test_settings (from ini file)
rootdir: C:\Users\keepingitsecret\workspace\CERN\certifier, inifile: pytest.ini
plugins: mock-1.10.0, django-3.3.3, cov-2.5.1
collected 118 items

runregistry\tests.py ....                                                [  3%]
tests\certhelper\test_dynamic_preferences.py .                           [  4%]
tests\certhelper\test_forms.py .                                         [  5%]
tests\certhelper\test_manager.py .......                                 [ 11%]
tests\certhelper\test_models.py ..................                       [ 26%]
tests\certhelper\test_myfilters.py ......                                [ 31%]
tests\certhelper\test_query.py ........................                  [ 51%]
tests\certhelper\test_shiftleaderreport.py ......                        [ 56%]
tests\certhelper\test_signals.py ....                                    [ 60%]
tests\certhelper\test_summaryreport.py .                                 [ 61%]
tests\certhelper\test_utilities.py ..................                    [ 76%]
tests\certhelper\test_views.py ..................                        [ 91%]
tests\certhelper\client\test_login.py ..                                 [ 93%]
tests\certhelper\client\test_urls.py ........                            [100%]

========================= 118 passed in 14.16 seconds =========================.p

Selenium

If the selenium tests should also be run one needs at least FireFox ESR and has to download the geckodriver which can be found on the mozilla GitHub repository: https://github.com/mozilla/geckodriver/releases

It should be kept in mind that selenium tests require a lot more time to execute since they run actual instances of FireFox.

Geckodriver

Linux:

wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux64.tar
.gz
mkdir geckodriver
tar -xzf geckodriver-v0.21.0-linux64.tar.gz -C geckodriver
export PATH=$PATH:$PWD/geckodriver

Windows:

Download and extract

https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-win64.zip

and export geckodriver.exe to the PATH environment variable

Arch Linux

On Arch Linux all that has to be done is:

sudo pacman -S geckodriver

Run Selenium tests

After installing the geckodriver it should be possible to run the selenium tests as well:

pytest tests/selenium

To see what is happening during the tests the --headless parameter in conftest.py can be commented out temporarily.

Write unit tests

https://docs.pytest.org/en/latest/

Whenever functionality is added a corresponding unit test should be written to ensure correctness and robustness against future changes.

A test function should be defined anywhere in the tests module for example tests/certhelper/test_utilities.py Any function in that file that starts with test_ is considered as a test function.

def test_my_new_function():
  assert "My expected output" == my_new_function()

pytest automatically detects test_my_new_function and includes it in the testing procedure.

Configure Travis CI

Travis CI sets up a testing environment according to the .travis.yml file. This file contains the python version, the Django version and the test commands.

addons:
  apt:
    packages:
      - postgresql-12
      - postgresql-client-12
  firefox: latest
  postgresql: "12"
after_success:
  - codecov
before_install:
  - "wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux64.tar.gz"
  - "mkdir geckodriver"
  - "tar -xzf geckodriver-v0.21.0-linux64.tar.gz -C geckodriver"
  - "export PATH=$PATH:$PWD/geckodriver"
before_script:
  - "psql -c \"create database testdb;\" -U travis"
env:
  global:
    - PGUSER=travis
    - PGVER=12
    - PGPORT=5433
install:
  - "pip install --upgrade -r requirements.txt"
  - "pip install --upgrade pytest pytest-django pytest-cov codecov mixer selenium"
language: python
python:
  - "3.8"
script:
  - "PYTHONWARNINGS=all travis_retry pytest --ds=dqmhelper.test_ci_settings --cov=. --ignore certifier/tests/test_certifier_views.py --ignore oms/tests/test_oms_utils.py"

Note

Certifier views tests and OMS utils tests have been disabled due to OMS being unreachable from outside CERN.

In Travis CI following environment variables have to be set:

DJANGO_DATABASE_ENGINE django.db.backends.postgresql_psycopg2
DJANGO_DATABASE_HOST localhost
DJANGO_DATABASE_NAME testdb
DJANGO_DATABASE_USER postgres
DJANGO_DEBUG True
DJANGO_SECRET_KEY dbwqabxpc2denpefq4hgfhijkl0usxi6d3tm4jk609zo85dqrw
OMS_CLIENT_ID <secret>
OMS_CLIENT_SECRET <secret>
PGPORT 5543
POSTGRES_DB test_postgres_db
POSTGRES_USER travis

Coverage Reports

If the all tests pass a coverage report is automatically uploaded to codecov and can be viewed on:

• https://codecov.io/gh/CMSTrackerDPG/certifier

The Report shows which files need further testing and how good the overall test coverage is.