Dev_test

Writing tests for Nvim

Writing tests for Nvim

Nvim has a powerful yet simple test framework. It's approximately 7x better than whatever you use at work.

Each test starts a new Nvim process (10-30ms) which is discarded after the test finishes. You assert stuff using t.eq() and screen:expect() (which automatically waits as needed). That's pretty much it.

TODO: Expose the test framework as a public interface, for use in 3P plugins: https://github.com/neovim/neovim/issues/34592

Test framework

Tests are broadly divided into unit tests (test/unit/), functional tests (test/functional/) and old tests (test/old/testdir/).

You can learn [Lua concepts 15 minutes](https://learnxinyminutes.com/docs/lua/), see also lua-guide. Use any existing test as a template to start writing new tests, or see dev-quickstart.

Tests are run by the /cmake/RunTests.cmake script using busted (a Lua test-runner). For some failures, .nvimlog (or $NVIM_LOG_FILE) may provide insight.

Depending on the presence of binaries (e.g., xclip) some tests will be skipped.

Test Layout

Running tests dev-run-test

EXECUTING TESTS

To run all tests (except "old" tests):

make test

To run only _unit_ tests:

make unittest

To run only _functional_ tests:

make functionaltest

LEGACY TESTS

To run all legacy Vim tests:

make oldtest

To run a _single_ legacy test file you can use either:

# Specify only the test file name, not the full path.
make oldtest TEST_FILE=test_syntax.vim

or:

make test/old/testdir/test_syntax.vim

DEBUGGING TESTS

Read more about dev-filter-test.

Then, in another terminal:

gdb -ex 'target remote localhost:7777' build/bin/nvim

If $VALGRIND is also set it will pass --vgdb=yes to valgrind instead of starting gdbserver directly.

See nvim_argv in test/functional/testnvim.lua.

Then put screen:snapshot_util() anywhere in your test. See the comments in test/functional/ui/screen.lua for more info.

DEBUGGING LUA TEST CODE

Debugging Lua test code is a bit involved. Get your shopping list ready, you'll need to install and configure:

1. [nvim-dap](https://github.com/mfussenegger/nvim-dap) 2. [local-lua-debugger-vscode](https://github.com/mfussenegger/nvim-dap/wiki/Debug-Adapter-installation#local-lua-debugger-vscode) 3. [nlua](https://github.com/mfussenegger/nlua) 4. [one-small-step-for-vimkind](https://github.com/jbyuki/one-small-step-for-vimkind) (called osv) 5. A nbusted command in $PATH. This command can be a copy of busted with exec '/usr/bin/lua5.1'" replaced with "exec '/usr/bin/nlua'" (or the path to your nlua)

The setup roughly looks like this:

┌─────────────────────────┐
│ nvim used for debugging │◄────┐
└─────────────────────────┘     │
           │                    │
           ▼                    │
  ┌─────────────────┐           │
  │ local-lua-debug │           │
  └─────────────────┘           │
          │                     │
          ▼                     │
     ┌─────────┐                │
     │ nbusted │                │
     └─────────┘                │
          │                     │
          ▼                     │
     ┌───────────┐              │
     │ test-case │              │
     └───────────┘              │
          │                     │
          ▼                     │
  ┌────────────────────┐        │
  │ nvim test-instance │        │
  └────────────────────┘        │
    │   ┌─────┐                 │
    └──►│ osv │─────────────────┘
        └─────┘

With these installed you can use a configuration like this:

local dap = require("dap")
local function free_port()
  local tcp = vim.loop.new_tcp()
  assert(tcp)
  tcp:bind('127.0.0.1', 0)
  local port = tcp:getsockname().port
  tcp:shutdown()
  tcp:close()
  return port
end
local name = "nvim-test-case" -- arbitrary name
local config = {
  name = name,
  -- value of type must match the key used in `dap.adapters["local-lua"] = ...` from step 2)
  type = "local-lua",
  request = "launch",
  cwd = "${workspaceFolder}",
  program = {
    command = "nbusted",
  },
  args = {
    "--ignore-lua",
    "--lazy",
    "--helper=test/functional/preload.lua",
    "--lpath=build/?.lua",
    "--lpath=?.lua",
    -- path to file to debug, could be replaced with a hardcoded string
    function()
      return vim.api.nvim_buf_get_name(0)
    end,
    -- You can filter to specific test-case by adding:
    -- '--filter="' .. test_case_name .. '"',
  },
  env = {
    OSV_PORT = free_port
  }
}
-- Whenever the config is used it needs to launch a second debug session that attaches to `osv`
-- This makes it possible to step into `exec_lua` code blocks
setmetatable(config, {
  __call = function(c)
    ---@param session dap.Session
    dap.listeners.after.event_initialized["nvim_debug"] = function(session)
      if session.config.name ~= name then
        return
      end
      dap.listeners.after.event_initialized["nvim_debug"] = nil
      vim.defer_fn(function()
        dap.run({
          name = "attach-osv",
          type = "nlua", -- value must match the `dap.adapters` definition key for osv
          request = "attach",
          port = session.config.env.OSV_PORT,
        })
      end, 500)
    end
    return c
  end,
})

You can either add this configuration to your dap.configurations.lua list as described in :help dap-configuration or create it dynamically in a user-command or function and call it directly via dap.run(config). The latter is useful if you use treesitter to find the test case around a cursor location with a query like the following and set the --filter property to it.

(function_call
  name: (identifier) @name (#any-of? @name "describe" "it")
  arguments: (arguments
    (string) @str
  )
)

Limitations:

This is a [local-lua-debugger limitation](https://github.com/tomblind/local-lua-debugger-vscode?tab=readme-ov-file#busted)

Filtering tests dev-filter-test

FILTER BY NAME

Tests can be filtered by setting a pattern of test name to TEST_FILTER or TEST_FILTER_OUT.

it('foo api',function()
  ...
end)
it('bar api',function()
  ...
end)

To run only test with filter name:

TEST_FILTER='foo.*api' make functionaltest

To run all tests except ones matching a filter:

TEST_FILTER_OUT='foo.*api' make functionaltest

FILTER BY FILE

To run a _specific_ unit test:

TEST_FILE=test/unit/foo.lua make unittest

or

cmake -E env "TEST_FILE=test/unit/foo.lua" cmake --build build --target unittest

To run a _specific_ functional test:

TEST_FILE=test/functional/foo.lua make functionaltest

or

cmake -E env "TEST_FILE=test/functional/foo.lua" cmake --build build --target functionaltest

To _repeat_ a test:

BUSTED_ARGS="--repeat=100 --no-keep-going" TEST_FILE=test/functional/foo_spec.lua make functionaltest

or

cmake -E env "TEST_FILE=test/functional/foo_spec.lua" cmake -E env BUSTED_ARGS="--repeat=100 --no-keep-going" cmake --build build --target functionaltest

FILTER BY TAG

Tests can be "tagged" by adding # before a token in the test description.

it('#foo bar baz', function()
  ...
end)
it('#foo another test', function()
  ...
end)

To run only the tagged tests:

TEST_TAG=foo make functionaltest

NOTE:

Writing tests dev-write-test

GUIDELINES

WHERE TESTS GO

Tests in /test/unit and /test/functional are divided into groups by the semantic component they are testing.

Fixing tests dev-fix-test

FIXING HARNESS WARNINGS

> Nvim session T123 took 2000 milliseconds to exit > This indicates a likely problem with the test even if it passed!

This may indicate a leak, because Nvim waits on uv handles before exiting. Example: https://github.com/neovim/neovim/pull/35768

FIXING LINT FAILURES

make lint (and make lintlua) runs [LuaLS](https://github.com/LuaLS/lua-language-server/wiki/Annotations) on the test code.

If a LuaLS/EmmyLS warning must be ignored, specify the warning code. Example:

---@diagnostic disable-next-line: unused-vararg

https://github.com/LuaLS/lua-language-server/wiki/Annotations#diagnostic

Ignore the smallest applicable scope (e.g. inside a function, not at the top of the file).

Configuration dev-test-config

(TODO: clean this up, too many variables and some of them are not used anymore.)

Test behaviour is affected by environment variables. Currently supported (Functional, Unit, Benchmarks) (when Defined; when set to _1_; when defined, treated as Integer; when defined, treated as String; when defined, treated as Number; !must be defined to function properly):