Idea: Write "JSON" Schemas for each test format and test on CI

[kylef]

I've done similar in the past where I've used a JSON Schema to provide a full schema for other files to be used during CI. This would give us a few benefits:

• As documentation for the format of each file, can also contain description of the format.
• Can be used to test that the test files are correct during any changes in CI. This will prevent any unintentional breakage cause by PRs.

As an example (schemas/userhost-split.yaml):

type: object
properties:
  tests:
    type: array
    items:
      '$ref': '#!/definitions/test'

definitions:
  test:
    type: object
    properties:
      source:
        type: string
        description: source is the usthost
      atoms:
        type: object
        properties:
          nick:
            type: string
          user:
            type: string
          host:
            type: string
    required: [source, atoms]

[DanielOaks]

For some prior art on the testing side, depending on what lang is used, see some extra stuff here.