Use Pydantic schemas to validate Mbed's JSON files (part 1)

[multiplemonomials]

Summary of changes

Ever since I started using Mbed, one of the biggest pain points has been working with JSON configuration files. Mbed uses these files as its configuration system, and they are used to store information such as the list of known targets and their properties, the options that can be configured for each macro and library, and the memory bank info for each target. This system isn't bad in concept, and basically any project as large and configurable as Mbed OS needs something like it.

However, the implementation has never been super solid: Mbed relies on python scripts that load tons of these JSON files, combine them together and munge the data in various ways into a single dict with every concievable setting in it, and then pass that dict off to the code that generates build flags and other configuration (which used to also be in python, but is now in CMake). This implementation meant that the valid properties for each JSON file, and what effect that they'd have within the configuration system, was in large part a mystery to everyone who did not directly work on the Mbed build system.

ARM did at least make some attempt to document the valid settings, but looking at the real JSON files shows that there were many more undocumented ones that are not mentioned in those pages. When you factor in that the way the config system merges JSONs means that target JSON attributes can be put in mbed_app.json and will "work" (though they might override other attributes!), and the fact that it has never issued warnings for unknown/unrecognized attributes, AND the fact that all of this stuff was JSON until recently and didn't allow comments, you get... a system made almost radioactive by a decade of cruft that no one wants to change for fear of breaking something. Even to relatively seasoned users, like me a few years ago, there were plenty of JSON settings that seemed like borderline magic incantations -- you put em in and something happens but you have no idea why,

Well, I am here to tell you that this ends today. Well, mostly. I've had the pleasure of using pydantic at my day job, and it's a super cool library -- it basically lets you define a schema for structured data as a python class, and then use that schema to parse, validate, and dump data. Me, and @VictorWTang as well, thought that this would be a great use for it.

For this PR, I read through many, many existing JSON files as well as nearly all the existing config code, and reverse engineered a schema for all of Mbed's JSON files. This schema should cover the large, large majority of existing use cases for these files, but removes all the legacy attributes which haven't had meaning since Mbed CLI 1 was being used. It also provides, at last, real documentation for every single legal field of each JSON file (right now it's in the form of a Python class, but we also have options to convert this to a JSON schema and, from there, markdown docs if we'd like to). Right now, the schema is being enforced for all mbed_lib.json and targets/custom_targets.json files, as these are mostly used only within Mbed, mbed_app.json, meanwhile, is validated against the schema but validation errors only are treated as a warning. This way, compatibility will not be broken for projects using mbed_app.json in unexpected ways.

For this PR, I did some refactoring of how the configuration is processed internally, mostly to keep things in the pydantic-model format instead of a dict where it makes sense. However, I did not change the fundamental method used to generate the final configuration (stuffing everything into a single god dict). This was both out of fear of making breaking changes, and because I didn't want to break mbed_app.json compatibility (since, as I said, this file was the total Wild West that could override any configuration setting). I called this PR "part 1" because eventually (years down the road), I'd like to go through and replace the god dictionary (config.Config) with a proper data model that stores each thing individually. But that will have to wait until users have gotten used to the new schema rules (and we've made any changes to the schema that we end up wanting!).

Oh, also, since I was in the guts of this code anyway, I took the chance to conquer one of the smaller evils of Mbed programming: the lack of any naming standard for config settings. I am defining here and now that all settings shall be in lowercase-skewer-case, and may not contain uppercase letters or underscores. Mbed will now print a warning if it sees an underscore in a setting name, and transform it into a hyphen. This means it's no longer possible to screw up your configuration by writing target.application_profile instead of target.application-profile and the like. This removes one of the easiest ways to make config mistakes and the biggest thing that kept me from remembering these names without having to check each time.

Impact of changes

• mbed_lib.json5 and custom_targets.json5 files (if they exist in your project) are now required to validate against their schemas
• mbed_app.json5 is also validated against the schema, but only a warning is printed if validation fails. If you are getting this warning and think that your mbed_app.json should be OK, please contact us ASAP as this warning will become an error in a future version.
• Bug fixed where declaring overrides and target_overrides in the same JSON file would conflict with each other. This could cause settings that were previously not being applied to now be applied.
• Config settings with underscores are now converted to use hyphens. This means that if you had misspelled settings in your JSON files that weren't getting applied, they may now be applied
  • Also, if you defined two different config settings which differed only by hyphen vs underscore in name (in which case I hate you!), that will break completely as they will be considered the same option.

Migration actions required

Documentation

---

Pull request type

[] Patch update (Bug fix / Target update / Docs update / Test update / Refactor)
[] Feature update (New feature / Functionality change / New API)
[X] Major update (Breaking change E.g. Return code change / API behaviour change)

---

Test results

[] No Tests required for this change (E.g docs only update)
[X] Covered by existing mbed-os tests (Greentea or Unittest)
[] Tests / results supplied as part of this PR

---