Doctor - keep your documentation healthy

akoutmos · June 5, 2019, 3:28pm

A problem that I have constantly had with large software projects is ensuring that there is adequate documentation coverage throughout the project so that new team members (and existing team members of course) can jump right in and understand what is going on and what things are supposed to do.

In order to scratch that itch in Elixir, I created Doctor. Doctor will validate that your documentation coverage (this includes module docs, function docs and type specs) is above your desired threshold. Think of unit test coverage minimum…but for documentation. The thresholds are also configurable to suit your team’s needs via a .doctor.exs file in your project.

I currently use this on my Elixir projects as part of CI/CD (Doctor will return a non zero exit code if validation fails) so that documentation is always treated like a first class citizen and it is never a matter of “I’ll go back and add docs later” because let’s face it I probably will forget ;).

Feel free to leave comments or feedback and curious if people have similar itches! This is still a work in progress but a 1.0.0 release should be out by the end of June.

Links to the project can be found here:

Github: https://github.com/akoutmos/doctor
Hex: https://hex.pm/packages/doctor

And a sample output for documentation coverage of the Phoenix project can be found here:

Doctor file not found. Using defaults.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DOC COV  SPEC COV  MODULE                                   FILE                                                                  FUNCTIONS  NO DOCS  NO SPECS  MODULE DOC
100%     0%        Mix.Phoenix                              lib/mix/phoenix.ex                                                    18         0        18        YES
0%       0%        Mix.Phoenix.Context                      lib/mix/phoenix/context.ex                                            6          6        6         YES
63%      0%        Mix.Phoenix.Schema                       lib/mix/phoenix/schema.ex                                             8          3        8         YES
100%     0%        Mix.Tasks.Compile.Phoenix                lib/mix/tasks/compile.phoenix.ex                                      2          0        2         YES
100%     0%        Mix.Tasks.Phx.Digest.Clean               lib/mix/tasks/phx.digest.clean.ex                                     1          0        1         YES
100%     0%        Mix.Tasks.Phx.Digest                     lib/mix/tasks/phx.digest.ex                                           1          0        1         YES
100%     0%        Mix.Tasks.Phx                            lib/mix/tasks/phx.ex                                                  1          0        1         YES
100%     0%        Mix.Tasks.Phx.Gen.Cert                   lib/mix/tasks/phx.gen.cert.ex                                         2          0        2         YES
100%     0%        Mix.Tasks.Phx.Gen.Channel                lib/mix/tasks/phx.gen.channel.ex                                      1          0        1         YES
86%      0%        Mix.Tasks.Phx.Gen.Context                lib/mix/tasks/phx.gen.context.ex                                      7          1        7         YES
100%     0%        Mix.Tasks.Phx.Gen.Embedded               lib/mix/tasks/phx.gen.embedded.ex                                     6          0        6         YES
100%     0%        Mix.Tasks.Phx.Gen.Html                   lib/mix/tasks/phx.gen.html.ex                                         4          0        4         YES
100%     0%        Mix.Tasks.Phx.Gen.Json                   lib/mix/tasks/phx.gen.json.ex                                         4          0        4         YES
100%     0%        Mix.Tasks.Phx.Gen.Presence               lib/mix/tasks/phx.gen.presence.ex                                     1          0        1         YES
100%     0%        Mix.Tasks.Phx.Gen.Schema                 lib/mix/tasks/phx.gen.schema.ex                                       7          0        7         YES
100%     0%        Mix.Tasks.Phx.Gen.Secret                 lib/mix/tasks/phx.gen.secret.ex                                       1          0        1         YES
100%     0%        Mix.Tasks.Phx.Routes                     lib/mix/tasks/phx.routes.ex                                           1          0        1         YES
100%     0%        Mix.Tasks.Phx.Server                     lib/mix/tasks/phx.server.ex                                           1          0        1         YES
100%     0%        Phoenix                                  lib/phoenix.ex                                                        3          0        3         YES
100%     0%        Phoenix.Channel                          lib/phoenix/channel.ex                                                12         0        12        YES
100%     0%        Phoenix.Channel.Server                   lib/phoenix/channel/server.ex                                         17         0        17        YES
100%     0%        Phoenix.CodeReloader                     lib/phoenix/code_reloader.ex                                          2          0        2         YES
40%      0%        Phoenix.CodeReloader.Proxy               lib/phoenix/code_reloader/proxy.ex                                    5          3        5         YES
33%      0%        Phoenix.CodeReloader.Server              lib/phoenix/code_reloader/server.ex                                   6          4        6         YES
88%      0%        Phoenix.Config                           lib/phoenix/config.ex                                                 8          1        8         YES
100%     0%        Phoenix.Controller                       lib/phoenix/controller.ex                                             42         0        42        YES
100%     0%        Phoenix.Controller.Pipeline              lib/phoenix/controller/pipeline.ex                                    6          0        6         YES
100%     0%        Phoenix.Digester                         lib/phoenix/digester.ex                                               2          0        2         YES
100%     0%        Phoenix.Endpoint                         lib/phoenix/endpoint.ex                                               25         0        25        YES
100%     0%        Phoenix.Endpoint.Cowboy2Adapter          lib/phoenix/endpoint/cowboy2_adapter.ex                               3          0        3         YES
0%       0%        Phoenix.Endpoint.Cowboy2Handler          lib/phoenix/endpoint/cowboy2_handler.ex                               5          5        5         YES
100%     0%        Phoenix.Endpoint.CowboyAdapter           lib/phoenix/endpoint/cowboy_adapter.ex                                2          0        2         YES
0%       0%        Phoenix.Endpoint.CowboyWebSocket         lib/phoenix/endpoint/cowboy_websocket.ex                              8          8        8         YES
100%     0%        Phoenix.Endpoint.RenderErrors            lib/phoenix/endpoint/render_errors.ex                                 3          0        3         YES
93%      0%        Phoenix.Endpoint.Supervisor              lib/phoenix/endpoint/supervisor.ex                                    15         1        15        YES
0%       0%        Phoenix.Endpoint.Watcher                 lib/phoenix/endpoint/watcher.ex                                       2          2        2         YES
NA       NA        Plug.Exception.Phoenix.ActionClauseErro  lib/phoenix/exceptions.ex                                             0          0        0         NO
NA       NA        Phoenix.NotAcceptableError               lib/phoenix/exceptions.ex                                             0          0        0         YES
100%     0%        Phoenix.MissingParamError                lib/phoenix/exceptions.ex                                             1          0        1         YES
0%       0%        Phoenix.ActionClauseError                lib/phoenix/exceptions.ex                                             2          2        2         NO
60%      0%        Phoenix.Logger                           lib/phoenix/logger.ex                                                 5          2        5         YES
83%      0%        Phoenix.Naming                           lib/phoenix/naming.ex                                                 6          1        6         YES
NA       NA        Phoenix.Param.Map                        lib/phoenix/param.ex                                                  0          0        0         NO
NA       NA        Phoenix.Param.Integer                    lib/phoenix/param.ex                                                  0          0        0         NO
NA       NA        Phoenix.Param.BitString                  lib/phoenix/param.ex                                                  0          0        0         NO
NA       NA        Phoenix.Param.Atom                       lib/phoenix/param.ex                                                  0          0        0         NO
NA       NA        Phoenix.Param.Any                        lib/phoenix/param.ex                                                  0          0        0         NO
0%       0%        Phoenix.Param                            lib/phoenix/param.ex                                                  1          1        1         YES
100%     0%        Phoenix.Presence                         lib/phoenix/presence.ex                                               17         0        17        YES
NA       NA        Phoenix.Router.NoRouteError              lib/phoenix/router.ex                                                 0          0        0         YES
100%     0%        Phoenix.Router                           lib/phoenix/router.ex                                                 11         0        11        YES
100%     0%        Phoenix.Router.ConsoleFormatter          lib/phoenix/router/console_formatter.ex                               1          0        1         YES
95%      0%        Phoenix.Router.Helpers                   lib/phoenix/router/helpers.ex                                         20         1        20        YES
100%     0%        Phoenix.Router.Resource                  lib/phoenix/router/resource.ex                                        1          0        1         YES
100%     0%        Phoenix.Router.Route                     lib/phoenix/router/route.ex                                           5          0        5         YES
100%     0%        Phoenix.Router.Scope                     lib/phoenix/router/scope.ex                                           9          0        9         YES
NA       NA        Phoenix.Socket.InvalidMessageError       lib/phoenix/socket.ex                                                 0          0        0         YES
57%      0%        Phoenix.Socket                           lib/phoenix/socket.ex                                                 14         6        14        YES
NA       NA        Phoenix.Socket.Reply                     lib/phoenix/socket/message.ex                                         0          0        0         YES
100%     0%        Phoenix.Socket.Message                   lib/phoenix/socket/message.ex                                         1          0        1         YES
NA       NA        Phoenix.Socket.Broadcast                 lib/phoenix/socket/message.ex                                         0          0        0         YES
50%      0%        Phoenix.Socket.PoolSupervisor            lib/phoenix/socket/pool_supervisor.ex                                 4          2        4         YES
NA       NA        Phoenix.Socket.Serializer                lib/phoenix/socket/serializer.ex                                      0          0        0         YES
0%       0%        Phoenix.Socket.V1.JSONSerializer         lib/phoenix/socket/serializers/v1_json_serializer.ex                  3          3        3         YES
0%       0%        Phoenix.Socket.V2.JSONSerializer         lib/phoenix/socket/serializers/v2_json_serializer.ex                  3          3        3         YES
100%     0%        Phoenix.Socket.Transport                 lib/phoenix/socket/transport.ex                                       6          0        6         YES
NA       NA        Phoenix.Template.UndefinedError          lib/phoenix/template.ex                                               0          0        0         YES
100%     0%        Phoenix.Template                         lib/phoenix/template.ex                                               11         0        11        YES
0%       0%        Phoenix.Template.EExEngine               lib/phoenix/template/eex_engine.ex                                    1          1        1         YES
NA       NA        Phoenix.Template.Engine                  lib/phoenix/template/engine.ex                                        0          0        0         YES
0%       0%        Phoenix.Template.ExsEngine               lib/phoenix/template/exs_engine.ex                                    1          1        1         YES
NA       NA        Phoenix.ChannelTest.NoopSerializer       lib/phoenix/test/channel_test.ex                                      0          0        0         YES
100%     0%        Phoenix.ChannelTest                      lib/phoenix/test/channel_test.ex                                      19         0        19        YES
100%     0%        Phoenix.ConnTest                         lib/phoenix/test/conn_test.ex                                         17         0        17        YES
100%     0%        Phoenix.Token                            lib/phoenix/token.ex                                                  2          0        2         YES
67%      0%        Phoenix.Transports.LongPoll              lib/phoenix/transports/long_poll.ex                                   3          1        3         YES
50%      0%        Phoenix.Transports.LongPoll.Server       lib/phoenix/transports/long_poll_server.ex                            4          2        4         YES
0%       0%        Phoenix.Transports.WebSocket             lib/phoenix/transports/websocket.ex                                   2          2        2         YES
100%     0%        Phoenix.View                             lib/phoenix/view.ex                                                   9          0        9         YES
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Summary:

Passed Modules: 72
Failed Modules: 7
Total Doc Coverage: 85.1%
Total Spec Coverage: 0.0%

Doctor validation has passed!```

hauleth · June 5, 2019, 3:56pm

Why create new tool instead of improving inch? What are advantages (except separate count for specs)?

akoutmos · June 5, 2019, 4:24pm

Good question @hauleth! Before creating Doctor I did come across inch and played around with it. At the time (this may have changed since of course) it did not seem like inch took into account module docs and type specs which was something that I personally wanted to take into account. In addition, it did not seem like “doc coverage” was a primary concern of inch. On the other hand for me, having configurable thresholds which I can validate via a CI/CD pipeline was. This way, commit to commit, the coverage of documentation does not drop. It helps ensure that a team upholds their own best practices.

Looking at the inch README again now it doesn’t look like our philosophies overlap a great deal and in fact, I wouldn’t be surprised if teams used both tools as they offer different benefits.

akoutmos · May 22, 2020, 5:15pm

A new version of Doctor has been published to Hex https://hex.pm/packages/doctor. Version 0.13.0 more accurately tracks documentation coverage for behaviour callbacks by validating that the @impl statements actually apply to behviour callbacks.

akoutmos · June 5, 2020, 3:31pm

A new version of Doctor has been published to Hex https://hex.pm/packages/doctor! Release 0.14.0 brings the ability to check that modules that have a struct defined also provide a typespec for the struct via

@type t :: %__MODULE__{
  a_field: boolean(),
  another_field: integer(),
  ...
}

Check it out and ensure your docs are healthy

akoutmos · June 23, 2020, 2:16pm

Version 0.15.0 of Doctor has been published to Hex https://hex.pm/packages/doctor! This exciting release introduces a new command to Doctor mix doctor.explain that should help when trying to figure out why a particular module failed validation. Enjoy

akoutmos · December 27, 2020, 8:25pm

Version 0.16.0 of Doctor has been published to Hex https://hex.pm/packages/doctor! This release includes some bug fixes as well as the ability to use Regex patterns to ignore any modules that match your Regex patterns.

Enjoy !

krp · December 28, 2020, 6:34am

Thanks for putting this together. Do you have plans to (or have you already) cover Norm ?

olivermt · December 28, 2020, 6:55am

Any plans to include description coverage for absinthe mutations and queries?

akoutmos · December 28, 2020, 10:14pm

@krp @olivermt

At the moment Doctor only supports Elixir language documentation constructs. I really like the idea of adding support for library specific documentation checks, but that will require a bit of an overhaul of Doctor in order to support some sort of plugin mechanism similarly to how Credo works.

Not saying it will never happen, but I don’t think it will happen in the next 6 months as I have been heads down with PromEx (https://github.com/akoutmos/prom_ex) and that has occupied all my “open source time” lol. Writing a library to take the Grafana and Prometheus experience to the next level has been quite the undertaking but I’ll announce that in a separate thread

olivermt · December 29, 2020, 6:45am

Thats a library that exposes prometheus structured data from Telemetry? Also looks cool