Vendoring is a feature of golang that allows you to ensure that 3rd party packages used by your project are consistent (of the same version) for anyone who is using your application or library. More specifically the code of these dependencies is placed inside the project and usually committed altogether with the project's source code. This mechanism was added around go 1.5 and it's available in the latest release of go via the go mod vendor command.
The Story
The early version of standard go tooling didn't support any sort of versioning of dependencies. Hence if you use some 3rd party modules and you are afraid that they will break backward compatibility (you should!) you would want to vendor these dependencies. Usually (in other tech stacks), you would pin the exact dependency version, but it wasn't possible in go tooling then. This became possible with the go mod tool and it's go.sum file which pins exact versions of dependencies.
However, there always has been another reason for dependencies vendoring. Go tooling doesn't have any sort of package repositories. Packages are downloaded from source code repositories in the form of source code. This means that your dependencies might just disappear or become inaccessible altogether with the repositories they are stored in. Of course, there's a way to solve this via modules proxy that was introduced in go 1.13.
So, it seems that with go mod and module proxies and go mod dependencies version management vendoring is not needed anymore, isn't it?
Even though it's possible to solve problems of libraries availability with modules proxy it requires some effort to set up a such proxy and maintain it. For companies doing go development, it might be a very good option to increase the stability of builds and overall it's not a huge investment to build infrastructure. For individual contributors and open source projects, this might not be a viable option. So the vendoring dependencies might be still an attractive option in such cases.
Developing VS Releasing
Go mod (and its predecessors) is built on the idea that the module name points to the location of the source control (development place) and the module release location (release place) is the same as its source control. In many cases, this location is a Github repository.
Merging development and release places is intriguing as it potentially allows to simplify things: no build steps are needed to make the library usable by outside users and therefore no noise commits are needed in the build process. The build process boils down to running tests and tagging.
Otherwise, if additional procedures are needed to make the module usable, the benefit of not having a separate release place disappears. If the build process needs to massage code or set dependencies references then such a build would have to create noise commits. The build might have just published artifacts to the side release place whatever it is and left the source code intact.
If there would be a separate release procedure to a separate release place then vendoring as we know it might not be needed at all. There could be just an option toggling how to release the module as self-contained zero-dependency code or a module with dependencies. Something like:
go mod release
go mod release --vendored
Something very similar exists in other tech stacks. The idea of libraries repositories is pretty widely used in most tools. And there's a practice of "vendoring" dependencies, like creating "jumbo-jar".
Monorepo
I have found that the Go strategy of merging module development place and release place can't be executed for practical cases of mono repositories aka multiple module source.
Imagine you are developing multiple modules in the same mono repository. Let's say one module is bar and another is foo. It will be more interesting if the bar module would be both library and tool at the same time. So that you can add it as a dependency via go get ... or install it as a tool via go install .... Your repo structure might look like the following:
monorepo/
├─ bar/
| └─ go.mod
| └─ main.go
└─ foo/
└─ go.mod
So far everything is fine we can even use library bar by its full name: go get github.com/john/monorepo/bar, assuming that the code is hosted on Github under the john account. Also, the bar is installable via go install .... But the situation deteriorates quickly if module bar uses code from module foo. The question would be: "what version of foo should be referenced in the bar module file?"
The one option is to have per library version tags, like: foo/v1.0.0 and bar/v1.1.0. Then in go.mod of bar you could write:
require github.com/john/monorepo/foo v1.0.0
This would work but now it's not very convenient to develop the bar module code. You will have to deal with the foo module version pinning. The whole advantage of a mono repository has gone and you also have got some "build process" where you need to manage version references within the same mono repository.
The other option would be: to use replace directives in the bar (the user) module:
replace github.com/john/monorepo/foo ../foo
require github.com/john/monorepo/foo v0.0.0-00010101000000-000000000000
Note how the foo requirement is pinned to v0.0.0-00010101000000-000000000000 - not a real version and there's a replace directive to tell go mod where the code is located. However, there's a little problem: now whenever you try to use the bar module anywhere via go get github.com/john/monorepo/bar it won't be possible because the go.mod file has replace directives to the local file system. You might use workspaces to move replace directives up. But then you come back to version pinning and the "build process" version refers to the foo library like it was in option one.
What is even worse modules with replace directives couldn't be used via go get ... or go install ..... So the libraries in the mono repo are usable only inside of the mono repo - you can't use them from outside unless you remove replace directives and make references to dependencies pinned or vendor dependencies.
To complete the picture vendored modules are just ignored in the go install commands. It is done so by design. So you can make your tool installable by vendoring its replaced dependencies. Such a tool would be installable.
Long Live Vendoring!
Frankly, I think that the idea of conflating module development place and module release place is not working very well. The modules proxies and vendroing are creative ways to solve issues arising from the design decision to do the conflation. And yet mono repositories are hard to use despite Google's affection for them.
What will I do? I don't want to sacrifice my convenience of development, so I won't stop developing my tools and libraries in the mono repository. Mono repo increases my velocity and improves code review and testing processes. This means that I need to invent some creative way to release my libraries and tools.
The idea is pretty simple I could just develop my modules in one place, test them, etc. And release my modules to a different place. Break the conflation of development space and release space. What place to use for releases? No need to invent anything: just another Github repository would work just fine.
Meet and greet the tool that automatically does the releases of go modules: goven.
The tool is doing a few simple things:
Internalizes the code of modules replace directives and remove them.
On-demand internalizes vendored dependencies (by standard go mod vendor).
Renames the module.
Pushes the resulting code to the target repository.
Creates release tag if requested.
The process of internalizing might be considered to be "ultimate vendoring" since it just includes the source code of the dependencies into the source code of the module itself.
Here's how to release the above-mentioned bar module to the new repo github.com/john/bar:
goven release -name github.com/john/bar -version v1.1.0
goven release -vendor -name github.com/john/bar -version v1.1.0
The resulting module at github.com/john/bar is fully usable via go get or go install and might come with vendored or referenced dependencies.
I have to mention that go provides such nice tools ready out of the box to work with .go sources as well as with go.mod files. Implementing goven was relatively easy and I was fighting more mental blocks stopping me from doing some other than "blessed" releases of go modules.
I have tested this process on a few of my libraries and tools and it works pretty well. Feel free to use it ask me any questions or submit bugs.
