Go Plugins and Avoiding Cyclic Dependencies

The other day, I started a project to eventually replace the backend of Kirsle.net with a Go program instead of the current Python one (Rophako). It will support a similar feature set (being modular with even the core functionality, like user accounts and web blogs, being served by built-in "plugins" and allowing users to extend it with their own plugins).

The plugin system will support both compile-time plugins (your main.go imports and registers all the plugins you need when compiling the binary), and run-time plugins using Go's plugins from *.so files support.

This post will focus on the former, compile-time plugins, and how I ran into a cyclic dependency issue and worked around it.

Code Structure for Plugins

I've tentatively named my Go app GopherType until I think of a better name, and I have a few Go packages that I'll talk about here:

• github.com/kirsle/gophertype is the root package of my Go application.
• github.com/kirsle/gophertype/plugin defines a plugin interface that the plugins adhere to. This is imported by the root Go package so it can maintain data structures like a map[string]plugin.Pluggable to keep track of plugins that implement the Pluggable interface.
• github.com/kirsle/gophertype/plugin/* are the "core plugin" implementations. The core package does not import these, instead the main command line program does (or your custom main.go if you want custom plugins on top of the core ones).

In an ideal world, the plugin.Pluggable interface would have references back to the parent app object (Gophertype) so that the plugins themselves could call methods on it, use it to register their own event handlers, and so on.

However, the gophertype/plugin package is not allowed to import the root gophertype package -- because the root package imports it, and this would lead to a "circular dependencies" exception at compile time.

Circular Dependencies

The first problem to solve, then, is how can the plugins themselves get access to the root Gophertype struct without a circular import error? The plugin interface can't refer to the Gophertype struct directly, so the best it can do is refer to it as interface{}, or, an object of unknown type that has to be cast to its real type later before it can be used.

I didn't want to just move the burden of dealing with this problem onto the plugin writers, though, because having to typecast that interface{} every time you use it would be annoying.

I eventually came up with a solution: instead of the plugins simply implementing the plugin.Pluggable interface themselves, they could base their plugin on a "common plugin" that handles all the dirty work of accepting the interface{}, type-casting it to a Gophertype struct, and storing it for easy access by the actual plugins.

Important Note: the circular dependency constraint only applies to the Gophertype root package importing the plugin sub-packages. If my main.go (outside of Gophertype), though, imports both Gophertype and a bunch of plugins, this is perfectly okay: Gophertype itself isn't getting into an import cycle, because the main package lives outside the whole mess and just marries them all together.

How about some source code? I simplified my app to just the bare essentials to demonstrate how I implemented the plugin system.

github.com/kirsle/gophertype (root package)

package gophertype

import (
    "fmt"
    "os"
    "path/filepath"
    "sync"

    "github.com/gorilla/mux"
    "github.com/kirsle/golog"
    "github.com/kirsle/gophertype/plugin"
    "github.com/urfave/negroni"
)

var log *golog.Logger

func init() {
    log = golog.GetLogger("gophertype")
    log.Configure(&golog.Config{
        Colors: golog.ExtendedColor,
        Theme:  golog.DarkTheme,
    })
}

// Gophertype is the parent object of the web server.
type Gophertype struct {
    // Public configuration fields.
    Debug        bool
    Address      string
    DocumentRoot string

    // Map of loaded plugins by name
    plugins    map[string]plugin.Pluggable

    // Map of which plugins subscribe to which event hooks
    pluginSubs map[string][]plugin.Pluggable

    // A mutex lock to protect the above maps from concurrent access
    pluginsMu  sync.Mutex

    // Web app objects (middleware and router)
    Negroni *negroni.Negroni
    mux     *mux.Router
}

// New creates a Gophertype instance with the default settings.
func New() *Gophertype {
    return &Gophertype{
        Address:    ":5000",
        plugins:    map[string]plugin.Pluggable{},
        pluginSubs: map[string][]plugin.Pluggable{},
    }
}

// Load a plugin and call its PreInit() and Init() functions.
func (g *Gophertype) Load(plugin plugin.Pluggable) error {
    g.pluginsMu.Lock()

    name := plugin.String()
    if _, ok := g.plugins[name]; ok {
        g.pluginsMu.Unlock()
        return fmt.Errorf("plugin %s has already been loaded", name)
    }

    g.plugins[name] = plugin
    g.pluginsMu.Unlock()

    plugin.PreInit(g)
    plugin.Init()

    return nil
}

// Subscribe a plugin to an event to receive callbacks for it.
func (g *Gophertype) Subscribe(event string, plugin plugin.Pluggable) {
    g.pluginsMu.Lock()
    defer g.pluginsMu.Unlock()

    log.Debug("Subscribe: plugin '%s' is interested in '%s'", plugin, event)
    g.pluginSubs[event] = append(g.pluginSubs[event], plugin)
}

// EmitEvent notifies subscribed plugins about a lifecycle phase in the app.
func (g *Gophertype) EmitEvent(event string) {
    g.pluginsMu.Lock()
    defer g.pluginsMu.Unlock()

    if subs, ok := g.pluginSubs[event]; ok {
        for _, plugin := range subs {
            log.Debug("Notify: '%s' event sent to plugin '%s'", event, plugin)
            var err error

            switch event {
            case "OnRegisterRoutes":
                err = plugin.OnRegisterRoutes(g.mux)
            default:
                log.Warn("EmitEvent: no such event '%s'", event)
            }

            if err != nil {
                log.Error("Error in %s#%s: %s", plugin, event, err)
            }
        }
    }
}

github.com/kirsle/gophertype/plugin (the plugin interface)

package plugin

import "github.com/gorilla/mux"

// Pluggable is the interface that GopherType plugins must adhere to.
type Pluggable interface {
    // String should return the name of your plugin.
    String() string

    // PreInit() is called when the plugin is being initialized. It is passed
    // a reference to the parent Gophertype object in order to prevent an import
    // cycle error when dealing with plugins. It is recommended that you base
    // your plugin on `plugin/common` which handles this all for you.
    PreInit(interface{}) error

    // Init is called on startup when your plugin is loaded.
    Init() error

    // OnRegisterRoutes is called when GopherType is setting up the router.
    // Your plugin should subscribe to this to register its custom routes.
    OnRegisterRoutes(*mux.Router) error
}

A couple of things are going on here:

• Plugins have to implement all of the methods that the plugin interface defines, otherwise they're not considered to have implemented the interface.
• The PreInit(interface{}) function accepts an untyped Go object. Ideally it should have said PreInit(*Gophertype) but it can't refer to that name without causing a cyclic dependency error.
• In the core app's LoadPlugin() function, it accepts any object that implements the plugin.Pluggable interface, loads it, and calls PreInit(g), passing its own object in as the interface{} to PreInit().

At this point, a plugin author could simply go from there and hack out a working plugin, but it would be like pulling teeth trying to work around all the layers of indirection to get access to that Gophertype struct. To make life significantly easier, though, I wrote a "common" plugin to base yours on that provides some useful common functionality:

github.com/kirsle/gophertype/plugin/common (a common base for plugins)

package common

import (
    "net/http"

    "github.com/gorilla/mux"
    "github.com/kirsle/golog"
    "github.com/kirsle/gophertype"
)

type Plugin struct {
    // Root is a reference to the parent Gophertype object that loaded the plugin.
    Root *gophertype.Gophertype

    // Log is a reference to the same logging engine that Gophertype itself uses.
    // You can hook into this to send log messages that respect the user's logging
    // preferences.
    Log *golog.Logger
}

// PreInit handles receiving the root Gophertype object "anonymously" to prevent
// an import cycle error.
func (p *Plugin) PreInit(sneaky interface{}) error {
    if app, ok := sneaky.(*gophertype.Gophertype); ok {
        p.Root = app
        p.Log = golog.GetLogger("gophertype")
    }
    return errors.New("PreInit() passed a wrong data type; expected a Gophertype")
}

// Subscribe notifies the Gophertype server that your plugin is interested in
// an event to be handled. As such, you should also define the corresponding
// event handler in your function.
func (p *Plugin) Subscribe(event string) {
    p.Root.Subscribe(event, p)
}

// Stub implementations for plugin.Pluggable
func (p *Plugin) String() string { return "core:common" }
func (p *Plugin) Init() error { return nil }
func (p *Plugin) OnRegisterRoutes(*mux.Router) error { return nil }

An Example Plugin

With all that boilerplate aside, here is an example of the source code for a 'real' plugin: one that handles unknown request URLs by trying to serve a static file from the filesystem.

I've simplified some of the logic of this plugin from how it actually exists (handling common file extensions and such) for the sake of this post.

github.com/kirsle/gophertype/plugin/static (static files plugin)

package static

import (
    "fmt"
    "net/http"
    "os"
    "path/filepath"
    "strings"

    "github.com/gorilla/mux"
    "github.com/kirsle/gophertype/plugin/common"
)

// Plugin object is based on common.Plugin to get its PreInit() handler and
// its attributes like Root and Log.
type Plugin struct {
    common.Plugin
}

// New plugin.
func New() *Plugin {
    return &Plugin{}
}

func (p Plugin) String() string { return "core/static" }

// Init handler.
func (p *Plugin) Init() error {
    p.Root.Subscribe("OnRegisterRoutes", p)
    return nil
}

// OnRegisterRoutes handler.
func (p *Plugin) OnRegisterRoutes(mux *mux.Router) error {
    mux.HandleFunc("/", p.handle)
    mux.NotFoundHandler = http.HandlerFunc(p.handle)
    return nil
}

// handle finding the file or give a 404 instead.
func (p *Plugin) handle(w http.ResponseWriter, r *http.Request) {
    path := r.URL.Path

    // Search for this file.
    abspath, err := filepath.Abs(filepath.Join(p.Root.DocumentRoot, path))
    if err != nil {
        p.Log.Error("%v", err)
    }

    // Found an exact hit?
    if stat, err := os.Stat(abspath); !os.IsNotExist(err) && !stat.IsDir() {
        http.ServeFile(w, r, abspath)
        return
    }

    w.WriteHeader(404)
    fmt.Fprintf(w, "File's not here, bud")
}

This way, all I had to do was extend my plugin from common.Plugin to get all that black magic bundled up into mine, and then my interface is much simpler to implement. In my Init() function I already have access to the Root attribute which gives me the full API surface area of the core Gophertype application, and I only have to define the event handler methods that I actually use.

Granted, at this point there is only one event handler in existence (OnRegisterRoutes), but as I add more I won't need to worry about implementing them if my plugin doesn't need them: the common.Plugin already satisfies the plugin.Pluggable interface by defining useless "stub" methods, and since I base my plugin on that one, I get those methods automatically. I can then cherry-pick which ones I override for my own purposes.

Summary

The high-level overview of what I needed to do:

• My core app imports the plugin interface package.
• The plugin interface package can not refer back to the core, but instead uses interface{} as its argument to PreInit() that will accept the core app's struct.
• And then, completely outside of Gophertype in the main.go you bring in the other pieces of the puzzle:
  • plugin/common implements plugin.Pluggable and uses the PreInit() function to type-cast the interface{} into a much easier to use *Gophertype, as well as providing other useful functionality.
  • The actual plugins that do actual work extend from plugin/common and have a friendly API to work with.