UPDATE: Watch a tech talk I recently gave at work about this process.
I’ve been doing some pretty cool work with Drupal 8, Composer, Github, Pantheon and CircleCI around Continuous Integration and Deployment at the day job. Well, not sure you can call Drupal “cool”, but the rest of it is kind of neat, so thought I’d share here.
So, we’ve got a pretty big project where we are:
- Moving from internal hosting to the “cloud” i.e. Pantheon
- Moving our main sites from Drupal 7 to Drupal 8
- Implementing a new design
Any one of those three things is a somewhat non-trivial undertaking, but put together it moves up to “fairly significant.” We’ve got a pretty forward-looking and thinking technical manager, and he has been pushing us to become more modern with our development and deployment workflows. So here’s where things stand as of now (they are constantly changing of course).
Moving to the Cloud
We chose Pantheon as our cloud host for our new Drupal sites. Initially we chose them for their cool features like “Custom Upstreams”, one-click core updates, easy deployments between Dev/Test/Live environments, Multidevs, and the fact that each site is just a Git repo at heart. However, as we’ve developed our workflow, we don’t use all of those features, and some of them are actually hinderances rather than helpers. For instance, the fact that each site is a Git repo is a pretty big headache for us now, since we basically just use Pantheon as a deploy target to dump and run our code (which is built elsewhere). Something like Terminus (Pantheon CLI tool), however, has been heavily used and appreciated.
Migrating to Drupal 8
Eventually we plan to move all (dozens) of our Drupal 7 sites to Drupal 8 but right now we’re just focusing on our two main “umbrella” sites, and one “news” site to serve both umbrella sites. We’re also doing a content refresh, so the only content we’ve had to migrate has been news articles. My coworker wrote a custom migration module to do that migration, and I won’t go into it here other than to say it wasn’t overly difficult (easy for me to say since I didn’t write it). The configuration management of Drupal 8 has also been nice, especially compared to Drupal 7.
Custom Design
Because Drupal is not the only web plaform we use, we built our new look-and-feel as a platform-agnostic project rather than a Drupal theme. It’s based on Foundation Sites and is just straight HTML, SCSS and Javascript. We use Grunt as the build tool, and when we have a new release, we just commit and push to Github. That triggers a CircleCI workflow that tags a new release and publishes the release artifact as an npm package to Artifactory. From there we can pull the npm package into any project that wants to use it, including Drupal. Note that the published package includes ony the CSS, JS, libraries and other assets, but after publishing, a static site is built with the package and corresponding HTML templates on a cloud host as a reference implementation. We also have BackstopJS tests for the design, but as of this moment they are not run on each build unless the developer does it manually. However, that is definitely in our backlog.
Deployment Workflow
Here is where the most interesting bits are. So we have an “upstream” repo on Github named “umbrella-upstream”. That repo is a composer-based Drupal 8 project with a custom install profile (including custom modules), package.json, and some deploy scripts. Each individual site (“umbrella-siteX”, “umbrella-siteY”, etc) is also in a Github repo as a composer-based Drupal 8 project, and has “umbrella-upstream” configured as a remote.
So when we push a change to the upstream repo, a set of CircleCI workflows kick off that runs some Codeception acceptance tests, then merges changes from umbrella-upstream down to each “umbrella-siteX/Y” repo. When that happens, another CircleCI workflow builds, tests and pushes a full Drupal “umbrella-siteX/Y” install to the corresponding Pantheon siteX/Y (all the way up to the Test environment). Any config changes on Pantheon are sent back to the site repos using Quicksilver hooks on Pantheon.
So say we want update our custom design and push it out to all three sites. The workflow would look like this:
-
In custom-design repo:
– code changes…
– git commit - In umbrella-upstream repo:
– npm update custom-design –save-dev
– grunt
– git commit
And that’s it. The change should show up in the Test environment of each site on Pantheon. CircleCi and some fairly gnarly build scripts handle all the building, composer-ing, testing, git-ing, deploying, terminus-ing, etc.
Other Notes
I created a custom Docker image for our CircleCI jobs. It has things like Terminus, some pre-installed composer packages, Codeception, etc.
Since each site is also it’s own repo, we can manage unique dependencies there as well. For instance, if siteX needs a module that siteY doesn’t, we can just run “composer require” in the umbrella-siteX repo, and the system is smart enough to ignore it and just merge in changes as normal from upstream.
If you want to test a feature branch of the upstream repo on Pantheon to see how it interacts, just name your branch beginning with “p-” and push to Github. CircleCI will see any branch named like “p-*” and build a new Multidev environment on Pantheon with our custom install profile installed.
Pantheon has been great, and Terminus is super helpful, but Terminus and the Pantheon API are pretttttttty slooooooow. I got our full build and deploy down to about 18 minutes (from 30+ minutes), and most of that time is just waiting for the Pantheon API to respond (e.g. switching between Git and Sftp mode). Here is where not having our Pantheon sites be git repos would be helpful. We just want to plop our built Drupal site on some host somewhere. Having it be a git repo just makes it harder to deploy.
Each time we push commits to Github or a CircleCI job runs, we get notified in Slack. It gets pretty noisy sometimes, but I like not having to stare at the CircleCI build queue waiting for it to update.
To keep on top of composer package updates, I wrote a CircleCI job that will checkout the umbrella repos, run “composer outdated”, and post a message to Slack. CircleCI is a pretty helpful “cron” replacement if you don’t want to manage a server.
Whew, that’s pretty long-winded, but maybe it will help someone. If you have feedback or questions or whatever, shoot me a message.
