Replacing the Rails asset pipeline with Gulp: Using Gulp to compile and cache our assets
At Bugsnag, we used Rails’s asset pipeline to compile and deploy our assets. It was really convenient since the asset pipeline is bundled into Rails, and it solved many problems for us right out of the box. It dealt with caching our assets and generating the correct URLs for them, plus cache busting. Not only that, it also compiled and minified all our assets. We also discovered Rails Assets, a way to bridge Bundler with Bower, which came in handy for handling our front-end dependencies easily. This allowed us to load in our bower components as gems.
So this all sounds perfect, right? Why would we change what we had? Unfortunately, the asset pipeline had significant issues. Any time we compiled locally or deployed, it was always painfully slow. We were also frustrated because it was extremely hard to debug and test what was going on since so much magic was happening behind the scenes with Rails. Plus, the asset pipeline didn’t allow us to harness sourcemaps to fix up our JavaScript stacktraces which we knew would make debugging our JavaScript way easier. We decided we wanted a better way.
What is Gulp?
After some research, we decided to use a tool called Gulp for our asset compilation needs. Gulp is a Node.js streaming build system, strictly providing streams and a basic task system. This means that we can use Gulp to automate common tasks we need in our website, like compiling assets for deployment. We decided to go with Gulp because of its intuitive code-over-configuration stance, as well as its simple but effective plugins.
To install Gulp, check out Gulp’s getting started document.
Setting up for development
In development, we don’t need to worry about caching, deployment, or any of the other tasks associated with pushing out to production. For now, we’ll want to pull out the Rails asset pipeline, install Gulp, and get Gulp building our development assets and serving them to [CODE]public/assets[/CODE].
Pulling out the Rails asset pipeline
In Rails 4, you can turn asset pipeline off easily. In your [CODE]config/application.rb[/CODE], you just need to disable your assets:
-- CODE language-ruby --
config.assets.enabled = false
And if you use generators, you’ll want to prevent the cli generator from creating assets when generating a scaffold:
-- CODE language-ruby --
config.generators do |g|
g.assets false
end
Hooking up Gulp
Gulp allows us to separate our backend Rails app from our frontend app. We’ll want to create a [CODE]frontend[/CODE] directory in the root of the application in which to keep our assets. From there, we’ll compile a [CODE]public/assets[/CODE] directory so the assets can be served up with Rails. We’ll have to be in the [CODE]frontend[/CODE] directory whenever we run any of our gulp commands.
Our [CODE]frontend[/CODE] directory will also contain our [CODE]gulpfile[/CODE], which will contain the code for our tasks. When we run [CODE]$ gulp[/CODE] from the command line, we want our default [CODE]gulp[/CODE] task to:
-- CODE language-coffeescript --
gulp.task "default", ["js", "sass", "images", "fonts"]
Concatenate all JS bower files, vendor files, and source files into one file; add to public; create JS sourcemaps
-- CODE language-coffeescript --
sourceMaps = require "gulp-sourcemaps"
liveReload = require "gulp-livereload"
filter = require "gulp-filter"
coffee = require "gulp-coffee"
gulp.task "js", ->
gulp.src("paths/to/your/js")
.pipe(sourceMaps.init())
.pipe(coffeeFilter)
.pipe(coffee())
.pipe(coffeeFilter.restore())
.pipe(concat("application.js"))
.pipe(sourceMaps.write("."))
.pipe(gulp.dest("../public/assets"))
.pipe(liveReload())
Create Sass sourcemaps, add to public
-- CODE language-coffeescript --
sass = require "gulp-sass"
autoPrefixer = require "gulp-autoPrefixer"
sourceMaps = require "gulp-sourcemaps"
liveReload = require "gulp-livereload"
gulp.task "sass", ->
gulp.src("sass/dashboard/application.css.scss")
.pipe(sourceMaps.init())
.pipe(sass({includePaths: "path/to/code", sourceComments: true, errLogToConsole: true}))
.pipe(autoprefixer())
.pipe(rename("application.css"))
.pipe(sourceMaps.write("."))
.pipe(gulp.dest("../public/assets"))
.pipe(liveReload())
Add our images to public
-- CODE language-coffeescript --
liveReload = require "gulp-livereload"
gulp.task "images", ->
gulp.src("images/**/*")
.pipe(gulp.dest("../public/assets/images/"))
.pipe(liveReload())
Add our fonts to public
-- CODE language-coffeescript --
liveReload = require "gulp-livereload"
gulp.task "fonts", ->
gulp.src("fonts/**/*")
.pipe(gulp.dest("../public/assets/fonts/"))
.pipe(liveReload())
After our default task finishes, we’ll have our assets compiled into [CODE]public/assets[/CODE] in a way we can use for development.
Live Reloading
We recommend setting up a [CODE]reload[/CODE] task that includes live reloading for your assets. We use this so that any time someone edits a JavaScript or CSS file, their browser will automatically refresh after compiling, saving time and effort:
-- CODE language-coffeescript --
gulp.task "reload", ["watch", "js", "sass", "images", "fonts"]
Since we’re using the Ruby gem rack-livereload, our [CODE]watch[/CODE] task can now enable live reloading:
-- CODE language-coffeescript --
gulp.task "watch", ->
liveReload.listen
gulp.watch "paths/to/your/sass", { interval: 500 }, ["sass"]
gulp.watch "paths/to/your/js", { interval: 500 }, ["js"]
Setting up for production
Next we need to set up for getting our code out to production. This entails setting up caching and cache busting, sending our assets up to S3, and setting up Capistrano for deployment.
Adding production Gulp tasks
gulp js
In our [CODE]js[/CODE] gulp task, we’ll want to add in uglification of JavaScript after concatenation.
-- CODE language-coffeescript --
sourceMaps = require "gulp-sourcemaps"
liveReload = require "gulp-livereload"
filter = require "gulp-filter"
coffee = require "gulp-coffee"
uglify = require "gulp-uglify"
coffeeFilter = filter ["**/*.coffee"]
gulp.task "js", ->
stream = gulp.src("paths/to/your/js")
.pipe(sourceMaps.init())
.pipe(coffeeFilter)
.pipe(coffee())
.pipe(coffeeFilter.restore())
.pipe(concat("../public/assets"))
if ["production, staging"].indexOf(railsEnv) != -1
stream = stream
.pipe(uglify())
stream.pipe(gulp.dest("../public/assets"))
.pipe(sourceMaps.write("."))
.pipe(liveReload())
gulp sass
Similar to our [CODE]js[/CODE] gulp task, we’ll want to add in minification of Sass.
-- CODE language-coffeescript --
sass = require "gulp-sass"
autoPrefixer = require "gulp-autoPrefixer"
sourceMaps = require "gulp-sourcemaps"
liveReload = require "gulp-livereload"
minifyCSS = require "gulp-minify-css"
gulp.task "sass", ->
stream = gulp.src("sass/dashboard/application.css.scss")
.pipe(sourceMaps.init())
.pipe(sass({includePaths: "path/to/code", sourceComments: true, errLogToConsole: true}))
.pipe(autoprefixer())
.pipe(rename("application.css"))
if ["production, staging"].indexOf(railsEnv) != -1
stream = stream
.pipe(minifyCSS())
stream.pipe(gulp.dest("../public/assets"))
.pipe(sourceMaps.write("."))
.pipe(liveReload())
gulp production
This task will run after we’ve built our production assets using [CODE]RAILS_ENV=production gulp[/CODE]. In addition to those tasks, gulp production will also have to do some production-specific things. We’ll use [CODE]gulp-rev-all[/CODE] to set up our files for caching, and to create a cache manifest to map our cached files to the real URLs.
-- CODE language-coffeescript --
gulp.task "production", ->
gulp.src(["../public/*assets/**"])
.pipe(revAll()
.pipe(gulp.dest("../public/assets/production"))
.pipe(revAll.manifest())
.pipe(gulp.dest("../public/assets/production"))
gulp publish
We’re using AWS to store our assets. We use Cloudfront as our CDN, which is set up to read from our AWS bucket. This task uses gulp-awspublish to gzip, cache our upload, and send our assets to AWS:
-- CODE language-coffeescript --
gulp.task "publish", ->
publisher = awspublish.create(bucket: 'your-s3-bucket-name')
gulp.src("../public/assets/production/**")
.pipe(awspublish.gzip())
.pipe(parallelize(publisher.publish({"Cache-Control": "max-age=31536000, public"}), 20))
.pipe(publisher.cache())
.pipe(awspublish.reporter())
Monkeypatching Rails
Since we’re changing where all of our assets are, we need to update some of our helper methods to find the correct assets from our Gulp manifests. Our [CODE]rev-manifest.json[/CODE] will map our cached URLs to the originals. We’ll need our Rails URL helpers to intercept the cached URLs from the manifest first if they’re available. We’re going to make an [CODE]AssetManifest[/CODE] class with all of the methods we need in [CODE]config/initializers/asset_manifest.rb[/CODE]:
-- CODE language-ruby --
class AssetManifest
def self.manifest
if File.exists?("rev-manifest.json")
@manifest ||= JSON.parse(File.read("rev-manifest.json"))
end
end
def self.stylesheet_path(url)
if AssetManifest.manifest
url += ".css" unless url.end_with?(".css")
AssetManifest.manifest[url] || url
else
url
end
end
def self.javascript_path(url)
if AssetManifest.manifest
url += ".js" unless url.end_with?(".js")
AssetManifest.manifest[url] || url
else
url
end
end
def self.asset_path(url)
if AssetManifest.manifest
AssetManifest.manifest[url] || url
else
url
end
end
end
This class helps us by mapping our asset filenames to the cached versions of the filenames produced by [CODE]gulp-rev-all[/CODE].
And then in your [CODE]app/helpers/application_helper.rb[/CODE]:
-- CODE language-ruby --
def stylesheet_link_tag(url, options={})
url = AssetManifest.stylesheet_path(url)
super(url, options)
end
def crossorigin_javascript_include_tag(url, options={})
url = AssetManifest.javascript_path(url)
super(url, options)
end
def image_tag(url, options={})
url = AssetManifest.asset_path(url)
super(url, options)
end
def image_path(url, options={})
url = AssetManifest.asset_path(url)
super(url, options)
end
def image_url(url, options={})
url = AssetManifest.asset_path(url)
super((ActionControllerBase::.asset_host || "") + url, options)
end
This way, we’ll call our [CODE]AssetManifest[/CODE] methods, giving us the correct URLs, and then call [CODE]super[/CODE] so we call the original Rails helper methods.
Setting up Capistrano 3 to use Gulp
We now need to hook up deployment of our site. For this, we’ll use Capistrano 3. In your [CODE]config/deploy.rb[/CODE] we’ll have to run our precompile script and upload our Gulp manifest to our machines:
-- CODE language-ruby --
namespace :assets do
desc "Publish assets"
task :publish do
run_locally do
execute "script/precompile.sh #{fetch(:current_revision)} #{fetch(:stage)}"
end
end
desc "Transfer asset manifest"
task :manifest do
on roles(:all) do
# All roles need to be able to link to assets
upload! StringIO.new(File.read("public/assets/production/rev-manifest.json")), "#{release_path.to_s}/rev-manifest.json"
end
end
after "deploy:updated", "assets:publish"
after "deploy:updated", "assets:manifest"
end
The [CODE]deploy.rb[/CODE] file is going to look for a [CODE]precompile.sh[/CODE] script, where precompiling assets will take place. That’s where you’ll run your [CODE]gulp[/CODE] tasks to compile your assets.
-- CODE language-bash --
#!/bin/bash
NEW_SHA=$1
RAILS_ENV=$2
mkdir -p precompile
cd precompile
git clone .. . || git fetch
git checkout $NEW_SHA
cd frontend
cp -r ../path/to/frontend/dependencies .
export RAILS_ENV=$RAILS_ENV
gulp install
gulp default
gulp production
gulp publish
Retrospective
We’re really happy with the results we’ve had since switching over to Gulp. Asset compilation times have gone down from minutes to seconds which is an amazing improvement. Local development is running more smoothly due to our site live reloading. And we now have access to our JavaScript sourcemaps which makes JavaScript debugging way easier. Not to mention we support JavaScript sourcemaps for better stacktraces at Bugsnag!
———
Hooking up Gulp as our asset pipeline has been very helpful for us. Let us know if this post ends up helping you change your asset pipeline by tweeting at us or emailing us!
