Apps + Coffee Connection: Release trends worth featuring

This blog is part of a series that delivers insights from the monthly meetups of Bugsnag’s Apps + Coffee Connection (ACC) community.

Know what really animates development leaders? Talking about release cycles! The ACC meetup in March was a lively discussion about release trends and strategies for getting new code out the door with speed—without sacrificing quality and the customer experience.

Humans still play a vital role in releasing production code 

Release cycles are faster than ever, spurred on by the introduction of techniques such as continuous integration/continuous delivery (CI/CD). Over the last ten years, these advancements in automation have accelerated the pace at which releases go out, which begs the question: Are humans still necessary in the release loop?

You bet we are! While a whole host of regression tests can be used to ensure a release is stable, most companies rely on people to review a release before pulling the trigger and sending it out the door to customers. As one participant explained, “While alpha and beta releases are uploaded and deployed automatically with passing tests, we want that final check by real people before releasing production to users. There are too many variables and too much risk to do it without involving human teams.”

In addition, the “eat your own dog food” strategy continues to be a popular release approach, with companies relying on internal users to hammer on a release and shake out all the parts of testing that aren’t bugs and crashes. “Our code gets built automatically and runs through our test suite, but then we use employees to conduct testing on various mobile devices and to help find things like missing buttons,” states a participant. “You sometimes have to convince internal users that it’s worth adopting the new version, fresh off the CI/CD pipeline builds, but we highly encourage it and then monitor metrics for adoption.”

Flag this: How to release faster 

For development leaders, a common concern is how to move faster with releases and reduce the length of time required for a full release cycle. Many participants stated that it’s not unusual to spend a week with a beta release to ensure it’s stable before going to production.

To reduce this seven-day stretch, suggestions from participants included feature flags, A/B experimentation, and phased rollouts. One participant described how her company uses a weekly mobile release on iOS and Android by using a combination of all three techniques, emphasizing that a lot of analytics monitoring is required to maintain this strategy.

While all three release strategies are gaining traction, feature flagging was highlighted as a popular way to accelerate rollout. It’s also viewed by some engineers as easier to use than phased rollouts, due to the simple toggle used to turn a feature on and off. 

“We do very aggressive feature flagging that allows code changes to go into production months ahead of a normal release schedule. We have 100-200 concurrent feature flags in production builds, which is code we intend to release but aren't confident enough to release to 100% of the code base yet,” stated one participant. “As long as that code doesn’t screw up other parts of the app, then the code changes become live features.”

Nonetheless, not everyone is comfortable with feature flags yet. Some teams are interested in, but haven’t adopted, this technique because obstacles exist within their organization. 

“One blocker is knowing which product, or products, to use because there’s a high cost associated with bringing on the wrong solution,” stated a participant. “Senior leaders are looking at and open to feature flags, but they want to talk to people who are actually using it before they commit.” 

Best practices to address feature flag challenges

Turning on and off features is easy. The more challenging aspects of feature flagging are having confidence that: 

1. You haven’t left features in your application that should have been turned off already, i.e. dead code; and
2. You understand whether features are healthy and doing what they’re supposed to do. 
   

To combat the first challenge, a best practice is to make feature flag removal part of your engineering strategy. “You don’t want to leave these landmines in the code, so you should either remind engineers to remove them regularly or set a time each quarter to clean up the code base,” one participant shared. “Someone might turn a feature flag off accidentally for a feature that everyone thinks is on, which is a bug in the making. That’s why you should always remove old flags and never, ever recycle a flag, as that can cause big issues with older app versions.”

The second challenge around healthy features speaks to measurement and monitoring of analytics, either by building your own method or using off-the-shelf solutions such as LaunchDarkly. However, these tools may not be enough because feature health can often be challenging to assess—especially when attempting to attribute health metrics to specific features or experiments. 

A lot of companies struggle with this challenge and have come to rely on correlation, alerting, and home-built visualization and analytics tools, none of which are perfect. ‍

Have feature flags? Consider hiring a Feature Manager

Not surprisingly, an abundance of feature flags can introduce an abundance of issues. That’s why the use of feature flags often comes with a new position: Feature Manager. Similar to a Release Manager, this role is the gatekeeper for software going into customers’ hands and is the keymaster for experiment and feature flags. 

As one participant stated, “We introduced this role when a lot of feature flags started to be used, and this person’s job is to determine when a feature goes to certain geographies or an increasing percentage of the user base.”

Feature Managers can be invaluable for managing when features are turned on and off and for owning a shared calendar so everyone knows when to expect feature flags to be enabled. Feature Managers can also help align competing priorities around the overall health of the app and the success of new features. ‍

Curious to learn more about the ACC community? Register here to join our vibrant community and monthly discussions.