In any technology company one of the fundamental aspects of its identity is the technology stack, and programming language that it’s built on. This is what defines types of tools that are fair game, and more importantly, defines the types of engineers who are hired and capable of succeeding there.
Back in the middle of the last decade, when Shutterstock had its beginnings, the tech team was made up primarily of die-hard Perl developers. The benefits of CPAN, and flexibility of the language were touted as the primary reasons why Perl was the right tool for anything we wanted to build. The only problem was our hiring pool was limited to people eager to work with Perl– and although the Perl folks who joined us were indeed some of our most passionate and skilled engineers, there were countless engineers outside the Perl community who we totally ended up ignoring.
Fast forward to the last few years here, and Shutterstock has become a much more “multilingual” place for software engineers to work. We have services written in Node.js, Ruby, and Java; Data processing tools written in Python; a few of our sites written in PHP, and apps written in Objective-C.
Even though we have developers who specialize in each language, it’s become increasingly important that we remove the barriers to letting people work across multiple languages when they need to, whether it’s for debugging, writing new features, or building brand new apps and services. At Shutterstock, there have been a few strategic decisions and technology choices that have facilitated our evolution to the more multilingual programming environment and culture we have today.
Service Oriented Architectures
One of the architectural decisions we made early on to support development in multiple languages was to build out all our core functionality into siloed services. Each service could be written in any language while providing a language-agnostic interface through REST frameworks. This has enabled us to write separate pieces of functionality in the language most suited to it. For example, search makes use of Lucene & Solr, and so Java made sense there. For our translation services, Unicode support is highly important, so Perl was the strongest option there for us.
Between languages there are numerous frameworks and standards that have been inspired or replicated by one another. When possible, we try to use one of those common technologies in our services. As mentioned above, all of our services provide RESTful interfaces, and internally we use Sinatra-inspired frameworks for implementing them (Dancer for Perl, Slim for PHP, Express for Node, etc). For templating we use Django inspired frameworks such as Template::Swig for Perl, Twig for PHP, and Liquid for Ruby. By using these frameworks we can help improve the learning curve when a developer jumps between languages.
When it comes down to the nuts and bolts of actually running code in a particular language, one of the obstacles that blocks new developers from getting into it is all the technical bureaucracy needed to manage each runtime — dependency management, environment paths, and all the command line settings and flags needed to do common tasks.
The tool we use at Shutterstock to simplify all this is Rockstack. Rockstack provides a standardized interface for building, running, and testing code in any of its supported runtimes (currently: Perl, PHP, Python, Ruby, Java). Have a java app that you need to spin up? Run “rock build” and “rock run”. Have a Perl service you want a java developer to debug? “rock build”, “rock run”.
Another major benefit of using Rockstack, is that not only do our developers get a standard interface for building, testing, and running code, but our build and deployment system only has to deal with one standard set of commands for running those operations as well for any language. Rockstack is used by our Jenkins cluster for running builds and tests, and our home-grown deployment system makes use of it for for launching applications in dev, qa, and production.
One of the biggest obstacles for people jumping into a new language is the cognitive load of having to figure out all the details of setting up and working with the development environment for that language. Once you remove that burden, people can actually focus their energy on the important engineering problems they need to solve.
In order to create a standardized method for testing all the services we have running, we developed (and open sourced!) NTF (Network Testing Framework). NTF lets us write tests that hit special resources on our services’ API’s to provide status information that show the service is running in proper form. NTF supplements our collection of unit and integration tests by constantly running in production and telling us if any functionality has been impaired in any of our services.
In addition to tools and frameworks, we also support our developers in learning and evolving their skillsets as well. On a regular basis, we’ll have internal meetups for Shutterstock’s Node developers, PHP Developers, or Ruby developers where they give each other feedback on code in progress, share successes or failures with third-party libraries, and polish up the style guide for their language. These meetups are a great way for someone new to a language to ask questions and improve on their coding skills.
Part of what makes it easy to jump into another language is that all the code for every Shutterstock site and service is available for everyone to look at on our internal Github server. This means that anyone can review anyone elses code, or check out a copy and run it. If you have an idea for a feature, you can fork off a branch, and submit a pull request to the shepherd of that service. Creating this sense of openness with our code helps prevent us from creating walled gardens, and encourages people to share ideas and try new things.
Even though language-agnostic engineering comes with some nice benefits, it’s crucial to bring a modicum of pragmatism to this vision. A completely language agnostic environment may be idealistic and impractical. Allowing developers to build services and tools in any language that interests them may lead to a huge amount of fragmentation. Having 50 tools written in 50 different languages would be a nightmare to maintain, and would kill any opportunities for code reuse between them. Additionally, with a greater breadth of technologies, it becomes much more difficult to have people on hand with the depth of knowledge needed to lead initiatives with them.
As a matter of practicality, we keep a list of Preferred Technologies which is broad enough to provide plenty of choice, but narrow enough so that we can trust we’ll have plenty of expertise on hand. If a new technology is vetted and deemed valuable it will be considered for addition to this list. However if one developer wants to go and write a new site in Haskell, they’ll probably be shot down*.
*we have nothing but respect for Haskell
Although we want to make it easy for all of our developers to work in any of our common languages, there’s always going to be a need for specialists. Every language is going to have its nuances, buggy behaviors, and performance quirks that only someone with extensive language experience will be able to recognize. For each of our preferred technologies, we have several people on hand with deep knowledge in it.
* * *
Since Shutterstock is built on a plethora of services, any one of our sites may be receiving data that came from something built in Perl, Java, Node, or Ruby. If one of those sites needs an extra tweak in an api resource, it’s incredibly helpful when a developer can jump in and help make the necessary change to any of those services regardless of the language it was written in. When developers can work in this way, it helps ease dependencies between teams, which helps the organization move faster as a whole.
Many of our strategies and tools are designed to help give engineers more language agnostic skills to better work across multiple languages. Whether it’s frameworks that share standards, build and runtime tools that work across languages, architecture strategies, or testing frameworks, having common approaches for all these things allows everyone in the organization to work together, instead of siloing themselves off based on language-specific skillsets.
As the world of programming languages becomes much more fragmented, it’s becoming more important than ever from a business perspective to develop multilingual-friendly approaches to building a tech company. Some of the tools and processes we developed at Shutterstock have helped us move in that direction, but there’s a lot more that could be done to facilitate an environment where the tech stack of choice isn’t a barrier to bringing in talent.