1201 Mapzen Blog.png

Users! We love them. And we love when they use CircleCI the way it was intended. That means our designs worked, which is not always an easy feat.

But it’s even more exciting when a customer uses one of our product’s features in a way we didn’t expect.

That’s exactly what Michal Migurski, VP of Mapzen, did when he co-wrote Precog. Precog is an open-source app that lets you preview static or Jekyll-generated websites built with CircleCI before they go live.

I sat down with Migurski to talk more about open-source projects, Jekyll blogging woes, and using CircleCI artifacts in creative ways.

Alek Sharma, CircleCI: How would you describe what Mapzen does?

Michal Migurski, Mapzen: We’re an open-source mapping platform, so we focus on all the different ways in which the common technologies of web mapping can be done using open-source and open-data technologies. So: OpenStreetMap, OpenAddresses, different kinds of rendering engines, open-source technologies for routing and for search… Basically, all of that but licensed in a way that allows a lot of freedom around it.

AS: Were you embedded in the geo world before Mapzen?

MM: I go pretty deep in this particular world… I was a partner at a data viz/design firm called Stamen Design for almost 10 years. Stamen was a lot of data visualization, but also cartography and geography and all kinds of web mapping. So some of the technologies that Mapzen uses now are things that we invented at Stamen.

For example, Stamen got on the bandwagon with OpenStreetMap back in 2005 or 2006 when it hired one of the project’s founders. Mapzen has about 45 people now, and there are probably about half a dozen people here that have worked for Stamen at various times. So we’ve been aware of and close to these things for a long time.

AS: So, there’s this tool called Precog, which uses CircleCI artifacts to create and share previews of static blog pages. You co-created it. Any particular reason why?

MM: When I arrived at Mapzen, we had a Jekyll-based blog, and we still do. One of the difficulties with it was that it was assumed that everybody would just run Jekyll on their own. Which was well-founded because at the time everyone was an engineer. We had a ‘README’ about how to post to the blog, and it was like, “Install Xcode, now install the Ruby version manager.” Two hours later, you’d be able to blog. But once we had non-technical folks contributing to the blog, we couldn’t make that assumption.

Another thing that was a bummer was when people didn’t understand, or weren’t up to date with, the Ruby process. Ruby moves quickly, Jekyll moves quickly, Rails moves quickly; it was hard to just install this stuff and forget it. You had to stay on top of it.

Sometimes when people’s local RVM install fell out of date they’d be frustrated, and they’d push their changes to preview them in the dev environment. The problem with using a dev server is then you’re in the chute for prod. If you look at it on dev and you have second thoughts about what you’ve written, what do you do? You’re already plugging the exit hole for everybody else’s updates. Blog posting and content changes became a major production exercise, which is a really bad scene.

openterrain.stamen.jpg

AS: Review needed to happen, but dev was already being used; there’s almost an overloading of a phase of development there. And this is when you began creating Precog?

MM: Well, the precursor to Precog was a Git-Jekyll preview tool I developed at my previous job at Code for America. We called it Jekit, like: “Jekyll: check it before you wreck it.”

Precog is also based on what I observed Mike Bostock and “The New York Times” graphics team doing. They had an editorial process, they had to be able to look at a lot of different versions of a graphic, the editors had to approve it, and then it went out.

Mike developed a system that was based on some work he had been doing with D3 — a popular javascript library he developed — and it was essentially what you would expect if you really treated Git as an infinite undo. It provided access to Git and could look at every single version of a thing through history. Then maybe that’s the link you send to your editor, and they can say, “Oh, I like this, but what about the next version?” It was a way for them to move forward and backwards in time on the graphics that they were doing.

AS: And how did CircleCI come into that process?

MM: I think I was showing Jekit to Mapzen’s VP of engineering and saying, “Hey, we should probably set this thing up because it’ll make it a little bit saner for people to edit the blog.” He was the one who mentioned that CircleCI has an artifacts feature, and suggested there’s some way we could use that. He clued me in that there was something else there.

AS: So Mapzen was already using CircleCI?

MM: I joined in late 2015, and CircleCI seems to be the CI of record. There’s a pretty strong, quick-release engineering culture at Mapzen. Basically, everything we did had some sort of CI component to it.

AS: How long did it take to turn Jekit into Precog?

MM: Let’s see: the translation of the Jekyll tool to a CircleCI thing was a couple of days. Building the original one was a lot longer because we were effectively building our own CI. The original version of Jekit took quite a long time because we were effectively creating CircleCI and Artifacts, all on their own. The version without all that stuff is a million times simpler.

And part of that is to your credit, part of that is to GitHub’s credit because their API is super, super solid. Between GitHub and CircleCI, we essentially have a thin binding layer of code, and that was just a little bit of work.

AS: So, that’s what make-it-so.py is?

MM: Yeah, it’s a Python-based Flask app. It doesn’t even really need to know anything in particular ahead of time. You can basically feed it any path. It’s: hostname/account/<repo_name>, and with that information, it’ll go talk to the CircleCI API and say, “Hi, do you know anything about this repo?” Then, it’ll talk to GitHub and get more information about the repo, and if it finds artifacts, it will show them.

AS: And then you can get more specific with commit SHAs on the actual branches.

MM: Exactly. You can look at different branches or individual commits. If you have a private repo, you can do a GitHub OAuth in order to show private stuff. For a long time, actually, private was the only thing we did, but then we started to see that there was a use-case for public repos inside Mapzen, so we set it up for both private and public repos.

AS: We’ve touched on this, but let’s talk about how Mapzen uses Precog now.

MM: Basically, any of our repositories that generates a public static file has Precog attached to it. Just four off the top of my head: our website runs on Jekyll; that’s got static output, so we use Precog. Our documentation system uses a Python library called Mkdocs. It’s kind of “Jekyll for documentation”, basically. Same deal: it outputs the artifacts, so we use Precog to preview it. Our Mapzen.js library has an NPM-based build process, so that outputs static files, so we use Precog with that. Then, our style guide, which is our front-end pattern library, uses Jekyll, so same deal. All of those things get Precog URLs; you can look at old versions and future branches of them.

AS: So, openness is a big deal. Has it always been that way?

MM: It’s part of Mapzen’s mission, and it was always founded to be that way… [with open source], you can support uses that are totally outside those predicted areas. For us, that means we can explore a bunch of side areas. What does mapping look like for gaming companies? We’ve seen “Pokémon Go” get really huge on the basis of all this mapping data. There are companies using Mapzen to build entire gaming ecosystems. Or what about mobility? Or cars, navigation, in-dash stuff and experiments around that?

We have a lot of partners who are focusing on these different areas. They wouldn’t be able to do so with Google or somebody else, unless they went into it with a big licensing arrangement upfront.

AS: Since Precog is open source, have you seen anyone else contributing?

MM: Not tons, actually. It doesn’t exactly have any kind of viral component to it. But it’s sort of funny because the people who use it are like, “I can’t imagine my life without this thing.”

One big part people seem to like about it is you can share different versions of posts. We’ve connected it via Webhook, so when you open up a pull request in GitHub, it pings CircleCI to start the build, then pings Precog to let it know. Precog sends back a URL from where the build will end up, so you can click on that immediately, rather than going to Precog and looking through different branches.

AS: The reference to “Minority Report” is spot on, too; you’re literally seeing things before they happen. Cool, Mike; thanks so much for your time!

MM: Yeah, thank you!

Michal Migurski is VP of Engineering at Mapzen, and Alek Sharma is Dev Evangelist at CircleCI. Be sure to check out Precog for your own projects!

Image credits, from top: Mapzen; Stamen Design