Today, we are going to talk about something that is very near and dear to all of our hearts, Application Programming Interfaces. Now, I know what you are thinking, and I’ll concede. You’re right! Software integration is quite possibly the most boring thing ever. It is technical, it is confusing, and most people don’t know why it exists. However, in the software process, this is one of the most crucial steps that will quite literally, make or break your application. Decisions made now can have impacts for the next 10 years, and sadly, in a lot of software efforts, this is rushed, and not well thought out.
Application Programming Interfaces (APIs) are actually kind of an interesting time in software development. On one side you have your data guys. They are a shy group but they can calculate the airspeed velocity required for an African Swallow to carry a coconut, drunk, without an eraser. You’d check their work, but you can’t even understand the symbols on the paper. On the other side, you have your creative, user experience, and web developers. These guys are so cool… they have awesome glasses, they talk with their hands, and when they write on the whiteboard you always take a picture, and consider framing the genius they so eloquently captured. Don’t, worry, there is adult supervision, in the room too is the business and project management team, keeping everyone in check.
But, there is a problem, these two software groups live in completely different worlds, both groups can write software, but the paradigms are completely different. Most programmers live in one camp or the other. There are very few ‘full-stack’ developers, and most would prefer to not be full-stack developers. In the last 20 years, the bar for great software is so high that specialization is necessary. Alas, with this application, the only way these two teams are going to talk, the only way their brilliance can come to life is through this mysterious element, the “API.”
What is an API?
An Application-Programming Interface (API) is a piece of software that separates two systems from one another. It has several purposes, which we’ll briefly discuss, but its main purpose is to act as a service for a system, or group of systems, to outside programmers. It sounds simple right? It’s just a bridge between two systems.
Well, like a bridge, if you mess it up, cars fall into the water, people complain, and everyone thinks your software is terrible. If you build it perfectly to specification nothing happens, no one complains, and you breathe a sigh of relief. However, and this is rare, if you make a really awesome bridge, think through it really well from both sides, it is marvel of human achievement, a landmark fit for a postcard!
Don’t be fooled, this is not easy; there are many problems when trying to develop an API, especially when it comes to big data. It’s not easy. All the powerful software systems on the market are designed for the CFO and COO. They can be constrained into neat pre-defined structures and there are 100 products, and 10 free ones, on the market that can do whatever you’d like out of the box. These, of course, don’t work with Social Media data, and definitely not with bleeding edge software. Here is a short list of the challenges:
Data Scientist Perspective
- Queries Are Intense – Queries can be very fast, but when executed over several terabytes of data, it slows down, tremendously. In a world where milliseconds are counted, you have to be careful.
- Multiple Technologies Required – Unlike enterprise software where one dataset pre-computes all the answers to be fast. You might have to ask a graph database one question, to figure out what to ask your document store next, to then compute the result set for the API.
- Technologies are Constantly Changing – In the enterprise world, it takes an act of Congress to upgrade a server. In the big data world, software releases, which are necessary, happen on a bi-weekly basis.
User Interface Perspective
- Simplicity – They want an API that is simple, easy to use, and consistent. This allows them to spend more time making cool things, instead of worrying about complicated nuances to your API.
- Business Logic – They want as much business logic to be handled on the API side as possible. The less code they have to write, the better.
- Fast – A slow API makes their software slow. Slow software is painful, really, really painful.
- Stay the Same – One small change in the API (a field moved, date format changed, parameter moved, etc.) and everything breaks.
- Don’t Break… Ever – Just like a bridge, it can’t break, period. Downtime in an API is a death sentence for everyone.
- Scalability – I assume that if 10 people can use it just fine, 100,000 can use it just fine too.
- Re-Usability – A common question I hear is, “you already built an API, why do you need another one”. Software is presumed to be re-usable or easily adapted. Often times, that isn’t the case without long term planning, and given the challenges above, it’s very hard to do.
- Since It’s Re-Usable – Given the point above, you are the only thing standing in the way to us finishing this project. Your API either needs to be brought up in hours, or, at minimum, they need something to test against.
- Solve Problems We can’t Anticipate – Deploying a new API can be risky. Even with a bunch of software tests, one error in your code and the software could go down. So you might as well just program solutions to problems we don’t even know exist yet.
- Business Units – Internal workings of business are complicated. They don’t necessarily share a common pattern from one another, nor, do they care about the same metrics.
All of this looks daunting, difficult, and it is. However, there are a few tricks that I would like to share with you. There is a way to structure this problem, engineer with a few principals in mind, and create a great API that everyone will love to use. In my next blog entry, we’ll discuss how we went about structuring our API and what principals guided our engineering.