Practical ReST Architecture - Part 2 - Cheat Sheet

This is my attempt to distill the majority of my ReST best practices down to a cheat sheet. I'll be expanding on each in later posts.

1. Resource URIs should be structured like so:

/{set}?param=value&param=value
/{set}/{unique id}

2. Nested resources should also follow this pattern:

/{set}/{unique id}/{set}?param=value&param=value
/{set}/{unique id}/{set}/{unique id}

e.g. /users/123/avatars/thumbnail

Don't go nuts with nested resources. They are only really required in very specific circumstances. Composite keys such as the thumbnail avatar example above is one case where they can be useful for uniquely identifying a specific resource. But where you have a surrogate or single valued natural key you should avoid nesting.

3. Use your verbs correctly.

• GET does not alter server state. Ever. It is cacheable.
• POST can have any number of side effects and can blow caches out anywhere.
• PUT is idempotent which means if you send the same request 1 time or 3 times in succession the result on the server will be the same. Use it to completely replace an individual resource or an entire set. It invalidates only caches containing those resources altered by this request.
• PATCH is also idempotent. Use it to make partial updates to individual resources, or to do upserts on sets. It invalidates only caches containing those resources altered by this request.
• DELETE removes a resource. Technically the response for a DELETE on a non-existent resource should be 404 Not Found, but  that requires extra client error handling, and because it's pretty harmless we have returned 200 OK in this circumstance for most cases. 

4. Use the Content-Type header to determine the incoming data format and handle it appropriately. Handling all types of data format just for giggles can bloat your codebase unnecessarily. What is important is that you respond with an informative response code (415 Unsupported Media Type) when the Content-Type is not one you can handle.

5. Use the Accept header to determine the format of the response. Referred to as content negotiation. Again, 415 Unsupported Media Type response when you don't support that format.

6. Use the standard http response codes. They are there for a reason. And they work like a charm. Just enough wiggle room to fit every use case I've found. Document every response code for every endpoint and have a test case for each.

7. Don't be afraid to use the response codes as a template for error types in your code (e.g. BadRequestError, NotFoundError, etc...), they cover most error situations and it helps to ensure that errors can be caught and propagated to the user with complete context.

8. Validate your inputs. We use JSON schema v4 for this. It's surprisingly flexible and powerful. Make sure your 400 error responses are in compliance with JSON schema v4 also so they can be consumed and actioned automatically or by a developer who needs a clue as to why their request resulted in a 400 response.

9. Don't validate output, it's a fools errand, if you want to check your data for corruption and repair it run a script. Don't respond to a GET request with a 500 error.

10. To avoid a chatty API an expansion query param can be used to denote the nodes of the resource graph to expand. The format for this is not standardized so you can make up your own. Here is how we do it:

/users?expand=avatars

This will return an array of users each with its related avatar resources as an array called avatars.
You can also get cute with paths down the graph using semicolon delimiters and multiple expansion using commas.

/users?expand=avatars,friends:avatars

This will return an array of users including avatars, friends, and the those friends avatars. All this in one response.
What is nice about this approach is that it isn't custom for each endpoint. It's a simple pattern that can be repeated and generic handlers can be written to make that easy.

A note on cache invalidation: Using expand gives us a different URL for caching purposes, which is ideal as long as you cache match on the URL including the query params. You do need to remember to match the first part of the URI when doing cache invalidation to invalidate all the expanded versions of the resource too.

11. There are different formats out there for returning links to other related resources. The approach we followed was to include links in the schema using JSON-schema v4 format. The schema can be retrieved by calling GET on a URI with an Accept header of application/schema+json. This is where we reach the Richardson maturity model nirvana and full Hypermedia status. But in reality I've found it of limited use beyond a verifiable form of documentation. None of our clients are traversing the graph via these links that we are aware of. They tend to just expand the graph to keep the number of calls to a minimum. That said, even if it isn't used in anger all that often it does come in very handy during the development phase and helps to clarify to developers how the resources are related.

12. Understanding query parameters is important. There are no standardized rules for how they should be used but here are the rules we try to stick with for consistency:

• Query params are used to filter lists/sets. e.g. GET /users?age=35
• The filter is reductive in nature when multiple are applied. e.g. GET /users?age=35&status=active (must be 35 and active)
• To do additive (google style) searching consider using a single query param named 'search' which includes a URL encoded search string (like you would type in google search). This string can be interpreted by your search system and has the advantage that you can embed quite complex searches that would be very unwieldy as simple query params. e.g. GET /users?search=age%3D35%20or%20status%3Dactive (age is 35 OR status is active)
• Query params can be used to modify the data returned from a single resource representation too, just as we did with the expand query param. 

13. Some forms of batch update can be performed using the PATCH method. We take the approach that a PATCH is a shallow merge not a deep merge. Which is to say that the attributes of the incoming data override those of the existing resource. This applies to lists also where you have a unique key. So if we want to add / update a list with the contents of another list (using the unique id to match elements) we make a PATCH on a list. Because a JSON object is an associative array, and an array of objects with a known unique key are more or less the same thing just represented differently we can apply the same kind of PATCH logic to update either.
For example if we want to upsert 3 friends to user 123's friend list we can do so in one call like so:

PATCH /users/123/friends
[{ id: 234 }, { id: 345 }, { id: 456 }]

14. Authorization can throw a real spanner in the works with ReST and caching. This is because resources need to be 'mediated' to only present data the user is authorized to see. This means the user's authorization now has to form part of the cache key. This can be handled without too much difficulty when caching the data if the authorization token is in the header but it does make cache invalidation tricky. You have to look at caching of authenticated resources on a case by case basis to determine how much benefit you will get from it vs the complexity. Cache invalidation is always important to get right but becomes even more so when authorization is involved as you run the risk of exposing sensitive information if you get it wrong.

15. Events and time series data can fit into a ReST paradigm by creating a POST to an imaginary set and returning a 202 Accepted response. Take our analytics reporting endpoint POST /tracking-events. In theory it's the set of all tracking events ever submitted. But there is no GET /tracking-events endpoint because we use it only for receiving the events and triggering various side-effects such as feeding the data into our analytics data stores. We use this little trick quite often when we need the client to initialize a process on the server. Turn the process into a side effect of addition to a somewhat made up resource set. It works, and keeps the API URIs from turning into verb soup.

That's all for now. I'll expand on these points in subsequent posts. Ping me if you have any questions or just want to berate me for not talking about hypermedia constraints.

:-D
Paul

Practical ReST Architecture - Part 1 - Dude where's my resource?

This is the first in what may become a series of posts on the tips and tricks I've learned while implementing a pretty strict approach to ReST architecture. Most of it will be drawn from the accumulated experience of building a Hypermedia API to power KIXEYE.com.

Hello

I'm a software architect. It's a pretty vague title I'll admit. It means, in essence, I design software systems at a macro level. I also code, a lot, because I believe good design requires one to feel the pain points of implementation in order to do better work.

Now on to the good stuff. ReST is a discipline. It is not, as most have come to believe, simply pretty URLs. I won't bang on about Roy Fielding's dissertation or the Richardson maturity model but if those things don't mean anything to you, and you think you are implementing ReST, then you probably aren't. So let's fix that.
ReST when explained by the experts can get cripplingly boring so I'll try to keep it to the stuff that matters because ReST is actually pretty badass if you grok the power in it's simplicity.

Dude where's my resource?

Firstly you need a resource. A resource in my world is a URI (Universal Resource Indicator) (implemented in an http API as a URL).
So lets say we want to retrieve the data for user 123.

Let's get it wrong a few times to illustrate a point about discipline.

1. /user?id=123
2. /users?id=123
3. /user/123
4. /getuser/123
5. /getuser?id=123

Nope, nope, nope, nope and nope.

The following is correct:

/users/123

But why? What does it matter if I call my resource /user/123 or /users?id=123 rather than /users/123 ?

Here is why: Because all individual resource URI's should follow a simple pattern.

/<set>?<query params> (for searching for results within a set)
/<set>/<unique identifier> (for an individual resource)

A set is a logical bag or list of resources, in this case all the /users. The unique identifier of the user we want is 123.

(I'm simplifying a little here, we will get into composite keys and nesting resources later, so bear with me)

Now here is why that rule exists:

POST /user
GET /user?gender=M

Those URIs seem a little off because the lack of pluralization makes it hard to tell you are working with the set of all users, not an individual user. They should read:

POST /users (add a new user to the set of all users)
GET /users?gender=M (return an array of all users with gender M)

Now this might seem like a pedantic point, (to pluralize or not to pluralize) but trust me, when you have hundreds of URIs this convention will keep you and your team sane.

Verbal

The other bad examples I gave include a cardinal sin of ReST URI construction: A verb in the resource name.

/getuser/123

other bad examples might be:

/deleteuser/123
/update/user/123
/adduser

URIs should always describe nouns. Either a logical set of resources (/users) or an individual resource within a set (/users/123). All actions on those nouns (verbs) come in the form of request methods (GET, POST, PUT, PATCH, DELETE) and are separate from the resource identifier for good reason. You want multiple verbs to act on a single noun, because a noun is a real thing, and you are doing something to that thing.

A series of tubes

And here we get to the reason why we need to be disciplined about our URIs and the crux of why one would want to have a ReST architecture at all. ReST resource representations are edge efficient. They have unique identifiers that allow us to cache them very efficiently and flush those caches at the appropriate times.
This suits the distributed nature of the internet and most http APIs very well because most are read heavy animals and caching can add a huge amount of optimization.

Here is a real world example:

1. GET /users/123 returns the user resource with unique id 123

this can be cached in memcached, varnish, name-your-edge-cache, CDN, etc...

2. GET /users/123 can be retrieved a million times without ever bothering more than the edge caches.


3. PUT /users/123 is called (the user data for user 123 is replaced)

Because we know the PUT verb means replacement, and we know from the URI that only a single resource is being replaced, and which resource it is, we can re-populate the edge caches with the new value after updating the data store.

We also know from the URI that /users/123 is an individual resource within the logical set of all /users so we can make calls to our edge caches to flush any caching of /users.
Response caches like Varnish can be used to cache query calls to logical sets like:

GET /users?gender=M

Because the number of requests for data usually far outnumbers the number of additions, updates and deletions our request cache can happily serve up most of the recurring requests, being flushed and populating only when an infrequent change is made to the set of all users. (by infrequent I mean we see thousands of requests per second and it could be many seconds or minutes between user updates)

Patterns

The cherry on the top is that because we are adhering to a strict pattern we can write generic code to do all of this without having to concern ourselves to any great extent with the actual resources. Just by virtue of the incoming request URI pattern we know some things need to happen, like flushing edge caches of resource sets.

This is one of the foundation stones of a ReST architecture. By following a strict discipline you allow optimizations both in the operational efficiency of the system and also the code itself. The ability to build upon valid assumptions accelerates development and eases maintenance.


In future posts I'll talk more about composite keys and resource nesting, content-types, json-schema, hypermedia and linking, graph expansion, batch updates, when to use query params, authorization, mediation and how that effects caching, eventing with ReST, and some tricks for working around some of the problems that seem an awkward fit for a ReST architecture.


:-)
-Paul Hill