API Definition Library

This library is used to define APIs.

Topics

• Core Concepts
  • Minimal API
  • API Input Data
  • API Output Data
  • Short-circuit Response Processing
    • Returning Errors
    • Custom Errors
    • Dynamic Errors
    • Return Success Response
    • Custom Success Status
  • Database Transactions
    • Pre and Post Commit Processing
    • Gotcha: Shared State
    • Gotcha: Expensive Computation
  • API Path
  • Long API Descriptions
  • Swagger Interactive Documentation
  • One-Time Setup
  • Asynchronous Processing
  • Cross Origin (CORS)
  • Calling other APIs
  • Sentry Context
• Niche Concepts
  • Other API Input Data Options
  • Custom Middleware
  • Gotcha: Sharing Schemas
  • Localhost Cross-Origin Resource Sharing (CORS)
• Appendix

Core Concepts

Minimal API

Define a new API by subclassing UnauthenticatedAPI and implementing at least these required members:

• PATH - the HTTP path used to call the API (more on this later). Should be camel-case (no underscores).
• DESC - a human-readable description of what the API does
• RESPONSE - describes the shape of the output JSON (more on this later)
• computeResponse() - processes the request and returns the response

class WhatTimeIsItAPI extends API {
  static PATH = '/whatTimeIsIt'
  static DESC = 'Returns the current date string'
  static RESPONSE = S.obj().prop('epoch', S.double)
  async computeResponse () {
    return { epoch: new Date().getTime() / 1000 }
  }
}

API Input Data

APIs can receive input data from a JSON-formatted HTTP request body. API input MUST ALWAYS be validated. To streamline this, API inputs must be described using Schema (S) like this:

class AddNumbersAPI extends API {
  static PATH = '/add'
  static DESC = 'returns the sum of a bunch of numbers'
  // this API only takes numbers and arrays of numbers, but a schema
  // can describe arbitrary JSON data including complex objects
  static BODY = {
    num1: S.double.desc('some number'),
    num2: S.double.default(10)
      .desc('another number; if omitted, a default value will be used'),
    more: S.arr(S.double).optional()
      .desc('optional array of more numbers to add')
  }

  static RESPONSE = RESPONSES.UNVALIDATED

  // Input data is validated prior to computeResponse() being called. If any
  // input is invalid, then an HTTP 400 response is returned. On test servers,
  // the response body will also include a description of the error.
  async computeResponse () {
    const numbers = (this.req.body.more || []).concat([
      this.req.body.num1, this.req.body.num2])
    let sum = 0
    for (let i = 0; i < numbers.length; i++) {
      sum += numbers[i]
    }
    return sum
  }
}

Notice that inputs can be configured to use a default value if none is provided (see num2). Inputs can also be configured to just have their value be undefined if omitted with the optional() marker (see more).

The BODY, PATH_PARAMS, HEADERS, and QS properties all support a mapping of field names to Schema objects. For advanced usages like specifying a default value or description, the mapping can be replaced with an Object schema S.obj({}). For example:

static BODY = S.obj({ a: S.str })

is equivalent to

static BODY = { a: S.str }

Gotcha: desc() should be called on the type to describe a property.

static BODY = S.obj({
  x: S.double.desc('describes x'),
  y: S.double
}).desc('describes BODY')

API Output Data

APIs can send output data in a JSON-formatted HTTP response body. To do this, simply return an object from the computeResponse() method:

  // when an object is returned, it is automatically converted to a JSON string
  // and the HTTP response's Content-Type header is set to application/json
  async computeResponse () {
    return {
      canHaveArbitraryJSONContent: true,
      hello: 'world',
      address: {
        houseNumber: 123,
        street: 'Bush St'
      },
      walkScore: 3.14
    }
  }

JSON response bodies must also be validated. The response schema documents the expected output, as well as double-checks the correctness of output generated by the API. Unexpected keys are considered as invalid. If you do not define an output schema, then no output is permitted. Output validation can be performed the same as input validation:

  // The RESPONSE getter defines which properties should be present in the
  // returned data. Only the defined properties are validated and sent. Any
  // extra properties would result in a 500 response.
  static RESPONSE = {
    canHaveArbitraryJSONContent: S.bool,
    hello: S.str,
    address: S.obj({
      apartmentNumber: S.int.optional(),
      houseNumber: S.int,
      street: S.str
    }),
    walkScore: S.double
  }

The response schema values can optionally be documented like this:

  static RESPONSE = S.obj({
    dragons: S.arr(S.str.desc('Dragon ID')).desc('Your dragons'),
    guineaPigs: S.arr(S.str.desc('name').desc('Your dragons')),
    optionalValue: S.int.desc('describe optional fields like this')
      .optional()
  }).desc('the mythic creatures you have collected')

If the API has no response body, then RESPONSE should be omitted.

Short-circuit Response Processing

Complex APIs will have a computeResponse() which calls other functions. When an error occurs deep in the call stack, it can be unwieldy and error-prone to pass that error information back up the call stack simply to return that information from computeResponse(). It can be useful to simply end request processing at any point by throwing an exception which specifies the output.

class ThrowToReplyAPI extends API {
  static PATH = '/throwToReturn'
  static DESC = 'returns some data by throwing'
  static QS = { shouldError: S.bool }
  static RESPONSE = RESPONSES.UNVALIDATED

  async computeResponse () {
    this.help1()
  }

  help1 () { this.help2() }

  help2 () {
    // imagine we're deep in a complicated call stack but discover we know the
    // response we need to send... instead of painstakingly passing it back up
    // the call stack, we should use an exception to simply send the output
    // from here
    if (this.req.query.shouldError) {
      throw new BadRequestException('run away!')
    } else {
      throw new RequestOkay({ hello: 'world' })
    }
  }
}

These may also be thrown from an API's constructor.

Returning Errors

This library defines several exceptions classes, for example:

• InternalFailureException
• BadRequestException
• UnauthorizedException
• ...

These exceptions are exported in the EXCEPTIONS variable.

import { EXCEPTIONS } from '@pbvision/fastify-firestore-service'
const { UnauthorizedException } = EXCEPTIONS

APIs must list the errors they may throw in ERRORS field.

class DupErrorCodeAPI extends API {
  static PATH = '/dupErrorCode'
  static DESC = 'API with multiple error with the same status code.'
  static BODY = {
    exception: S.str
  }

  static ERRORS = {
    NotFoundException,
    DupNotFoundException
  }

  async computeResponse (req) {
    this.setSentryTag('test', 'me')
    this.setSentryTag('another', 'test')
    this.setSentryUserInfo({ email: 'x@example.com', id: '123' })
    if (req.body.exception === 'notfound') {
      throw new NotFoundException() // default error message "Not found"
    } else {
      throw new DupNotFoundException('custom message')
    }
  }
}

Custom Errors

Custom errors should subclass RequestError base class, and provide STATUS and SCHEMA.

class SessionExpiredException extends RequestError {
  static STATUS = 403
  static SCHEMA = S.obj().max(0)

  constructor (message = 'session expired', data = {}) {
    super(message, data)
  }
}

Dynamic Errors

APIs may need to re-throw an exception returned from an external API. Dynamic error exceptions can be instantiated with new RequestError(message, data, code).

Return Success Response

Similar to short circuiting return errors, exceptions can be thrown to return success responses. It is useful to avoid a long stack of return statements. Success responses should throw new RequestOkay(data). Data must match the RESPONSE schema.

Custom Success Status

A non-standard 2xx status code can be used to indicate request success. The RESPONSE field must be assigned with a subclass of RequestDone like this

class NonStandardResponse extends RequestDone {
  static STATUS = 201
}

Database Transactions

DatabaseAPI wraps your request in a database transaction context from the Firestore ORM library:

class SomeAPI extends DatabaseAPI {
  static IS_READ_ONLY = false
  async computeResponse () {
    const someDoc = await this.tx.get(SomeModel, ...)
    // changes are automatically saved when the request completes (assuming the
    // transaction succeeds, i.e., it doesn't encounter excessive contention or
    // some other problem)
    someDoc.someField += 1
  }
}

The per-request transaction only attempts to commit if the response code is less than 400 (e.g., HTTP 200 "OK" or HTTP 302 "Moved Temporarily"). Response codes 400 and higher are considered errors, and the transaction will be aborted (no changes will be saved).

By default, DatabaseAPI uses a read-only transaction. Set IS_READ_ONLY to false (like in the above example) to allow database writes.

Other options for the database context can be set in the CONTEXT_OPTIONS object:

  static CONTEXT_OPTIONS = { retries: 3 }

Note: If you leave the API read-only and add the inconsistentReads: true to CONTEXT_OPTIONS then the database context will not be a transaction. See Firestore ORM library's documentation for more detail.

Pre and Post Commit Processing

You can perform extra processing just before the per-request transaction attempts to commit by overriding the preCommit(respData) function (not called if the transaction is being aborted because the HTTP response code indicates an error as discussed in the previous section):

  async preCommit (respData) {
    // you can do more work within the original transaction if desired
    assert(this.tx !== undefined) // tx is still available

    const user = await tx.get(User, this.req.body.userID)
    if (!user) {
      // you can change the response code (and data)
      throw new RequestError('unknown user')
    }
    else {
      // you can also just modify the response data
      respData.userName = user.name
    }
    return respData
  }

Similarly, you can perform extra processing just after the per-request transaction successfully commits by overriding the postCommit(respData) function (not called if the transaction fails to commit):

  async postCommit (respData) {
    // you can do more work UNRELATED to the original transaction (this.tx is
    // not even available here because it isn't valid here)
    assert(this.tx === undefined) // tx has already ended!

    // like preCommit(), you can modify response data (or throw RequestDone or
    // any of its subclasses to change both the response code and data)
    respData.postCommitMsg = 'commit succeeded'
    return respData
  }

--WARNING--
- In rare cases, the post-commit hook **MAY NOT RUN** after a transaction runs
- (e.g., the machine processing the request loses power after the database
- commits the transaction but before the post-commit hook runs). Take care to
- ONLY use the post-commit handler whose logic is okay to not run (on
- rare occasions).

Gotcha: Shared State

Transactions sometimes retry due to contention, etc. It's important to not store state on this, req or other heap variables while your transaction runs, and then reference that data in a retry on accident.

class RememberingTooMuchAPI extends DatabaseAPI {
  static NAME = 'unwise memory use'
  static DESC = 'shares state across tx attempts and requests'
  static PATH = '/overshare'
  static IS_READ_ONLY = false
  static CONTEXT_OPTIONS = { retries: 3 }
  static BODY = {
    numTries: S.int.min(0)
  }

  static RESPONSE = {
    numTries: S.int.min(0),
    numTriesOnThisMachine: S.int
  }

  static numTriesOnThisMachine = 0
  constructor (fastify, req, reply) {
    super(fastify, req, reply)
    this.numTries = 0
  }

  async computeResponse (req) {
    // The API instance (this) is created ONCE for each request. It isn't
    // recreated if the transaction retries. So any changes you make to
    // `this` will persist and be visible across retries!
    this.numTries += 1

    // Updating a static variable like this will affect ALL requests being
    // processed by the same machine! Module variables are stored in RAM and
    // are never cleared. They're only in their initial state when a machine
    // first starts. (This is the case regardless of whether you're using
    // transactions).
    this.constructor.numTriesOnThisMachine += 1

    if (this.numTries < req.body.numTries) {
      // force the tx to retry to demonstrate a point
      const err = new Error()
      err.retryable = true
      throw err
    }
    return {
      numTries: this.numTries,
      numTriesOnThisMachine: this.constructor.numTriesOnThisMachine
    }
  }
}

Gotcha: Expensive Computation

It's best to do expensive computation outside transactions in order to minimize how long the transaction runs and thus minimize the opportunity for contention. Of course, this is only possible if the computation is not dependent on data from the database.

Your request should do expensive computation unrelated to the database in its constructor (tx is not defined at that point, and no transaction is running and so transaction retries will not cause your constructor to re-run).

class SomeAPI extends DatabaseAPI {
  constructor (fastify, req, reply) {
    super(fastify, req, reply)
    this.doSomeExpensiveComputation()
    this.computeCalls = 0
    this.postComputeCalls = 0

    // the transaction hasn't started yet (or even been set yet) so the
    // expensive computation will never be re-run, even if the tx retries
    assert(this.tx === undefined) // tx hasn't started yet!
  }
  // ...
}

You can also do expensive pre-computation in the async preTxStart() function. This is more flexible than the constructor in that it is async and so you can use async/await in it, unlike the constructor:

  // this runs after the constructor but before the tx starts
  async preTxStart () {
    assert(this.tx === undefined) // tx hasn't started yet!
    // ...
  }

API Path

An API's request path is of the form /<Service Name><API PATH>. For example, a leaderboard service might have an API to get leaderboard entries at the path /leaderboard/entriesById.

Long API Descriptions

An API description may sometimes be too long to fit on a single line. Use Node's multiline string for this (also, keep in mind that markdown can be used in descriptions):

  static DESC = `
this will
get combined
into **one** string`
// The APIs description would be: "this will get combined into **one** string"

Swagger Interactive Documentation

With the help of fastify-swagger, APIs are automatically documented using Swagger UI. In addition to reviewing an APIs specification, you can also try the API from your browser (making API requests and seeing their output right in your browser!). If your service exposes many APIs, consider specifying a tag to group them in the generated documentation:

  static TAG = 'Some Group Name'

If TAG isn't specified, it will be "default". If TAG is set to null then the API will be omitted from the Swagger documentation.

One-Time Setup

Some APIs may need to do some one-time initialization work. This sort of thing can be done by defining a static setup() method on your API:

static async setup (fastify) {
    fastify.redis.defineCommand(/* add some custom local redis command... */)
}

Asynchronous Processing

A service can accept multiple requests for processing at once. However, request processing is single-threaded (in a single process). JavaScript's async/await syntax allows us to achieve lightweight concurrency; understanding this concurrency model is important, but beyond the scope of this documentation so check this async/await primer to learn more.

This is a common setup because APIs tend to spend a lot of their time blocked waiting on I/O, typically in the form of HTTP requests to other services (e.g., the database, cache or other API services). It is critical to never perform synchronous I/O as this would stall the CPU (and every other request) until the I/O completes. Use async/await to asynchronously block on I/O and enable the server to continue productively using the CPU to process other requests. Services with long-running CPU-bound requests may need to tweak (lower) request processing concurrency to reduce queuing delays as the CPU works on one request for an extended time, ignoring and starving other requests and possibly leading to unacceptably high latency.

// the "async" keyword on this method is actually very important!
async computeResponse() {
  // whenever possible, asynchronous work should be done in PARALLEL (rather
  // than serially); in this example we make two API calls at the same time
  // and then yield until both API calls have returned
  const results = await Promise.all([
    this.callAPI({/* params omitted from example */}),
    this.callAPI({/* params omitted again */})
  ])
  // use the two results to compute our response...
}

Cross Origin (CORS)

APIs which need be accessed from a web browser must indicate which hostnames they may be accessed from. This is required for compatibility with browser safety features. Without it, browsers will prevent the APIs from being requested.

You can specify the CORS origin through the CORS_ORIGIN flag. For example,

static CORS_ORIGIN = 'some.example.com'

You may also set the API to allow any origin like this:

static CORS_ORIGIN = '*'

Calling other APIs

• If you want to have one API call another, see the callAPI() helper method.
• If you want to have your API redirect to a web application and optionally pass some information in a cookie, see the redirectToWebApp() helper method.

Sentry Context

By default, query parameter, path parameter and body parameters are included as context for reports to Sentry. To disable or customize this, override the getInputsToTrackWithSentry() method. You can also use the setSentryContext(), setSentryTag() and setSentryUserInfo() methods to add custom metadata for the request.

Niche Concepts

This section explains niche functionality.

Other API Input Data Options

APIs are typically requested via the HTTP POST method. API-specific inputs are typically sent in the request body in JSON format (though some universal inputs are sent in HTTP headers, such as which app and user sent the request). API outputs are typically transmitted as JSON data in the HTTP response body.

Occasionally, it may be necessary for an API to use a different HTTP method, or different input or output types or formats. One possibility is when integrating with a third-party who requires this. However, APIs should generally be consistent and stick to HTTP POST data with HTTP request and response bodies being JSON formatted.

Here is an example API which differs from our convention in every way:

• It is requested via the HTTP PUT method
• It receives input from many different sources:
  • query string
  • request body (in a custom format, not JSON)
  • headers
  • request path
• Its response body is XML formatted

class NonStandardAddNumbersAPI extends API {
  static METHOD = 'PUT'
  static PATH = '/add/:num6/:num7'
  static DESC = 'returns the sum of a bunch of numbers'

  // we can send input as part of the query string (e.g., ?num=1&num2=...)
  // sensitive data (such as a password hash) should not be sent in the query
  // string as query strings are logged in request logs
  static QS = {
    num1: S.double.desc('some number'),
    num2: S.double.desc('another number')
  }

  // input can also be sent in the body (for POST and PUT requests); this is
  // preferred for large of complex data types like JSON objects
  static BODY = {
    num3: S.double.desc('yet another number'),
    num4: S.double.desc('a 4th number to add'),
    more: S.arr(S.double).optional()
      .desc('optional array of more numbers to add')
  }

  // headers may also be used to communicate inputs, but are not typically used
  // by APIs; they're primarily used for data sent on every request like user
  // credentials
  static HEADERS = {
    num5: S.double.desc('a fifth number to add')
  }

  // input data may also be communicated in the path itself
  static PATH_PARAMS = {
    num6: S.double,
    num7: S.double
  }

  static RESPONSE = RESPONSES.UNVALIDATED

  // Input data is validated prior to computeResponse() being called. If any
  // input is invalid, then an HTTP 400 response is returned. On test servers,
  // the response body will also include a description of the error.
  async computeResponse () {
    const numbers = (this.req.body.more || []).concat([
      this.req.query.num1, this.req.query.num2,
      this.req.body.num3, this.req.body.num4,
      this.req.headers.num5,
      this.req.params.num6, this.req.params.num7
    ])
    let sum = 0
    for (let i = 0; i < numbers.length; i++) {
      sum += numbers[i]
    }
    // respond with an XML string instead of JSON
    return `<sum>${sum}</sum>`
  }

  // add a custom content-type parser that will be available to just this API
  // (to handle the custom data format sent to this API)
  static async registerAPIWithFastify (fastify, fullPath) {
    function parser (req, body, done) {
      if (!body.startsWith('wacky!')) {
        done(new InternalFailureException('invalid wacky format'))
      } else {
        try {
          const jsonStr = body.substring(6)
          done(null, JSON.parse(jsonStr))
        } catch (err) {
          err.statusCode = 400
          done(err, undefined)
        }
      }
    }
    fastify.addContentTypeParser(
      'wackyCustom_thing', { parseAs: 'string' }, parser)
    super.registerAPIWithFastify(fastify, fullPath)
  }
}

Custom Middleware

If desired, each API can be run with custom middleware and other fastify options. To do this, you'll need to override the register() method and call app.register() as desired.

Gotcha: Sharing Schemas

In order for multiple APIs to share these parameters via inheritance:

  static get BODY () {
    return S.obj({
      leaderboard: S.str,
      hasTiebreakers: S.bool.default(true).desc(
        'must be set to true if the leaderboard uses tiebreakers')
    })
  }

Notice that BODY is defined as a getter function, not a static member variable. The reason is that subclasses will call super.BODY to extend the object. They each need their own copy of the schema (which the getter creates and returns). If they both built from the same copy, it would cause internal problems with the schema object.

Localhost Cross-Origin Resource Sharing (CORS)

When running a service locally for testing purposes, it will enable CORS requests for all APIs from http://localhost:3000. This facilitates testing of web applications also running locally that depend on the service. In the cloud, CORS is not enabled (unless you choose to enable it).

Appendix

The samples in this readme can be found in the APIs defined for unit testing this library in ../examples/docs.js.

When necessary it is possible to use static getter functions instead of static member variables. For example, here's a contrived example that only accepts "n" as an input on localhost:

  // we use a getter function here so that we can more convenient define th
  // body using more complex logic than usual
  static get BODY () {
    if (process.env.NODE_ENV === 'localhost') {
      return S.obj({ n: S.int })
    }
    return S.obj()
  }