Generate Typescript Types from Protobuf files

Published on .

#tech #CI #codegen #frontend #protobuf #typescript

In this article, I show how to use the ts-proto package to generate API response and payload types from protobuf files. I export only subset of all types generated, and do that from a single index.ts file for the ease of consumption (import). Lastly, we create an internal npm package with generated types, so that build process and internal folder structure are concealed from the main application code which uses generated types.

Problem Statement

In Andromeda Security SPA (single-page or, simply, web application), we use two types of APIs: GraphQL and REST. GraphQL powers read-only views, such as dashboard and CRUD list, while REST is used for configuration. The mighty gang of GET, PUT and POST methods powers our modal windows.

Web-app codebase is written in TypeScript and requires domain specific types for code modules working with API servers, or, as we had for some time, hardcoded types, matching backend ones.

Hardcoded types are not only frustrating to write and maintain, but are also subject to regressions, whenever backend types change without introducing matching updates to the UI code.

We can catch errors like that with thorough integration tests, but it is more expensive and labor-intensive than making sure our TS project compiles correctly on every CI run. On top of that we get type-aware code suggestions in an IDE.

For our specific case, we want to generate types, interfaces and enums only, without any implementation code, such as RPC clients wth encode/decode or json methods.

We don’t want to expose elaborate folder structure of original protobuf files. In fact, we are interested in a subset of types only, and would like to be able to import them from a single index file, as we usually do with any third-party JS library.

Solution

ts-proto Plugin

To generate TypeScript types from protobuf I have chosen the popular ts-proto package, which is a protoc plugin. I also briefly tried ts-protoc-gen, but it has a shorter list of options and seems to be incapable of generating implementation-free types.

List of ts-proto options used:

  • onlyTypes=true — to skip implementation code generation. Implies false for outputEncodeMethods, outputJsonMethods and outputClientImpl flags
  • outputIndex=true — to create index files for every folder with proto files (protobuf namespace)
  • stringEnums=true — to use string enums instead of numeric ones
  • useOptionals=messages — mark object-type fields as optional with fieldName? syntax, rather than having T | undefined as a type. This allows for partial interfaces creation
  • useSnakeTypeName=false — generate names for nested messages/interfaces without using underscore as a separator
  • comments=false — remove comments from generated code
  • esModuleInterop=true — for tweaked CommonJs module imports in ESM module environment
  • env=browser — at the moment we don’t use generated types in a node environment

Let’s introduce a sample proto files folder structure we are working with:

- proto/
  - api/
    - config/
      - provider/
        - provider.proto
      - user/
        - user.proto
      - options.proto
  - metrics/
    - ...
  - thirdparty/
    - buf/
      - ...

bash

Here proto/ is the root folder for all protobuf files we use in the application (across the application stack). proto/api/config contains type definitions we need for the UI code, and thirdparty is the folder with third-party protobuf files defining structures and types our own protobuf files import and use.

Even though we are interested in types defined in a proto/api/config subfolder only, we need to pass the root proto/ folder path to protoc CLI command as an -I (--proto_path) flag. So that protoc could resolve all relative imports when compiling code.

The output folder structure created by ts-proto replicates the structure of the root protobuf folder, passed as the -I flag value.

With the outputIndex option every generated subfolder (which represents a protobuf namespace) gets an index file generated in the root output folder, with the index file name representing a path to a corresponding subfolder it re-exports stuff from:

- ts-proto-output/
  ... nested folders
  - index.api.ts
  - index.api.config.ts
  - index.api.config.provider.ts
  - index.api.config.user.ts
  - index.metrics.ts
  - index.thirdparty.ts
  - index.thirdparty.buf.ts
  - index.ts

bash

Now, if your intuition says Let’s import what we need from the index.ts!, you are not alone. Let’s see what is exported from the top level index file:

export * as api from "./index.api";
export * as metrics from "./index.metrics";
export * as thirdparty from "./index.thirdparty";

ts

Top level index.ts re-exports exports of all immediate subfolders in the root folder. All subfolder exports are named or, rather, exported as a namespace named after the folder those types are defined in.

Note how there is no line for the index.api.config file we are interested in. That’s because it’s two levels deep from the root: proto/api/config/.

Now a subfolder’s index file looks like this:

export * as provider from "./index.api.config.provider";
export * as user from "./index.api.config.user";
export * from "./api/config/options.ts";

ts

Very much the same thing, but, unlike subfolder re-exports, individual file exports are re-exported with a wildcard (as is).

Now to import a generated User type in our application code, we need to use the following syntax:

import { api } from '/ts-proto-output'; // "api" namespace, representing proto/api subfolder types

/* nested types and namespaces (subfolder) available via dot notation */
const user: api.config.user.User = {};
const provider: api.config.provider.Provider = {};
const options: api.config.Options = {};

// or, alternatively:

import type { Options } from './ts-proto-output/index.api.config'; // direct import from correct namespace
import type { User } from './ts-proto-output/index.api.config.user';
import type { Provider } from './ts-proto-output/index.api.config.provider';

const secondUser: User = {};
const secondProvider: Provider = {};
const secondOptions: Options = {};

// also combination of the above can be used

ts

Both import approaches require knowledge of the generated files folder structure (namespaces) and tightly couples application code with the original protobuf files folder structure. All exports are scoped by subfolder path (sequence of namespaces), expressed in an index filename or an exported object (namespace) properties path.

To be fair, this is ok for the most projects, since protobuf files structure is not expected to change that often and subfolder structure is meaningful and semantic, but ideally we would want to use the following syntax in our generated types imports:

import type { User } from './ts-proto-output';

ts

where all types, interfaces and enums are re-exported as is from a single index.ts file.

Effectively we would like to “flatten out” all exports from a proto/api/config subfolder — the only proto subfolder we are interested in.

Custom index File

All we need to do is to create another index.ts file in a folder next to ts-proto-output/ which re-exports all (and only) types we are interested in:

// subfolder index files exporting individual files only
export * from './ts-proto-output/index.api.config.user';
export * from './ts-proto-output/index.api.config.provider';

// individual files
export * from './ts-proto-output/api/config/options';

ts

This file is written manually (hardcoded), and care is necessary when writing it. We would like to avoid collisions and unintentional named (namespaced) exports (such as user, provider or config we saw above), that’s why we export from:

  • individual files with a wildcard — safest, but most laborious option
  • subfolder index files not having nested subfolders/namespaces in them

For example, if we do export * from './ts-proto-output/index.api.config' instead of an export * from ./ts-proto-output/api/config/options, along with the Options type we are after, we will export provide and user namespaces (objects), since config/ folder contains provider/ and user/ inside, and index.api.config.ts file re-exports them (and all nested folder & files) as namespaces (objects).

With ts-proto plugin and a custom index file we are able to generate types we care about, and export them from a single file, without use of namespaces.

NPM Package

In Andromeda UI team we are using a mono-repo powered by lerna, and it makes sense to move REST API generated types into an internal npm package.

Separate package allows us to hide the internal folder structure (including ts-proto-output/) from our web application code and have a separate build process defined as an npm script in a package.json file.

Generated types package folder structure:

- dist/ <-- to be populated by npm run build
- ts-proto-output/ <-- to be populated by ts-proto
- build-script.mjs <-- js script to find all proto files and run ts-proto
- index.ts <-- custom index file to "flatten out" exports
- package.json

bash

In order to export types from an npm package we need to generate a combo of .js/.d.ts files, since there is no way to export pure .ts files — which is all we had by now.

Fortunately, it’s very easy to do with a single tsup --experimental-dts command and two development dependencies: tsup and @microsoft/api-extractor.

TS config file to compile our project:

{
  "compilerOptions": {
    "target": "esnext",
    "module": "esnext",
    "moduleResolution": "Bundler",
    "declaration": true,
    "noEmit": false
  },
  "files": ["index.ts"]
}

tsconfig.json

In your package.json add type: "module" flag and the following exports block:

{
  "exports": {
    ".": {
      "import": {
        "default": "./dist/index.js",
        "types": "./dist/index.d.ts"
      }
    }
  }
}

package.json

package.json NPM scripts to build the package:

{
  "scripts": {
    "build": "npm run ts-proto-process && npm run pack",
    "ts-proto-process": "node ./build-script.mjs",
    "pack": "tsup --experimental-dts"
  }
}

json

Where build-script.mjs finds all proto files in a folder we are generating types from, and passes them to a proto CLI command as CLI arguments:

import { spawn } from 'node:child_process';
import { dirname, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';

const currentDir = dirname(fileURLToPath(import.meta.url));

const tsProtoPluginPath = resolve(
  `${currentDir}/../node_modules/ts-proto/protocProcess-gen-ts_proto`,
);

export async function runProtocGenerate(repoRootDir, targetDir, filesList) {
  let resolve;
  let reject;

  const promise = new Promise((_resolve_, _reject_) => {
    resolve = _resolve_;
    reject = _reject_;
  });

  // options explained above
  const tsProtoOptions = [
    'onlyTypes=true',
    'outputIndex=true',
    'stringEnums=true',
    'useOptionals=messages',
    'useSnakeTypeName=false',
    'esModuleInterop=true',
    'env=browser',
    'comments=false',
  ];

  const protocProcess = spawn(
    'protoc',
    [
      `--plugin=${tsProtoPluginPath}`,
      '--proto_path=proto/',
      '--proto_path=thirdparty/googleapis',
      `--ts_proto_out=${targetDir}`,
      ...tsProtoOptions.map((value) => `--ts_proto_opt=${value}`),
      ...filesList,
    ],
    { cwd: repoRootDir },
  );

  protocProcess.stdout.on('data', (data) => {
    console.log(`stdout: ${data}`);
  });

  protocProcess.stderr.on('data', (error) => {
    console.log(`stderr: ${error}`);
  });

  // fail the parent process on spawn process failure
  protocProcess.on('close', (code) => {
    if (code) {
      reject(code);
    }

    resolve();
  });

  return promise;
}

const repoRootDir = resolve(`${currentDir}/../`); // <= change this
const protoRootDir = resolve(`${repoRootDir}/proto/`); // <= and this one

// find all proto files in directory
const findProcess = spawnSync(
  'find',
  ['pathToProtoRootSubfolder', '-iname', '*.proto'],
  {
    cwd: protoRootDir,
  },
);

const findProcessError = findProcess.stderr.toString();

if (findProcessError) {
  throw Error(findProcessError);
}

const protoFilesList = findProcess.stdout.toString().trim().split('\n');

const targetDir = resolve(`${currentDir}/../ts-proto-output`); // <-- change this path

// run protoc command with ts-proto plugin
await runProtocGenerate(repoRootDir, targetDir, protoFilesList);

js

Summary

We learned how use ts-proto protoc plugin to generate TypeScript types from protobuf files, and how to export them from a custom index.ts file without the use of automatically generated namespaces.

Then we wrapped the generated types folder into an npm package with the help of tsup and experimental dts generator.

Npm packaging gave us the benefit of encapsulating the build process and verbose folder structure of types generated from proto files with protoc.

Now we can use generated TS types just like as any other third-party package. We can also change the build process, tooling or folder structure of our types package without breaking our web application (types consumer) code — as long as we keep exporting he same types, interfaces and enums from a top level index.ts file.

Happy type-safe coding!