@eve-online-tools/eve-sde

Updated June 25, 2026 · By Xaroth Brook

@eve-online-tools/eve-sde is a library that allows you to include parts of the SDE in your vite/rollup applications. It automatically downloads and processes the EVE Static Data Export (SDE) and allows you to convert and/or optimize the data specifically for your applications.

This library can be used both for Vite as well as Rollup build pipelines.

Show config examples

vite.config.tsts

import { sde } from '@eve-online-tools/eve-sde/vite'
import { defineConfig } from 'vite'

import { certificatesProcessor } from './certificates'
import { masteriesProcessor } from './masteries'

export default defineConfig({
  plugins: [
    sde({
      outputDir: 'src/generated/sde',
      tables: ['types', 'groups'],
      processors: [masteriesProcessor(), certificatesProcessor()],
    }),
  ],
})

rollup.config.tsts

import { sde } from '@eve-online-tools/eve-sde/rollup'

import { certificatesProcessor } from './certificates'
import { masteriesProcessor } from './masteries'

export default {
  input: 'src/main.ts',
  plugins: [
    sde({
      outputDir: 'src/generated/sde',
      root: process.cwd(),
      tables: ['types', 'groups'],
      processors: [masteriesProcessor(), certificatesProcessor()],
    }),
  ],
}

But why?

When dealing with the SDE, you often only want parts of it, either single files, or even parts of a single file. This means that you have to create manual processing pipelines every time you want to toy around. This library provides that easy to use interface, allowing you to get the data you need in the way that you want it.

Processors

For projects like eve-ship-tree, we would want to include some data, but not all of it. We only really care about the factions and the ships that are in the ship tree, and not at all about anything else. This can easily be accomplished by creating a processor that only fetches the data we need.

Show processor examples

masteries.tsts

import { type SdeProcessor, applyStripFields, writeSdeRecord } from '@eve-online-tools/eve-sde'

import { collectGeneratedKeys } from './utils/collect-generated-keys'

const masteriesJsonlPath = 'generated/masteries.jsonl'

export const masteriesProcessor = ({
  keepLanguages,
  fallbackLanguage,
}: {
  keepLanguages: string[]
  fallbackLanguage: string
}): SdeProcessor => ({
  id: 'masteries',
  version: [...keepLanguages].sort().join(','),
  run: async ({ generatedStream, loadStream, streamJson }) => {
    const typeIds = await collectGeneratedKeys(generatedStream, 'types')
    const out = await streamJson(masteriesJsonlPath)

    for await (const record of loadStream('masteries')) {
      if (!typeIds.has(record.key)) {
        continue
      }

      const stripped = applyStripFields(record.value, {
        keepLanguages,
        fallbackLanguage,
      })
      await writeSdeRecord(out, record, stripped)
    }

    await out.close()
  },
})

certificates.tsts

import {
  type SdeProcessor,
  type SdeProcessorContext,
  applyStripFields,
  writeSdeRecord,
} from '@eve-online-tools/eve-sde'

const certificatesJsonlPath = 'generated/certificates.jsonl'

type MasteryRecord = {
  _value?: Array<{ _value?: number[] }>
}

const collectMasteryCertificateIds = async (
  generatedStream: SdeProcessorContext['generatedStream'],
): Promise<Set<number>> => {
  const certificateIds = new Set<number>()

  for await (const row of generatedStream('masteries')) {
    const mastery = row as MasteryRecord

    for (const entry of mastery._value ?? []) {
      if (!Array.isArray(entry._value)) {
        continue
      }

      for (const certificateId of entry._value) {
        certificateIds.add(certificateId)
      }
    }
  }

  return certificateIds
}

export const certificatesProcessor = ({
  keepLanguages,
  fallbackLanguage,
}: {
  keepLanguages: string[]
  fallbackLanguage: string
}): SdeProcessor => ({
  id: 'certificates',
  version: [...keepLanguages].sort().join(','),
  run: async ({ generatedStream, loadStream, streamJson }) => {
    const certificateIds = await collectMasteryCertificateIds(generatedStream)
    const out = await streamJson(certificatesJsonlPath)

    for await (const record of loadStream('certificates')) {
      if (!certificateIds.has(record.key as number)) {
        continue
      }

      const stripped = applyStripFields(record.value, {
        keepLanguages,
        fallbackLanguage,
        fields: ['recommendedFor'],
      })
      await writeSdeRecord(out, record, stripped)
    }

    await out.close()
  },
})

collect-generated-keys.tsts

import type { SdeProcessorContext } from '@eve-online-tools/eve-sde'

export const collectGeneratedKeys = async (
  generatedStream: SdeProcessorContext['generatedStream'],
  processorId: string,
): Promise<Set<number | string>> => {
  const keys = new Set<number | string>()

  for await (const row of generatedStream(processorId)) {
    const record = row as { _key?: number | string }

    if (record._key !== undefined) {
      keys.add(record._key)
    }
  }

  return keys
}

In masteries.ts, we first collect the keys that we want to keep. In a previous step we already generated a types table, which only contains the ships that are present in the ship tree. We then load the SDE's masteries table, filter out all the entries that we don't need, and strip all the fields and languages we don't need, before writing it as a masteries.jsonl file. (Ironically, for this file, we want to keep all the files, and there are no languages to strip.)

Taking it one step further, certificates.ts loads the results from the masteries table, and collects the certificate IDs that are used in the masteries. It then performs the same process of filtering and striping the data, before writing it as a certificates.jsonl file.

The result of this is that you can end up with much more streamlined SDE files, which makes for easier loading for users.

bash

before  after   file
567K    79K     certificates.jsonl  (-86.2%)
148K    123K    masteries.jsonl     (-17.3%)
144M    389K    types.jsonl         (-99.7%)