Skip to content

Importing DryScores

This page goes over hydrating a DryScore into a fully fledged Tachi score, and then goes over how it is imported.

Warning

Performance has been squeezed out of this process rather aggressively, as such, some of the more 'obvious' ways to do things have been ignored for performance gains.

Info

The code that handles this logic is found in src/lib/score-import/framework/score-importing/score-importing.ts.

Return Format

The ImportIterableDatapoint returns ImportProcessingInfo or null. The format for the former can be found in tachi-common, and should be read before following this document!

As a rough outline: 

{
    // whether this import worked or not
    success: boolean; 
    // ScoreImported implies success: true, and all others imply success: false.
    type: "ScoreImported" | "SongOrChartNotFound" // ... and more;
    // An error message.
    message: String | null;
    // Some errors return some data about the error, or things like scoreImported returns the score that was imported.
    content: see_implementation
}

Dealing with converter returns

As mentioned in Parsers and Converters, converters will return a DryScore, and its matching chart and song on success.

However, failures are also an expected throw from a converter function. We handle these throws by logging them dependent on their severity, and returning that this score has failed to be imported properly in the ImportProcessingInfo format.

If the converter was successful, we now have a DryScore, song and chart to work with.

Hydration

Our first step is to turn that DryScore into a 'real score'.

Info

The code for this is found in HydrateScore.

Before anything, we calculated the ScoreID for this score. This is used to dedupe scores, and is a checksum of the following properties.

  • userID
  • chartID
  • lamp
  • grade
  • score
  • percent

We fill out all the properties that can be calculated about the score, such as the aptly named calculatedData, and things like gradeIndex and lampIndex, which are just the index of the grade/lamp string in the set of grade/lamps.

Other properties, such as timeAdded (the time this data was inserted into the database) can be trivially attached onto the score.

Insertion

Now that we have a real Tachi score, we can insert it into the database.

The obvious solution here is something like: 

await db.scores.insert(scoreDocument);

But this has performance implications for large sets of scores.

To solve these performance issues, we use a queue to insert scores.

ScoreDocuments are appended to a queue of 500. If the queue hits 501 scores, the queue is bulk-written to the database. At the end of hydrating and queueing all scores, the queue is flushed one last time.

This simple optimisation provides large performance benefits.

Final Returns

Importing each datapoint gives us an ImportProcessingInfo object OR null. We want to return only the former, as null is for skipped scores.

We filter out all the nulls and flush the queue before returning the array of ImportProcessingInfo objects that were returned from our import process.