Skip to content

merge feature/cursor

Ghost User requested to merge feature/cursor into develop

Merge in changes which use paginated requests when writing tokens instead of pulling all in to memory. Also includes:

  • Updates to how we handle filtering when tokenizing, another spot where tokens could be pulled into memory
  • Testing against the Tokenizer and some of the new functionality

Merge request reports