merge feature/cursor

Merge in changes which use paginated requests when writing tokens instead of pulling all in to memory. Also includes:

• Updates to how we handle filtering when tokenizing, another spot where tokens could be pulled into memory
• Testing against the Tokenizer and some of the new functionality