Introduction

Avalanche.Tokenizer contains string tokenizing utilities, [git], [www].

Add package reference to .csproj.

<ItemGroup>
    <PackageReference Include="Avalanche.Tokenizer"/>
</ItemGroup>

This class library tokenizes strings.

// Create tokenizer
var valueTokenizer = IntegerTokenizer.Instance;
var commaTokenizer = new ConstantTokenizer<SeparatorToken>(",");
var malformedTokenizer = new UntilTokenizer<MalformedToken>(new AnyTokenizer(new ConstantTokenizer(","), WhitespaceTokenizer.Any));
var slotTokenizer = new SequenceTokenizer<ValueToken>(
    (WhitespaceTokenizer.Any, false, false),
    (commaTokenizer, false, false),
    (WhitespaceTokenizer.Any, false, false),
    (new AnyTokenizer(valueTokenizer, malformedTokenizer), true, true),
    (WhitespaceTokenizer.Any, false, false)
);
var tokenizer = new WhileTokenizer(slotTokenizer, false);
// Tokenize string
IToken token = tokenizer.TakeAll<IToken>("1, x, 3".AsMemory());
// Print token
System.Console.WriteLine(token.PrintTree(format: TokenPrintTreeExtensions.PrintFormat.DefaultLong));

[0:7] CompositeToken:  "1, x, 3"
├── [0:1] ValueToken:  "1"
│   └── [0:1] DecimalToken:  "1"
├── [1:4] ValueToken:  ", x"
│   ├── [1:2] SeparatorToken:  ","
│   ├── [2:3] WhitespaceToken:  " "
│   └── [3:4] MalformedToken:  "x"
└── [4:7] ValueToken:  ", 3"
    ├── [4:5] SeparatorToken:  ","
    ├── [5:6] WhitespaceToken:  " "
    └── [6:7] DecimalToken:  "3"

The following GlobalUsings.cs can be used to include all extension methods.

global using Avalanche.Tokenizer;
global using Avalanche.Utilities;

Class libraries:

• Avalanche.Tokenizer.dll contains implementations.
• Avalanche.Tokenizer.Abstractions.dll contains interfaces.

Dependency libraries:

• Avalanche.Utilities.dll
• Avalanche.Utilities.Abstractions.dll contains interfaces.

Full Example

Full example

using Avalanche.Tokenizer;

class index
{
    public static void Run()
    {
        {
            // <01>
            // Create tokenizer
            var valueTokenizer = IntegerTokenizer.Instance;
            var commaTokenizer = new ConstantTokenizer<SeparatorToken>(",");
            var malformedTokenizer = new UntilTokenizer<MalformedToken>(new AnyTokenizer(new ConstantTokenizer(","), WhitespaceTokenizer.Any));
            var slotTokenizer = new SequenceTokenizer<ValueToken>(
                (WhitespaceTokenizer.Any, false, false),
                (commaTokenizer, false, false),
                (WhitespaceTokenizer.Any, false, false),
                (new AnyTokenizer(valueTokenizer, malformedTokenizer), true, true),
                (WhitespaceTokenizer.Any, false, false)
            );
            var tokenizer = new WhileTokenizer(slotTokenizer, false);
            // Tokenize string
            IToken token = tokenizer.TakeAll<IToken>("1, x, 3".AsMemory());
            // Print token
            System.Console.WriteLine(token.PrintTree(format: TokenPrintTreeExtensions.PrintFormat.DefaultLong));
            // </01>
        }
    }
}