Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(docs): Adds graph debug documentation to book #379

Merged
merged 4 commits into from
Jun 10, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions book/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
+ [Using `Extras`](./extras.md)
+ [Using callbacks](./callbacks.md)
+ [Common regular expressions](./common-regex.md)
+ [Debugging](./debugging.md)
+ [Examples](./examples.md)
+ [Brainfuck interpreter](./examples/brainfuck.md)
+ [JSON parser](./examples/json.md)
Expand Down
85 changes: 85 additions & 0 deletions book/src/debugging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Debugging

Gain deeper insights into your code's behavior with this debugging section.
afreeland marked this conversation as resolved.
Show resolved Hide resolved

## Visualizing Logos Graph

Logos works by creating a graph that gets derived from the tokens that you defined. This graph describes how the lexer moves through different states when processing input.

Hence, it may be beneficial during debugging to be able to visualize this graph, to understand how Logos will match the various tokens.

If we take this example:
```rust,no_run,noplayground
use logos::Logos;

#[derive(Debug, Logos, PartialEq)]
enum Token {
// Tokens can be literal strings, of any length.
#[token("fast")]
Fast,

#[token(".")]
Period,

// Or regular expressions.
#[regex("[a-zA-Z]+")]
Text,
}
fn main() {
let input = "Create ridiculously fast Lexers.";

let mut lexer = Token::lexer(input);
while let Some(token) = lexer.next() {
println!("{:?}", token);
}
}
```

Logos actually constructs a graph that contains the logic for matching tokens:
```
graph = {
1: ::Fast,
2: ::Period,
3: ::Text,
4: {
[A-Z] ⇒ 4,
[a-z] ⇒ 4,
_ ⇒ 3,
},
7: [
ast ⇒ 8,
_ ⇒ 4*,
],
8: {
[A-Z] ⇒ 4,
[a-z] ⇒ 4,
_ ⇒ 1,
},
9: {
. ⇒ 2,
[A-Z] ⇒ 4,
[a-e] ⇒ 4,
f ⇒ 7,
[g-z] ⇒ 4,
},
}
```
This graph can help us understand how our patterns are matched, and maybe understand why we have a bug at some point.

Let's get started by trying to understand how Logos is matching the `.` character, which we've tokenized as `Token::Period`.

We can begin our search by looking at number `9` for the character `.`. We can see that if Logos matches a `.` it will jump `=>` to number `2`. We can then follow that by looking at `2` which resolves to our `::Period` token.

Logos will then continue to look for any matches past our `.` character. This is required in case there is potential continuation after the `.` character. Although, in the _input_ we provided there are no any additional characters, since it is the end of our input.

We also can try to identify how the token `fast` works by looking at `9`, first, and seeing that `f` will cause Logos to jump to `7`. This will then resolve the last letters of our word _fast_ by matching `ast` which jumps to `8`. Since our provided _input_ to the lexer does not include alphabetic characters after the word "fast", but rather a whitespace, the token `::Fast` will be recognized. Then, the graph will look for further potential continuation (here, `[g-z] => 4`)

### Enabling
jeertmans marked this conversation as resolved.
Show resolved Hide resolved

To enable debugging output you can define a `debug` feature in your `Cargo.toml` file, like this:

```
// Cargo.toml
[dependencies]
logos = { version = "1.2.3", features = ["debug"] }
```
3 changes: 3 additions & 0 deletions logos-codegen/src/generator/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ impl<'a> Generator<'a> {
let rendered = Self::fast_loop_macro();
let meta = Meta::analyze(root, graph);

#[cfg(feature = "debug")]
dbg!(graph);

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you don't need this, because I already added a debug print of the graph in another place

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See here:

debug!("Generating code from graph: {graph:#?}");

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @afreeland! Could you address this review please? So I can accept your PR and merge it :-)

Generator {
name,
this,
Expand Down
Loading