chore: Generate our own node ids instead of reusing tree-sitter's #393

With this change in place, maybe we could expand this test a little?
5d4b5787c6/src/visualizer.rs (L200)

While working on the test, I noticed that the description of the id field is now outdated. Could you please update it?

So I made a test, not sure that we'll use it exactly as it is now. But while working on it, I made this:

#[test]
fn node_ids_multiline_escape() {
    let ctx = ctx();

    let src = r#"let foo = "line 1\
    line 2
    line 3";"#;
    let tree = ctx.parse_rust(src);

    let root = tree.root();
    assert_eq!(root.id, 14);
    
    let let_declaration = root[0];
    assert_eq!(let_declaration.id, 13);
    
    let [let_kw, foo, eq, str_literal, semicolon] = let_declaration[..] else {
        unreachable!()
    };
    assert_eq!(let_kw.id, 1);
    assert_eq!(foo.id, 2);
    assert_eq!(eq.id, 3);
    assert_eq!(str_literal.id, 11);
    assert_eq!(semicolon.id, 12);

    let [quote_open, line1, newline1, lines2and3, quote_close] = str_literal[..] else {
        unreachable!()
    };
    assert_eq!(quote_open.id, 4);
    assert_eq!(line1.source, "line 1");
    assert_eq!(line1.id, 5);
    assert_eq!(newline1.source, "\\");
    assert_eq!(newline1.id, 6);
    assert_eq!(lines2and3.source, "line 2\nline 3");
    assert_eq!(lines2and3.id, 9);
    assert_eq!(quote_close.id, 10);

    let [line2, line3] = lines2and3[..] else {
        unreachable!()
    };
    assert_eq!(line2.source, "line 2");
    assert_eq!(line2.id, 7);
    assert_eq!(line3.source, "line 3");
    assert_eq!(line3.id, 8);
}

One can see that the resulting tree structure ended up being a bit weird, but the ASCII tree kind of shows why that is:

└let_declaration
  ├let
  ├pattern: identifier foo
  ├=
  ├value: string_literal
  │ ├"
  │ ├string_content line 1
  │ ├escape_sequence \
  │ ├string_content
  │ └"
  └;

so one level of nesting is created because the grammar splits the string on the escape sentence, and another we create ourselves, when splitting the second string_content, which corresponds to line 2\nline3, into multiple lines. This can be something to keep in mind when working on improved handling of multiline string literals, since even though we don't split line 1 into a separate line ourselves, it's still ends up effectively split away by the grammar.

That aside, here's the more normal-looking version of the test:

#[test]
fn node_ids_multiline() {
    let ctx = ctx();
    
    let src = r#"let foo = "line 1
line 2
line 3";"#;
    let tree = ctx.parse_rust(src);

    let root = tree.root();
    assert_eq!(root.id, 13);
    
    let let_declaration = root[0];
    assert_eq!(let_declaration.id, 12);

    let [let_kw, foo, eq, str_literal, semicolon] = let_declaration[..] else {
        unreachable!()
    };
    assert_eq!(let_kw.id, 1);
    assert_eq!(foo.id, 2);
    assert_eq!(eq.id, 3);
    assert_eq!(str_literal.id, 10);
    assert_eq!(semicolon.id, 11);

    let [quote_open, lines, quote_close] = str_literal[..] else {
        unreachable!()
    };
    assert_eq!(quote_open.id, 4);
    assert_eq!(lines.id, 8);
    assert_eq!(quote_close.id, 9);

    let [line1, line2, line3] = lines[..] else {
        unreachable!()
    };
    assert_eq!(line1.source, "line 1");
    assert_eq!(line1.id, 5);
    assert_eq!(line2.source, "line 2");
    assert_eq!(line2.id, 6);
    assert_eq!(line3.source, "line 3");
    assert_eq!(line3.id, 7);
}

As can be seen, removing the \ made the whole tree a bit more understandable.

The reason I wasn't sure we want to keep the test like is that the particular mapping of ids to nodes doesn't feel like something we want to guarantee. Since we're only really interested in the ids being unique, I think this much simpler test would be enough:

#[test]
fn node_ids_all_unique() {
    let ctx = ctx();

    let src = r#"let foo = "line 1
line 2
line 3";"#;
    let tree = ctx.parse_rust(src);

    let ids = tree.dfs().map(|n| n.id).collect_vec();
    let len = ids.len();
    assert!(ids.iter().all_unique());
    assert_eq!(*ids.iter().max().unwrap(), len);
}

the max thing would be just to ensure that we actually use all 13 (=len) available ids, not skipping anything -- but even that requirement is not strictly necessary

Great! My preference goes to the second test, because I agree what matters here is the uniqueness of the ids, not their particular values. I wouldn't mind having the other one too, as long as we feel empowered to change or remove it later on if needed.
Are you able to commit them on my branch already or should I do it?

Well not commit directly, but I'll open a PR, one sec