Auto-indent behavior

[sehugg]

Most of the C examples have two spaces per indent, the assembler examples have a mix of tabs and 8 spaces per indent.

Maybe an auto-indent detector is the way to go. Otherwise two spaces is a good default since we have limited horizontal space.

[fredsa]

I've been working on an auto-formatter. At this point, it's in pretty good shape, almost ready for a PR.

• You can manually format the code with Alt-Shift-f
• There's a tab analyzed that can determine the major tab stops in 6502 and Z80 source code
• Currently a button in the settings dialog triggers the analysis and lets you override tap stops
• The code analyzes the source and counts the most common opcode column positions; then, using just those lines, counts the most common column positions for opcodes and comments
• There's a tab stop bar above the editor showing the current tab stops
• Clicking the tab stop bar is a short cut for settings
• Settings support tab stops using a fixed tab size, or custom settings with zero or more tab stops

WIP (includes PRs #248, #249, #250 on which this one is based): https://github.com/fredsa/8bitworkshop/tree/tabstops

TODO:
Figure out how best to manage source files with different tab stops. Some 'ingredients' that could be merged into ideas to manage tab stops:

1. Maintain separate tab stop settings for each source file
2. Auto detect tab stops when an editor/file is first opened
3. Maintain (along with each "project") tab stop settings for each file
4. Maintain global user tab stop preferences that are always used
5. Maintain different tab stops by language, i.e. 6502, Z80, C
6. Support magic comments at the top of the file to specify tab settings, like vim "modelines"

These aren't all compatible with one another, and an 'all of the above' strategy with a bunch of confusing settings isn't desirable. Key considerations:

• Settings per language (6502, Z80, C) is pretty much a requirement (even if many users only stick to one)
• There's a great deal of diversity in real world assembly files which does not lend itself well to 'one format to rule them all', so we probably want the ability to support different tab stop settings within a language
  • However, real users probably only care to (re)format files that they're working on, so a single global editor setting (per language) will likely serve most users really well
  • In this world, using the built-in formatter to reformat all the (differently) formatted file in the project would be a one-time (manageable) pain (for maintainers)
• Fancier solutions can wait: don't let perfect be the enemy of the good

So, a concrete initial proposal would be:

1. Implement global per-language (6502, Z80, C) tab stop settings
2. Set reasonable defaults for each language that work well for new projects
3. When creating a new project, automatically reformat the new file based on the current settings
   • Verify (one-time) that the default files auto format reasonably well, as long as tab stop settings are reasonable
   • Reformatting happens with the user's current (language-specific) settings, which may be different from built-in defaults
4. Keep the diversity of formats across built-in samples
   • User are unlikely to try to reformat built-ins, though they are free to do so

Settings dialog:

Tab stop bar above the editor:

[sehugg]

A lot to unpack here. I see anything involving settings tweaks as for "power users" or experimental so I'm just going to address the default user experience.

So the assumption is that assembler files are treated differently, and have tab stops following the label, operation, comment pattern?

Would structured languages like C be analyzed? It seems they would just have an indent level and not individual tab stops.

Would the analysis apply to auto-indent as well as tabs?

Would this affect actual tab characters? I would rather new users not even be aware tabs exist, as the auto-indent and backspace should work mostly the same, though I haven't checked CM6.

Are per-language defaults needed if we can infer from source files?

If the file has mixed tab stops, can the user still be productive, albeit a little annoyed?

Do CM6 features help here? How do other editors handle this?

[fredsa]

A lot to unpack here. I see anything involving settings tweaks as for "power users" or experimental so I'm just going to address the default user experience.

Agreed. I expect 99% of users will only use (and know about) the defaults. And, agreed we should optimize for them. Power users <https://xkcd.com/1172/> will find the settings.

So the assumption is that assembler files are treated differently, and have tab stops following the label, operation, comment pattern?

Correct. And, tab stops only matter when you are editing the code, pressing the TAB key, or formatting the file (Alt-Shift-F). Would structured languages like C be analyzed? It seems they would just

have an indent level and not individual tab stops.

Agreed that custom tab stops don't make much sense for C file. C works well with a fixed tab size, e.g. tab size 8 gives you automatic tab stops every 8 columns. I haven't written any analysis code for C, but it should be fairly easy to scan the source and auto detect the most common indent unit so we can set the tab size automatically per file. Would the analysis apply to auto-indent as well as tabs?

- For 6502/Z80, the analysis determines visual column positions based on how the code is formatted, e.g. `zx/hello.asm` gives us "12 17 37". - For C, we'll probably end up with 2, 4, or 8, for most files as those are common character indentation levels. If you (re)format the file, any lines that need to be adjusted to match the settings are adjusted. `apple2/lz4fh.a` is a great example: most of the file is formatted to "9 15 26", but a handful of opcodes at the top, and a handful at the bottom of the program are off by one column. Would this affect actual tab characters? I would rather new users not even

be aware tabs exist, as the auto-indent and backspace should work mostly the same, though I haven't checked CM6.

For assmebly code, to be able to use custom tab stops (i.e. something other than a fixed tab size like 8), we have to insert spaces into the source to align with columns. This happens 'magically' for the user. The tab key works just fine, i.e. pressing tab will take you to the next tab stop. Because we use spaces, the source code is portable and displays consistently on different devices. For C code we can give (advanced) users the option to use actual tab characters if they want. Are per-language defaults needed if we can infer from source files?

I guess not, if there's always some non-zero number of lines of source code to infer settings from. - If 'New project' always has a template, then detecting settings from that template would work. - When adding a new include file, we could apply the settings from the main file. Are there cases in the IDE where you can include a text based file into a project that's in a different language than the main project? Like would you include a pure assembly file into a C project? And would you have enough code in the assembly file that you'd want different formatting there than in the main C file?

If the file has mixed tab stops, can the user still be productive, albeit a little annoyed?

A mix of tabs (\t) and spaces (' ') in source code code can be a bit annoying: - left/right keys behavior differently on different lines - moving the cursor vertically can make the horizontal position jump around - indentation of tabs changes when tab size is changed, but doesn't change for lines with spaces I recommend we replace the handful of tab characters that exist in some of the project's assembly files with spaces. Advanced users who import an outside project can override or use different settings that suit them.

Do CM6 features help here? How do other editors handle this?

CM6 has pretty much everything we need. There are a handful of subtle things that different editors do differently, and some of these are even language specific. Facny editor like VS Code have a bunch of settings to further tweak the behavior, as well as robust language hooks to allow extensions modify behavior and add new behavior. The only thing that I sort of expected CM6 to implement which it does not it custom tab stop with \t characters. Specifically, if I want tab stops at columns 12, 17, and 37, then I'd want one tab on a blank line to take me to 12, two to 17, and 3 tabs to 37. CM6 doesn't appear to support this. You can definitely implement custom tab stops as I have in this PR, but the editor still has a native notion of tab size (which is actually exposed back out to the browser layer) and any \t characters are rendering using this tab size. Not a problem, but interesting. Thanks for the reply. I think your idea of just always detecting file tab stops is a good one. I'll go will that.

[fredsa]

Updated https://github.com/fredsa/8bitworkshop/tree/tabstops:

• When opening a project, automatically analyze and detect tab stops / tab size from main file
• Add indentation detection for C, tested on a dozen or so files in presets, works well
• Additional polish to settings pane & top tab bar
• Add 6809 opcodes (to support indentation detection)

[sehugg]

I was surprised that CM6 didn't auto-detect indentation levels at least for C-like files. Glad you got that sorted. What happens if you paste and replace with another file -- should the analysis be re-run when you do bulk operations like paste or revert? Will weird stuff happen when you clear the text and start from scratch?

Is there a distinction between physical and logical tab stops? For the default user, I think changing the physical tab stops from 8 would be confusing, esp. when importing/exporting files. So if you analyze a C file with indents of 4 spaces, would that change the physical tab stops, the logical tab stops, or the indent level? (and are those separate things?)

It seems like for C-like languages, Tab means "indent" (usually with spaces) but for assembly files, tab really means "insert a tab", unless you have custom tab settings.

I think the tradeoff of custom tab stops and auto-formatting is you can't have blocks with different column spacing in the same assembly file. Users that want to do that can opt not to use those features.

I guess I'll have to marinate on this which will take a while.

[fredsa]

I was surprised that CM6 didn't auto-detect indentation levels at least for C-like files. Glad you got that sorted.
What happens if you paste and replace with another file -- should the analysis be re-run when you do bulk operations like paste or revert?

Leaning on https://en.wikipedia.org/wiki/Principle_of_least_astonishment, we could probably make the argument either way:

• for changing the tab settings when the file contents are changed, because of course it's like opening the editor for the first time
• for not changing the tab settings once they've been set, because that would be confusing, and maybe the they're set exactly as the user wants them

I think the right answer is something like:

• this is not a common use case
• it's probably not worth adding app complexity and corner cases to test
• (advanced) users always have the option of opening settings (by clicking the tab bar) and manually clicking 'analyze'
• not doing anything special is the least amount of work, and leaves the user in control
• uncommon workflows are exactly where all the corner cases like to live, so, yeah, leave the user in control
• :)

Will weird stuff happen when you clear the text and start from scratch?

With the current logic (auto detect settings when loading a new project / mail file), there's nothing special about clearing the file, adding/removing an include file, or swapping out file contents. The settings just stay what they are until the user changes them, or until they reload the app. Nothing breaks. Nothing strange happens.

Is there a distinction between physical and logical tab stops?

More below, but we have:

• tabSize (CodeMirror and browser built-ins) could be called "physical" tab stops: they determine how much space a \t takes up
• tab stops (implemented in this pending PR) could be called "logical" tab stops: they're implemented in the IDE code and have no direct 1:1 correspondence to CodeMirror or browser capability, though the concept is common in word processing (e.g. Google Docs), or physically (https://en.wikipedia.org/wiki/Typewriter#Tab_key)

For the default user, I think changing the physical tab stops from 8 would be confusing, esp. when importing/exporting files.

Agreed, 8 is very universal. And, 8 is almost certainly the right default (for ~ everyone). Advanced users can change this setting if they want. (That setting will stay persistent in local storage. And, having changed the setting manually, they'll understand the behavior when viewing any files with tabs.)

As a data point, looking at the ~80 main files in presets/, I found 2 files (presets/vcs/vcs.h, presets/nes/famitone2.dasm) that appear to have been written with 4-space tabs. Excluding these two files, tabs can be replaced with (the appropriate number of) spaces (to indent to multiples of 8 chars tab stops) in all the other files.

So if you analyze a C file with indents of 4 spaces, would that change the physical tab stops, the logical tab stops, or the indent level?

We would in this case:

• set the "indentation unit" to be 4 spaces, so that (re)format all, indent/unindent, etc. all work as expected
• leave tabSize = 8, in case there's a stray \t character somewhere in the file
• set the editor to to use spaces for (any new) indentation

The user can still paste in new code that contains tab characters. They would render with tabSize=8. If they reformat the file or as they indent and unindent lines, tabs will be converted to spaces. The editor will know \t means "go to the next column that's a multiple of 8", and will use the appropriate number of spaces to make code align to multiples of 4 (the indent unit). Everything "just works".

(and are those separate things?)

CodeMirror provides a few basic ingredients for us to work with:

• tabSize, at it's most basic level, is literally used to set the tab-size CSS property, which controls how a tab characters (\t) in the source code are rendered to the screen, i.e. how much horizontal space does they take up
• insertTab command, equivalent to inserting a \t character, boring
• indentUnit, which can be a \t, or a space, or 8 spaces, or 5 spaces, etc. is what the editor will use to (re)indent a line, used by indentMore, indentLess, indentSelection, indentRange

The in flight PR adds the notion of "tab stops" (different and separate from tabSize), to make working with 6502/Z80 more pleasant and convenient:

• Ability to reformat files, so that opcode, operand and comments all line up vertically under whatever tab stops we have set
• We can customize the behavior when Tab and Shift-Tab is pressed:
  • We can make Tab move the cursor to the next Tab stop, so on a new line, 1x Tab takes you to the opcode column, another Tab takes you to the operand column, and a third takes you to the comment column
  • We can make Shift-Tab unindent the line you're on
  • We can make it so when you have selecting something (partial line, an entire line, multiple lines, whatever), Tab and Shift-Tab always indentMore/indentLess

I put together a CodeMirror toy example to play with:

The code below should appear on the left. A working editor will appear on the right. If you make any edits to the source on the left, click the "run" button to recompile. Open Chrome Dev Tools to see any compile errors.

import { EditorState } from "@codemirror/state";
import { basicSetup } from "codemirror"
import { EditorView, keymap, highlightSpecialChars, highlightTrailingWhitespace, highlightWhitespace } from "@codemirror/view"
import { cpp } from "@codemirror/lang-cpp"
import { indentRange, indentUnit } from "@codemirror/language";
import { indentLess, indentMore, insertTab } from "@codemirror/commands";

const reformatAll = (view) => {
  view.dispatch({
    changes: indentRange(view.state, 0, view.state.doc.length)
  });
};

const view = new EditorView({
  parent: document.body,
  // Poorly formatted C program as an example.
  doc: "/*\n0123456789\n*/\n   func main() {\n\t\t\treturn 42;\n }\n",
  extensions: [
    // Basic, fairly usuable editor.
    basicSetup,

    // C, C++ language support, supports:
    //   parsing, syntax highlighting, indentation logic
    cpp(),
    
    // Uncomment to visualize spaces and tabs.
    // highlightWhitespace(),

    // Set to exaggerated value for demonstration purposes.
    EditorState.tabSize.of(12),

    // Intentionally indent with a different size, to
    // illustrate effect of tabSize vs. indentUnit
    indentUnit.of(" ".repeat(5)),

    // To have the editor use tabs instead of spaces for
    // (re)indentation/formatting, comment out above line
    // and replace with:
    // indentUnit.of("\t"),

    // Keyboard shortcuts
    // Tab / shift-tab aren't mapped for accessibility
    keymap.of([
      // `insertTab` acttually:
      // - inserts a tab character at the cursor -OR-
      // - if something is selected, calls `indentMore`
      { key: "Tab", run: insertTab },

      // Deindent the line, if possible:
      { key: "Shift-Tab", run: indentLess },
      
      // Reformat file using current indentUnit,
      // use CMD-SHIFT-F on macOS:
      { key: "Shift-Mod-f", run: reformatAll },
    ]),

    // Visualize otherwise invisible characters
    highlightSpecialChars(),

    // Red highlight trailing whitespace
    highlightTrailingWhitespace(),
  ],
});

It seems like for C-like languages, Tab means "indent" (usually with spaces) but for assembly files, tab really means "insert a tab", unless you have custom tab settings.

For advanced users, there's a great deal of variety in what developers "expect" to happen. Customizability options are almost infinite, but I think about at least these options (for both C and asm):

1. If there's a tab character in my source, what tab size should be used to render that line of source?
2. Do I want the editor to insert tabs (\t) or spaces when indenting lines?
3. When I press TAB with nothing selected, I want the editor to:
   • move my cursor to the next tab stop using the current indentation setting (tabs or spaces)
   • always insert a \t character
   • indent the current line or selected range (regardless of where my cursor is)
4. When I press TAB with something selected, I want the editor to:
   • indent the line (or selected range)
   • replace the selection with a \t

As for differences:

• C: use a fixed tab size, i.e. automatic tab stops at multiples of the tab size
  • if tab size = 8, then tab stops are 8, 16, 24, 32, …
• asm: two different models "work":
  • just use the C model of fixed tab sizes; it "works", though tab size of 8 is usually not that convenient, meaning columns 8. 16, 24, 32, 40, … don't layout that nicely
  • use custom tab stops, adjust to the style and preference of the author; each author has their preferences, and different programs might have different needs

I think the tradeoff of custom tab stops and auto-formatting is you can't have blocks with different column spacing in the same assembly file. Users that want to do that can opt not to use those features.

We can always look at fancier tab stops in the future. We could one day support magic section changes through /*{tabStops: 12,17,37}*/ style comments.

I guess I'll have to marinate on this which will take a while.
I'll keep polishing for a bit. I'll have you play with the final version and we can take it from there. I am feeling good about being able to ship an experience that offers solid and unsurprising defaults, but also offers some creature comforts and little magic.

[fredsa]

Here's a fresh PR (#251) informed by this thread, and by some additional experimentation:

• I've kept the new functionality as straightforward as possible
• I did my best to build up the PR out of smaller commits to make review easier
• 6502/Z80 parsing and syntax highlighting improvements #248, Improve asm/C indentation #249, Update settings #250 have been closed as they've been incorporated into the new PR