Things I learned building a text editor for the terminal¶
TextArea is the latest widget to be added to Textual's growing collection.
It provides a multi-line space to edit text, and features optional syntax highlighting for a selection of languages.
TextArea to your Textual app is as simple as adding this to your
Enabling syntax highlighting for a language is as simple as:
Working on the
TextArea widget for Textual taught me a lot about Python and my general
approach to software engineering. It gave me an appreciation for the subtle functionality behind
the editors we use on a daily basis — features we may not even notice, despite
some engineer spending hours perfecting it to provide a small boost to our development experience.
This post is a tour of some of these learnings.
Vertical cursor movement is more than just
When you move the cursor vertically, you can't simply keep the same column index and clamp it within the line. Editors should maintain the visual column offset where possible, meaning they must account for double-width emoji (sigh 😔) and East-Asian characters.
Notice that although the cursor is on column 11 while on line 1, it lands on column 6 when it arrives at line 3. This is because the 6th character of line 3 visually aligns with the 11th character of line 1.
Edits from other sources may move my cursor¶
There are two ways to interact with the
- You can type into it.
- You can make API calls to edit the content in it.
In the example below,
Hello, world!\n is repeatedly inserted at the start of the document via the
Notice that this updates the location of my cursor, ensuring that I don't lose my place.
This subtle feature should aid those implementing collaborative and multi-cursor editing.
This turned out to be one of the more complex features of the whole project, and went through several iterations before I was happy with the result.
Thankfully it resulted in some wonderful Tetris-esque whiteboards along the way!
Sometimes stepping away from the screen and scribbling on a whiteboard with your colleagues (thanks Dave!) is what's needed to finally crack a tough problem.
Many thanks to David Brochart for sending me down this rabbit hole!
Spending a few minutes running a profiler can be really beneficial¶
While building the
TextArea widget I avoided heavy optimisation work that may have affected
readability or maintainability.
However, I did run a profiler in an attempt to detect flawed assumptions or mistakes which were affecting the performance of my code.
I spent around 30 minutes profiling
using pyinstrument, and the result was a
~97% reduction in the time taken to handle a key press.
What an amazing return on investment for such a minimal time commitment!
pyinstrument unveiled two issues that were massively impacting performance.
1. Reparsing highlighting queries on each key press¶
I was constructing a tree-sitter
Query object on each key press, incorrectly assuming it was a
This query was completely static, so I moved it into the constructor ensuring the object was created
This reduced key processing time by around 94% - a substantial and very much noticeable improvement.
This seems obvious in hindsight, but the code in question was written earlier in the project and had been relegated in my mind to "code that works correctly and will receive less attention from here on out". pyinstrument quickly brought this code back to my attention and highlighted it as a glaring performance bug.
2. NamedTuples are slower than I expected¶
NamedTuples are slow to create relative to
tuples, and this cost was adding up inside
an extremely hot loop which was instantiating a large number of them.
pyinstrument revealed that a large portion of the time during syntax highlighting was spent inside
Here's a quick benchmark which constructs 10,000
Here's the same benchmark using
tuple resulted in another noticeable increase in responsiveness.
Key-press handling time dropped by almost 50%!
Unfortunately, this change does impact readability.
However, the scope in which these tuples were used was very small, and so I felt it was a worthy trade-off.
Syntax highlighting is very different from what I expected¶
In order to support syntax highlighting, we make use of the tree-sitter library, which maintains a syntax tree representing the structure of our document.
To perform highlighting, we follow these steps:
- The user edits the document.
- We inform tree-sitter of the location of this edit.
- tree-sitter intelligently parses only the subset of the document impacted by the change, updating the tree.
- We run a query against the tree to retrieve ranges of text we wish to highlight.
- These ranges are mapped to styles (defined by the chosen "theme").
- These styles to the appropriate text ranges when rendering the widget.
Another benefit that I didn't consider before working on this project is that tree-sitter parsers can also be used to highlight syntax errors in a document. This can be useful in some situations - for example, highlighting mismatched HTML closing tags:
Before building this widget, I was oblivious as to how we might approach syntax highlighting. Without tree-sitter's incremental parsing approach, I'm not sure reasonable performance would have been feasible.
Edits are replacements¶
All single-cursor edits can be distilled into a single behaviour:
This replaces a range of characters with some text.
We can use this one method to easily implement deletion, insertion, and replacement of text.
- Inserting text is replacing a zero-width range with the text to insert.
- Pressing backspace (delete left) is just replacing the character behind the cursor with an empty string.
- Selecting text and pressing delete is just replacing the selected text with an empty string.
- Selecting text and pasting is replacing the selected text with some other text.
This greatly simplified my initial approach, which involved unique implementations for inserting and deleting.
The line between "text area" and "VSCode in the terminal"¶
A project like this has no clear finish line. There are always new features, optimisations, and refactors waiting to be made.
So where do we draw the line?
We want to provide a widget which can act as both a basic multiline text area that anyone can drop into their app, yet powerful and extensible enough to act as the foundation for a Textual-powered text editor.
Yet, the more features we add, the more opinionated the widget becomes, and the less that users will feel like they can build it into their own thing. Finding the sweet spot between feature-rich and flexible is no easy task.
I don't think the answer is clear, and I don't believe it's possible to please everyone.
Regardless, I'm happy with where we've landed, and I'm really excited to see what people build using
TextArea in the future!