In Part 1 I addressed:
syntax tables, indentation, font locking, and context-sensitive syntax.
We now build out a "Hyde" with: shell/async process integration, Eldoc support,
and Autocompletion. Also "shift-K" documentation lookup for the symbol-at-point.
Building Inferior Hy
The basics
The easiest way to add a REPL to your major-mode is through setting the variable
inferior-lisp-program, possibly adding setup code through
inferior-lisp-load-command, then running the inferior-lisp command.
;; within the define-derived-mode hy-mode setup...
(setq-local inferior-lisp-program "hy")
(setq-local inferior-lisp-load-command "(print \"Hy there!\")")
This isn't sufficient for advanced shells. The custom is to create a
function named run-hy (eg. there is run-python, run-haskell...)
which starts up the shell, sets inferior-hy-mode , and switches to the
shell.
Comint-mode
It is a "Major mode for interacting with an inferior interpreter". Most-all
inferior modes will derive from comint-mode. It provides many utilities for
interacting with shell-like processes.
Hy mode's prompt is a right arrow.
(define-derived-mode inferior-hy-mode comint-mode "Inferior Hy"
"Major mode for Hy inferior process."
(setq-local indent-tabs-mode nil)
;; How to dispaly the process status in the mode-line
(setq mode-line-process '(":%s"))
;; This disables editing and traversing the "=>" prompts
(setq-local comint-prompt-read-only t)
;; Lets comint mode recognize the prompt
(setq-local comint-prompt-regexp (rx bol "=>" space))
;; ... other specialized config introduced later ...
)
There are several comint components we will make use of:
comint-last-prompt - a cons cell of begin/end markers of last prompt.comint-send-string - perform process-send-string with comintcomint-redirect-send-command-to-process and comint-redirect-completed - for sending strings asynchronously.comint-(pre)output-filter-functions - entry points into capturing andmake-comint-in-buffer - entry point into comint mode.
Managing buffers and processes
We must manage: the standard hy shell process, the internal hy process used
for autocompletion and eldoc, and temporal buffers for more advanced buffer
transformations of the standard hy shell process.
The configuration required of inferior-hy-mode :
;;; Configuration
(defconst hy-shell-interpreter "hy"
"Default Hy interpreter name.")
(defvar hy-shell-interpreter-args "--spy"
"Default arguments for Hy interpreter.")
;;; Internal
(defconst hy-shell-buffer-name "Hy"
"Default buffer name for Hy interpreter.")
(defconst hy-shell-internal-buffer-name "Hy Internal"
"Default buffer name for the internal Hy process.")
(defvar hy-shell-buffer nil
"The current shell buffer for Hy.")
(defvar hy--shell-output-filter-in-progress nil
"Whether we are waiting for output in `hy-shell-send-string-no-output'.")
(defvar hy--shell-font-lock-enable t
"Whether the shell should font-lock the current line.")
Now the building blocks of the shell can be defined.
The implementations are rather straightforward. To keep space down, only
name and docstring are provided:
(defun hy--shell-format-process-name (proc-name)
"Format a PROC-NAME with closing astericks.")
(defun hy-shell-get-process (&optional internal)
"Get process corr. to `hy-shell-buffer-name'/`hy-shell-internal-buffer-name'.")
(defun hy--shell-current-buffer-process ()
"Get process associated with current buffer.")
(defun hy--shell-current-buffer-a-process? ()
"Is `current-buffer' a live process?")
(defun hy--shell-get-or-create-buffer ()
"Get or create `hy-shell-buffer' buffer for current hy shell process.")
(defun hy--shell-buffer? ()
"Is `hy-shell-buffer' set and does it exist?")
(defun hy--shell-kill-buffer ()
"Kill `hy-shell-buffer'.")
(defun hy--shell-calculate-command (&optional internal)
"Calculate the string used to execute the inferior Hy process.")
;; Straightforward string formatting - see: `shell-quote-argument'
Starting up the shell
The commands above are enough to build out some basic shell support. Lets
look at run-hy :
(defun run-hy (&optional cmd)
"Run an inferior Hy process.
CMD defaults to the result of `hy--shell-calculate-command'."
(interactive)
(unless (executable-find "hy")
(message "Hy not found, activate a virtual environment with Hy."))
(-> (or cmd (hy--shell-calculate-command))
(hy--shell-make-comint hy-shell-buffer-name 'show)
get-buffer-process))
Most of the work is delegated to hy--shell-make-comint as we also must
have the internal variant:
(defun run-hy-internal ()
"Start an inferior hy process in the background for autocompletion."
(interactive)
(unless (executable-find "hy")
(message "Hy not found, activate a virtual environment containing Hy to use
Eldoc, Anaconda, and other hy-mode features."))
(when (and (not (hy-shell-get-process 'internal))
(executable-find "hy"))
(-let [hy--shell-font-lock-enable
nil]
(prog1
(-> (hy--shell-calculate-command 'internal)
(hy--shell-make-comint hy-shell-internal-buffer-name nil 'internal)
get-buffer-process)
(hy--shell-send-internal-setup-code)
(message "Hy internal process successfully started")))))
This is a simple variation of run-hy that passes the internal argument
through the hy shell building blocks and also sends setup code for
eldoc-mode and company-mode.
Now we are ready to startup our inferior-hy-mode :
(defun hy--shell-make-comint (cmd proc-name &optional show internal)
"Create and return comint process PROC-NAME with CMD, opt. INTERNAL and SHOW."
(-when-let* ((proc-buffer-name
(hy--shell-format-process-name proc-name))
(_
(not (comint-check-proc proc-buffer-name)))
(cmdlist
(split-string-and-unquote cmd))
(buffer
(apply 'make-comint-in-buffer proc-name proc-buffer-name
(car cmdlist) nil (cdr cmdlist)))
(process
(get-buffer-process buffer)))
(with-current-buffer buffer
(inferior-hy-mode))
(when show
(display-buffer buffer))
(if internal
(set-process-query-on-exit-flag process nil)
(setq hy-shell-buffer buffer))
proc-buffer-name))
All the work is once again delegated to our shell building blocks. There are
several things to notice:
- If the process is meant to be autostarted/quited, make sure to use
set-process-query-on-exit-flag to nil.
- The
cmdlist car is "hy" and cdr is the hy interpreter arguments.
- Further accessing of the shell is done with the
hy-shell-buffer variable.
Working with the shell
The shell is now functional, but we still don't have methods to send strings
to the shell (for instance, sending the current-form or the buffer for
evaluation). Nor do we have any support for asynchronously sending and
extracting information from our internal process.
Like always, lets define some utilities:
(defun hy--shell-end-of-output? (string)
"Return non-nil if STRING ends with the prompt."
(s-matches? comint-prompt-regexp string))
(defun hy--shell-output-filter (string)
"If STRING ends with input prompt then set filter in progress done."
(when (hy--shell-end-of-output? string)
(setq hy--shell-output-filter-in-progress nil))
"\n=> ")
hy--shell-output-filter-in-progress is the critical component. Lets see how
it is used:
(defun hy--shell-send-string (string &optional process internal)
"Internal implementation of shell send string functionality."
(let ((process (or process (hy-shell-get-process internal)))
(hy--shell-output-filter-in-progress t))
(comint-send-string process string)
(while hy--shell-output-filter-in-progress
(accept-process-output process))))
The shell process is obtained, we set it to be in progress, and send it off
to comint. But how and when is the filter reset?
We come back to the comint-(pre)output-filter-functions. When we send the
string via comint-send-string, part of its bookkeeping is to apply these
filter functions to the output. However, the output can come in chunks, so
simply accepting the process output is not sufficient. We must recognize
when the last of the expected process output is retrieved and signal to stop
accepting output.
The difference between the pre and standard filters is when they are
applied. The pre variation is executed before the process output is inserted
into the buffer.
Looking back to hy--shell-output-filter, what are the outcomes of using it
as a pre or standard filter? Lets look at the exposed send strings:
(defun hy-shell-send-string-no-output (string &optional process internal)
"Send STRING to hy PROCESS and inhibit printing output."
(-let [comint-preoutput-filter-functions
'(hy--shell-output-filter)]
(hy--shell-send-string string process internal)))
(defun hy-shell-send-string (string &optional process)
"Send STRING to hy PROCESS."
(-let [comint-output-filter-functions
'(hy--shell-output-filter)]
(hy--shell-send-string string process)))
Lastly, these functions won't work for asynchronous ops like Eldoc and
Autocompletion. You will see a Blocking call inhibiting process output error messaged in the minibuffer.
The asynchronous version is different. We redirect the process output to a
temporary buffer and capture its output.
The key is the 100ms timeout argument passed to accept-process-output.
(defun hy--shell-send-async (string)
"Send STRING to internal hy process asynchronously."
(let ((output-buffer " *Comint Hy Redirect Work Buffer*")
(proc (hy-shell-get-process 'internal)))
(with-current-buffer (get-buffer-create output-buffer)
(erase-buffer)
(comint-redirect-send-command-to-process string output-buffer proc nil t)
(set-buffer (process-buffer proc))
(while (and (null comint-redirect-completed)
(accept-process-output proc nil 100 t)))
(set-buffer output-buffer)
(buffer-string))))
Our shell is now ready for autocompletion, eldoc, and other awesome IDE features.
I originally planned to go into font-locking the prompt input (highly
non-trivial), but given the length of this post I will provide and link to
it as a separate future post.
Autocompletion
All the work for autocompletion was in setting up the asynchronous process
support. Lets see how easy autocompletion becomes:
(defconst hy-company-setup-code
"(import [hy.completer [Completer]])
(setv --HYCOMPANY (Completer))"
"Autocompletion setup code to send to the internal process.")
(defconst hy--company-regexp
(rx "'"
(group (1+ (not (any ",]"))))
"'"
(any "," "]"))
"Regex to extra candidates from --HYCOMPANY.")
(defun hy--company-format-str (string)
"Format STRING to send to hy for completion candidates."
(when string
(format "(.%s --HYCOMPANY \"%s\")"
(cond ((s-starts-with? "#" string) ; Tag matches broken in Hy atm
"tag-matches")
((s-contains? "." string)
"attr-matches")
(t
"global-matches"))
string)))
(defun hy--company-candidates (string)
"Get candidates for completion of STRING."
(-when-let* ((command (hy--company-format-str string))
(candidates (hy--shell-send-async command))
(matches (s-match-strings-all hy--company-regexp candidates)))
(-select-column 1 matches))) ; Get match-data-1 for each match
(defun company-hy (command &optional arg &rest ignored)
(interactive (list 'interactive))
(cl-case command
(prefix (company-grab-symbol))
(candidates (hy--company-candidates arg))
(meta (-> arg hy--eldoc-get-docs hy--str-or-empty))))
Completer is a hy builtin that completes a given string and does all the
work here. We simply call it's appropriate method, extract the items in the
retrieved list, and hand it off to company.
company-grab-symbol gets the current symbol which is handed off as arg in
the subsequent call.
The meta argument shows the eldoc output for the current selected company
candidate in the minibuffer, as seen in this post's initial image.
company-hy can then be enabled either through adding to company-backends or
for Spacemacs users adding:
;; Technically this should be within a hy layer, but this still works uncaptured
(spacemacs|add-company-backends
:backends company-hy
:modes hy-mode inferior-hy-mode)
Developing a major-mode that accommodates Spacemacs users will be touched on
in future posts.
Eldoc
For those unfamiliar, in the initial image eldoc-mode provides the formatted
docstring and arguments in the minibuffer for the symbol-at-point (or
completion candidate).
Perhaps surprisingly, Eldoc is a lot more challenging than autocompletion.
For starters, your language won't provide Eldoc like strings (formatted
argument list + first line of docstring) by default. How difficult inspecting
language constructs is entirely dependent on the language. Hy in particular is
difficult due to how macros are implemented and namespaced. I won't provide
the hy-eldoc-setup-code here, it can be found within the source.
Next, your implementation must mirror any relevant DSLs. For lisps, Eldoc
inspects the form opener. It is hydiomatic to:
(setv x "hi")
(.format "{} there" x)
(setv a-list [])
(.append a-list "friend")
We need to send str.format and a-list.append - the form opener alone is
insufficient.
Implementation
Eldoc is setup via the eldoc-documentation-function :
(defun hy-eldoc-documentation-function ()
"Drives `eldoc-mode', retrieves eldoc msg string for inner-most symbol."
(-> (hy--eldoc-get-inner-symbol)
hy--eldoc-get-docs))
(defun hy--mode-setup-eldoc ()
(make-local-variable 'eldoc-documentation-function)
(setq-local eldoc-documentation-function 'hy-eldoc-documentation-function)
(eldoc-mode +1))
There are three core components:
hy--eldoc-send for sending a formatted string and cleaning its output.hy--eldoc-get-inner-symbol getting opening form and completing the dot DSL.hy--eldoc-fontify-text for highlighting the final text string like in the image.
Lets look at fontifying first. We can't blindly apply Hy's font-locks as the
docstring isn't captured in quotes. Since the text is static, we just add the
faces to the string ourselves.
(defun hy--fontify-text (text regexp &rest faces)
"Fontify portions of TEXT matching REGEXP with FACES."
(when text
(-each
(s-matched-positions-all regexp text)
(-lambda ((beg . end))
(--each faces
(add-face-text-property beg end it nil text))))))
(defun hy--eldoc-fontify-text (text)
"Fontify eldoc TEXT."
(let ((kwd-rx
(rx string-start (1+ (not (any space ":"))) ":"))
(kwargs-rx
(rx symbol-start "&" (1+ word)))
(quoted-args-rx
(rx "`" (1+ (not space)) "`")))
(hy--fontify-text
text kwd-rx 'font-lock-keyword-face)
(hy--fontify-text
text kwargs-rx 'font-lock-type-face)
(hy--fontify-text
text quoted-args-rx 'font-lock-constant-face 'bold-italic))
text)
Next lets see the sending and formatting of the shell's raw eldoc output.
(defun hy--eldoc-send (string)
"Send STRING for eldoc to internal process returning output."
(-> string
hy--shell-send-async
hy--eldoc-chomp-output
hy--eldoc-remove-syntax-errors
hy--str-or-nil))
The string/output formatting are implementation details specific to Hy and so
won't be detailed. If we are dealing with an empty string, we return nil
rather than the empty string to pass-by parent when clauses.
The meat of Eldoc is in extracting the innermost symbol of the current point
(defun hy--eldoc-get-inner-symbol ()
"Traverse and inspect innermost sexp and return formatted string for eldoc."
(save-excursion
(-when-let* ((_ (hy-shell-get-process 'internal))
(state (syntax-ppss))
(start-pos (hy--sexp-inermost-char state))
(_ (progn (goto-char start-pos)
(not (hy--not-function-form-p))))
(function (progn (forward-char)
(thing-at-point 'symbol))))
;; Attribute method call (eg. ".format str") needs following sexp
(if (s-starts-with? "." function)
(when (ignore-errors (forward-sexp) (forward-char) t)
(pcase (char-after)
;; Can't send just .method to eldoc
(?\) (setq function nil))
(?\s (setq function nil))
(?\C-j (setq function nil)) ; newline
;; Dot dsl doesn't work on literals
(?\[ (concat "list" function))
(?\{ (concat "dict" function))
(?\ (concat "str" function)) ; the " is deleted in blog as breaks rainbow.js
;; Otherwise complete the dot dsl
(_ (progn
(forward-char)
(concat (thing-at-point 'symbol) function)))))
function))))
So Eldoc's path is to call hy--eldoc-get-inner-symbol if an internal process
is active, syntax-ppss indicates we are within a form, and that the
innermost form is a symbol. The completed string is sent off to the internal
process we've built up, the output is chomped of quote characters and the
prompt and syntax errors (eg. completing "str." while we are still typing)
are ignored. The result is fontified and returned by the documentation function.
Spacemacs shift-k documentation lookup
A feature of Spacemacs is typing "K" to perform
spacemacs/evil-smart-doc-lookup to get the full documentation of the
symbol-at-point in a separate buffer.
Using Eldoc's documentation functions, with slightly different formatting, we
already have most of shift-K implemented.
We moved most of hy-eldoc-documentation-function into hy--eldoc-get-docs which
distinctly accepts an optional argument for buffer-style rather than
eldoc-style formatting.
We then create a mirror of the documentation function as
hy--docs-for-thing-at-point. We format the text to account for newlines
(newlines from process output are escaped so we must trim one backslash from
each newline).
(defun hy--docs-for-thing-at-point ()
"Mirrors `hy-eldoc-documentation-function' formatted for a buffer, not a msg."
(-> (thing-at-point 'symbol)
(hy--eldoc-get-docs t)
hy--format-docs-for-buffer))
(defun hy--format-docs-for-buffer (text)
"Format raw hydoc TEXT for inserting into hyconda buffer."
(when text
(-let [kwarg-newline-regexp
(rx ","
(1+ (not (any "," ")")))
(group-n 1 "\\\n")
(1+ (not (any "," ")"))))]
(--> text
(s-replace "\\n" "\n" it)
(replace-regexp-in-string kwarg-newline-regexp "newline" it nil t 1)))))
It is interesting how "K" is actually called, I'm not sure if any other
function operates quite the same. "K" calls the function bound to "SPC m h h".
(spacemacs/set-leader-keys-for-major-mode 'hy-mode
"hh" 'hy-describe-thing-at-point)
Lastly we need to create, switch-to, and insert the retrieved docs as hy-describe-thing-at-point.
(defun hy-describe-thing-at-point ()
"Implement shift-k docs lookup for `spacemacs/evil-smart-doc-lookup'."
(interactive)
(-when-let* ((text (hy--docs-for-thing-at-point))
(doc-buffer "*Hyconda*"))
(with-current-buffer (get-buffer-create doc-buffer)
(erase-buffer)
(switch-to-buffer-other-window doc-buffer)
(insert text)
(goto-char (point-min))
(forward-line)
(insert "------\n")
(fill-region (point) (point-max))
;; Eventually make hyconda-view-minor-mode, atm this is sufficient
(local-set-key "q" 'quit-window)
(when (fboundp 'evil-local-set-key)
(evil-local-set-key 'normal "q" 'quit-window)))))
Closing
There are several other features worth discussion like font-locking the
shell prompt input and the send-(form/region/buffer)-to-shell that are
sizable enough to warrant their own posts later. With this post, the series is
caught up to the current featureset. Linting would be the next big problem to
attack. I'd also like to integrate ert.
Going without Company and Eldoc has helped me appreciate the value in IDEs.
Hy is quickly becoming a pleasant development experience.
My only guidance was source code. I hope this series make the problem more
tractable for prospective major mode authors.
with Word: shudder. 
