Timo Geusch: Emacs 26.1-RC1 on the Windows Subsystem for Linux

As posted in a few places, Emacs 26.1-RC1 has been released. Following up my previous experiments with running Emacs on the Windows Subsystem for Linux, I naturally had to see how the latest version would work out. For that, I Read More

The post Emacs 26.1-RC1 on the Windows Subsystem for Linux appeared first on The Lone C++ Coder's Blog.

-1:-- Emacs 26.1-RC1 on the Windows Subsystem for Linux (Post Timo Geusch)--L0--C0--April 19, 2018 10:07 PM

Irreal: Blogging with Hugo and Org Mode

Over at ZCL.SPACE there’s an informative post about blogging with Hugo and Org mode. The post serves as a reasonable go-by for setting up and using Hugo for an attractive, static blog. Although Hugo now supports Org markup natively, the post recommends exporting to Hugo compatible format using ox-hugo.

The ZCL blog is a bit unique because it is divided into distinct sections. Each section has its own Org file in which each level-one heading is a post. If you’re looking for a way of writing a static blog with Emacs and Org mode, take a look at this post for some good ideas.

While I still prefer to blog with WordPress, my ongoing problems org2blog and the XML-RPC interface to WordPress that it uses are tempting me to consider switching over to a simpler static blog. There’s a lot to be said for a blogging platform in which pushing a post amounts to transferring a file to the site.

-1:-- Blogging with Hugo and Org Mode (Post jcs)--L0--C0--April 18, 2018 03:32 PM

sachachua: 2018-04-16 Emacs news

Links from reddit.com/r/emacs, /r/orgmode, /r/spacemacs, Hacker News, planet.emacsen.org, YouTube, the changes to the Emacs NEWS file, and emacs-devel.

-1:-- 2018-04-16 Emacs news (Post Sacha Chua)--L0--C0--April 16, 2018 11:59 PM

Irreal: <t> Is Undefined

Is anyone else seeing the error message <t> is undefined when using Emacs? Occasionally Emacs will respond to a keypress with that error in the echo area and Messages buffer. The key is otherwise discarded but pressing it again will result in the correct action.

I’ve been seeing this error since Emacs 25. I vaguely remember seeing somewhere that it was a known error and was already fixed (it was probably this Emacs Stack Exchange question). I was disappointed to see the error still occurring under Emacs 26, although it doesn’t seem to happen as often.

I’m getting ready to file a bug report and am trying to gather as much information as I can. If you have any information on this bug or even if you’ve seen it please leave a comment. If you’ve seen the error, please include the Emacs version and your OS.

-1:-- &lt;t&gt; Is Undefined (Post jcs)--L0--C0--April 16, 2018 03:32 PM

Marcin Borkowski: counsel-org-clock

Some time ago I blogged about my Org-mode hydra. It turned out that that a link to that short article was posted to Reddit, and one of the commenters pointed out his counsel-org-clock package.
-1:-- counsel-org-clock (Post)--L0--C0--April 15, 2018 07:18 AM

Rubén Berenguel: More emacs configuration tweaks (multiple-cursor on click, minimap, code folding, ensime eval overlays)

At Affectv we use a wide range of editors: Sublime, Atom, Emacs, Pycharm, IntelliJ... Actually only two people use the same editor! As such, from time to time I see things in other people's editors that I would like to have as well. So, yesterday I decided to improve on some configuration settings on Spacemacs.

Click for multiple-cursors

I saw this on Jordi's Sublime, and it is much more comfortable than using more-like-this or similar helper functions, even if I need to use the trackpad to do so. After all, a multi-cursor edit (proper edit, not as a substitute for a macro) is rare enough that I can tolerate leaving the home row. Easy enough to configure thanks to Magnar Sveen.

(global-unset-key (kbd "M-<down-mouse-1>"))
(global-set-key (kbd "M-<mouse-1>") 'mc/add-cursor-on-click)

Minimap

Also from Sublime, I used to have this on my old emacs setup. As simple as adding minimap to the list of additional packages and configuring its property group. See animation below.

dotspacemacs-additional-packages '(helm-dash key-chord pig-mode mmm-mode minimap origami ansible)

Folding

I have always loved how clean vim's folding works, and how Sublime has this nice folding. Then I found origami-mode and my emacs-life was complete. I tweaked a little the folding functions so that minimap was updated on fold (for some reason it is not, I guess minimap is tied to the "modified" hook or similar). I bound z and Z (and A-z which maps to æ in Colemak) to the basic fold operations.

(eval-after-load 'origami
'(progn
(defun rb-show-only (buffer point)
(interactive (list (current-buffer) (point)))
(progn (origami-show-only-node buffer point)
(minimap-new-minimap)))

(defun rb-toggle-rec (buffer point)
(interactive (list (current-buffer) (point)))
(progn (origami-recursively-toggle-node buffer point)
(minimap-new-minimap)))

(define-key evil-normal-state-map "æ" 'rb-show-only)
(define-key evil-normal-state-map "Z" 'origami-toggle-node)
(define-key evil-visual-state-map "Z" 'origami-toggle-node)
(define-key evil-insert-state-map "C-Z" 'origami-toggle-node)
(define-key evil-normal-state-map "z" 'rb-toggle-rec)
(define-key evil-visual-state-map "z" 'rb-toggle-rec)
(define-key evil-insert-state-map "C-z" 'rb-toggle-rec)
)))


For some reason just advising the functions with after didn't work, this is not great but does work. I left the Z bindings as they are, since I have not used them yet, and will probably delete them if I keep not using them.



Note (2018): I moved to use yafolding (specifically yafolding-toggle-all and yafolding-toggle-element) at the end of 2017. Although origami is very good, yafolding works seamlessly under more languages. I bound Return in normal mode to toggle element and C-return to toggle all, as well as unbounding all the default bindings.

Execution overlays in Ensime (Scala)

I saw this for Cider in the emacs church meeting from August, and heard @fommil (I think it was him) mention that it was coming to ensime. And indeed it was. And it's easy enough to use C-c C-v C-r (thing of it as extended command, eval, region to remember), given an open inferior Scala interpreter. Symbol prettify does not apply to overlays, so you need to customise the arrow used therein.




-1:-- More emacs configuration tweaks (multiple-cursor on click, minimap, code folding, ensime eval overlays) (Post Rubén Berenguel (noreply@blogger.com))--L0--C0--April 14, 2018 04:13 PM

Marcin Borkowski: counsel-yank-pop

Some time ago I wrote a tip about yanking, and I was informed in a comment that helm has helm-show-kill-ring, which is a replacement for M-y. Turns out that the Ivy suite has counsel-yank-pop, which is a similar thing. Even better, instead of binding M-y to it manually, you can just turn counsel-mode on, and many Emacs commands will be automatically remapped to their counsel equivalents. Nice!
-1:-- counsel-yank-pop (Post)--L0--C0--April 09, 2018 08:04 PM

sachachua: 2018-04-09 Emacs news

Links from reddit.com/r/emacs, /r/orgmode, /r/spacemacs, Hacker News, planet.emacsen.org, YouTube, the changes to the Emacs NEWS file, and emacs-devel.

-1:-- 2018-04-09 Emacs news (Post Sacha Chua)--L0--C0--April 09, 2018 02:31 PM

Rubén Berenguel: SyncTeX and pdf-view-mode for emacs

Or, destiny is cruel
Back in the days of yore, when I was switching between my Windows machine and a Linux machine, I remember having SyncTeX active in my Windows machine. It was a wonderful experience: SyncTeX lets you click anywhere on a generated file from LaTeX and gets back to your editor, to the place generating where you clicked. This was extremely useful, specially later on when you need to adjust many formulas to fit and you need a bit of back-and-forth-ing.

Then I got a Mac, and since Preview is so handy I slowly forgot about SyncTeX. Time went on, and I merrily kept on editing LaTeX files as usual. I even managed to deliver my PhD dissertation a couple weeks ago, the formal speech will be in a month or two (come at your own risk). AucTeX’s preview saved most of the days, so I slowly even forgot SyncTeX existed. Shame on me indeed.

The other day I got an annotated PDF from one of my advisors, and I just couldn’t open the annotations. I tried all programs I had for Mac, and no luck: annotations weren’t showing, just saw the icons. Surveying for some command-line tool to extract annotations (just in case) I found pdf-tools, a replacement for DocView based on Poppler. It had the awesome ability of actually displaying annotations, with it it was pretty clear the annotations were broken in that PDF. I got a new set of PDFs from my advisor with the annotations in place, though. While waiting for it to arrive…

I saw SyncTeX was an option of pdf-tools. I had been using that, hadn’t I? So, I activated SyncTeX in AucTeX (it is TeX-source-correlate-method, see here) and indeed: I could have two frames, one with the actual LaTeX sources and the other with a PDF, and go from one to the other. Even hyperreferences in PDF work! See (well, click on the full-screen mode to see it larger or you won't see anything)!



Getting pdf-tools to work wasn’t incredibly tricky (given the hoops you need for some packages, sometimes). Just

brew install pdf-tools

and after reading

brew info pdf-tools

I was told to run

  emacs -Q --batch --eval "(package-install-file 
\"/usr/local/Cellar/pdf-tools/0.60/pdf-tools-0.60.tar\")"

and this does the trick (well, change emacs for your actual emacs, which likely is /Applications/Emacs.app/Contents/MacOS/Emacs) You’ll also need to add to your .emacs file (or temporarily in your *scratch* buffer)


(setenv "PKG_CONFIG_PATH" (concat "/usr/local/Cellar/zlib/1.2.8/lib/pkgconfig" ":"

"/usr/local/lib/pkgconfig:/opt/X11/lib/pkgconfig"))


(getenv "PKG_CONFIG_PATH")


and run

(pdf-tools-install)

as advised in the package’s README. And that's it, open a PDF and activate pdf-view-mode to check everything is in place. Well worth it!
-1:-- SyncTeX and pdf-view-mode for emacs (Post Rubén Berenguel (noreply@blogger.com))--L0--C0--April 08, 2018 06:22 PM

Chen Bin (redguardtoo): Use Imenu to list comments in current buffer

evil-nerd-commenter v3.2.0 has a new function evilnc-imenu-create-index-function.

Imenu could use this function to list all comments in current file.

Usage:

(require 'counsel)
(defun counsel-imenu-comments ()
  "Imenu display comments."
  (interactive)
  (let* ((imenu-create-index-function 'evilnc-imenu-create-index-function))
    (counsel-imenu)))

Screen cast:

counsel-imenu-comments.gif

-1:-- Use Imenu to list comments in current buffer (Post Chen Bin)--L0--C0--April 08, 2018 02:12 AM

Alex Bennée: Working with dired

I’ve been making a lot more use of dired recently. One use case is copying files from my remote server to my home machine. Doing this directly from dired, even with the power of tramp, is a little too time consuming and potentially locks up your session for large files. While browsing reddit r/emacs I found a reference to this post that spurred me to look at spawning rsync from dired some more.

Unfortunately the solution is currently sitting in a pull-request to what looks like an orphaned package. I also ran into some other problems with the handling of where rsync needs to be run from so rather than unpicking some unfamiliar code I decided to re-implement everything in my own package.

I’ve still got some debugging to do to get it to cleanly handle multiple sessions as well as a more detailed mode-line status. Once I’m happy I’ll tag a 0.1 and get it submitted to MELPA.

While getting more familiar with dired I also came up with this little helper:

(defun my-dired-frame (directory)
  "Open up a dired frame which closes on exit."
  (interactive)
  (switch-to-buffer (dired directory))
  (local-set-key
   (kbd "C-x C-c")
   (lambda ()
     (interactive)
     (kill-this-buffer)
     (save-buffers-kill-terminal 't))))

Which is paired with a simple alias in my shell setup:

alias dired="emacsclient -a '' -t -e '(my-dired-frame default-directory)'"

This works really nicely for popping up a dired frame in your terminal window and cleaning itself up when you exit.

-1:-- Working with dired (Post Alex)--L0--C0--April 07, 2018 10:12 AM

scripter.co | Emacs: Org Table Spreadsheet

Normalizing

Table 1: Org table Normalizing
Normalized col $1 Normalized col $3 Ratio of col $2 / $4
Col $1 Col $2 Col $3 Col $4 Col $5
79922.1 6.85 0.146993824 4.27 1.60
79185.9 6.78 0.146976008 4.27 1.59
64041.0 5.48 0.147067031 4.27 1.28
15452.5 1.32 0.034456713 1.00 1.32
11675.7 1.00 0.034460162 1.00 1.00

Equations Breakdown

Equation: #+tblfm: $2=$1/@7$1;%0.2f::$4=$3/@7$3;%0.2f::$5=$2/$4;%0.2f

  • $2=$1/@7$1 – Assign the value of a field in column 2 ($2) to the value of the field in column 1 in the same row ($1) divided by the value of the field in row 7, column 1 (@7$1).
    • @ is row, $ is column.
    • Row and column numbers start with 1 (not 0).
  • $4=$3/@7$3 – Similar to above, but with different row/column numbers.
  • $5=$2/$4 – Assign the value of a field in column 5 ($5) to the value of the field in column 2 in the same row ($2) divided by the value of the field in column 4 in the same row as well ($4).
  • ; is used to separate the equation from the formatting.
  • For all the 3 equations, formatting is applied using %0.2f i.e. the numbers will be formatted with 2 decimal places. See Field Formatters in Org table for more info.
  • Each of those equations are applied to the same table by concatenating them with ::.

Evaluate the whole table

Do C-u C-c C-c with point anywhere in the table, or do C-c C-c with point on the #+tblfm line.

References

-1:-- Org Table Spreadsheet (Post Kaushal Modi)--L0--C0--April 06, 2018 02:32 PM

scripter.co | Emacs: Optimize your FontAwesome

This post briefly goes through the FontAwesome 5 setup, the problem statement and a solution for that.

I signed up for the FontAwesome 5 Pro Kickstarter, and I have to say that this is one of my few Kickstarter fundings that delivered really well!

    Thank you FontAwesome!

FontAwesome 5 Setup

FontAwesome 5 supports an awesome new way of loading icons, using SVG with JS. The benefit is that you simply load one fontawesome-all.js file and everything just works!.

<head>
  <!--load everything-->
  <script defer src="/static/fontawesome/fontawesome-all.js"></script>
</head>
<body>
  <!--icon-->
  <i class="fas fa-user"></i>
</body>
Code Snippet 1: Fontawesome 5 Setup

Problem

With FontAwesome 5 Pro, I get about 2700 icons packed in that fontawesome-all.js. While that is great, there are 2 issues:

  1. I don’t need all of those 2700 icons on my site. That file is roughly 1.7 MB (minified or not) which I need not unnecessarily load for all my visitors.
  2. Anyone can easily pirate the whole JS file and use it for free!

Solution

The solution is simple — I just comment out all the lines with SVG code for the icons that I don’t use.

But searching for the icons I need in all the icon packs (Solid, Regular, Light, Brand) in that single 5k+ line JS file is not fun, especially when I want to add/remove icons occasionally.

And so fontawesome-choose was born!

  • I list the icons I need in fontawesome-choose-icons.
  • Run M-x fontawesome-choose in the fontawesome-all.js file buffer.

The code is in the next section, but you can also find the latest version in my repo.

Note that while I have used this on Fontawesome 5 Pro, it should work just as well on Fontawesome 5 Free too.

Code

;; Helper function `fontawesome-choose' used to uncomment only the
;; icons the user cares about in `fontawesome-all.js'.

(defconst fontawesome-choose-icons '("list-alt" ;categories
                                     "tags"
                                     "rss"
                                     "link"
                                     "heart" ;like
                                     "reply"
                                     "retweet"
                                     "github" ;"github-alt" "github-square"
                                     "twitter" ;"twitter-square"
                                     "gitlab")
  "List of icons to choose from fontawesome-all.js.
Used in `fontawesome-choose'.")

(defun fontawesome-choose ()
  "Comment out all icons in fontawesome-all.js except the selected few.

Minifying the resultant .js will then remove the commented icons,
thus drastically reducing the minified JS size.

Set the `fontawesome-choose-icons' variable to the list of icons
that you want to keep uncommented."
  (interactive)
  (let ((case-fold-search nil)
        (count 0))
    (save-excursion
      (goto-char (point-min))
      (while (re-search-forward "^var icons" nil :noerror)
        (let ((begin (progn
                       (forward-line 1)
                       (point)))
              end)
          (re-search-forward "^\\};")
          (forward-line 0)
          (backward-char 1)
          (setq end (point))
          ;; First comment all the lines
          (save-excursion
            (narrow-to-region begin end)
            (goto-char (point-min))
            (while (re-search-forward "^\\s-*\\(\"\\)" nil :noerror)
              (replace-match "// \\1" nil nil nil 1))
            (widen))
          ;; Now uncomment only the selected icons
          (save-excursion
            (narrow-to-region begin end)
            (goto-char (point-min))
            (let* ((icon-regexp (regexp-opt fontawesome-choose-icons 'symbols))
                   (regexp (format "^\\s-*\\(//\\s-*\\)\"%s\":" icon-regexp)))
              (while (re-search-forward regexp nil :noerror)
                (replace-match "" nil nil nil 1)
                (setq count (1+ count))))
            (widen))))
      (message "fontawesome-choose: Uncommented %d icons matching %S"
               count fontawesome-choose-icons))))
Code Snippet 2: fontawesome-choose — Comment out all the FontAwesome icons you don't need

Result

This solution solves the first problem wonderfully — After minification, fontawesome-all.js is just 71 KB (down from 1.7 MB).

That’s a 25x factor reduction! The minification step is important because that removes all the commented lines from the JS. I do the minification using tdewolff/minify.

The second problem, that of piracy, is difficult to solve completely using this. But whoever pirates this reduced FontAwesome from my site won’t get the real deal 😎.

Emacs-Lisp saves the day once again!

-1:-- Optimize your FontAwesome (Post Kaushal Modi)--L0--C0--April 04, 2018 07:20 PM

Alexander Gromnitsky: Emacs, xkb & xxkb

The song that never ends.

Every kid has its own scheme. Through the years I've picked out mine.

The prerequisites:

  1. You have tuned up xkb and know what key is your X layout switcher. (In the past I've used left Ctrl to toggle en/ru, but recently switched to capslock.)
  2. You're using xxkb as your X Window indicator of the current layout. Not from Gnome, not from KDE, etc.

We want: use capslock for emacs toggle-input-method too. We don't want to modify xorg system files or hack part of emacs written in C.

The problem is that emacs (under xorg-7.5.1, for example) ignores ISO_Next_Group completely. You cannot bind anything to the key that emits that event.

But there is a clever trick I've googled. You translate ISO_Next_Group to some unused key in emacs, for example Super-A. (Super == left win key on my machine.):

 (setf (gethash #xfe08 x-keysym-table) (aref (kbd "s-a") 0)) 

#xfe08 is ISO_Next_Group code. Then bind Super-A to toggle emacs keyboard layout:

 (global-set-key (kbd "s-a") '(lambda () (toggle-input-method))) 

OK. Next we need somehow disable system X switcher for all emacs windows. For xxkb this is done by adding this to its configuration file:

 XXkb.app_list.wm_class_class.alt_group1: Emacs 

And that's all.

As a bonus, I've added another very handy indicator of the current input method in emacs: cursor color.

;; default cursor color for all X frames
(add-to-list 'default-frame-alist '(cursor-color . "black"))

;; toggle the input method and the cursor color in one place
(global-set-key (kbd "s-a") '(lambda ()
                             (interactive)
                             (toggle-input-method)
                             (if window-system
                                 (set-cursor-color
                                  (if current-input-method
                                      "red3"
                                    "black")))
                             ))
-1:-- Emacs, xkb &amp; xxkb (Post ag (noreply@blogger.com))--L0--C0--April 02, 2018 01:25 PM

Alexander Gromnitsky: 'indent-region' in the Emacs batch mode

I had an old project of mine that was using an archaic tab-based indentation w/ the assumed tab width of 4. The mere opening of its files in editors other than Emacs was causing such pain that I decided to do what you should rarely do--reindent the whole project.

(A side note: if you're setting the tab width to a value other than 8 & simultaneously is using tabs for indentation, you're a bad person.)

The project had ~40 files. Manually using Emacs indent-region for each file would have been too wearisome. Then I remembered that Emacs has the batch mode.

A quick googling gave me a recipe akin to:

$ emacs -Q -batch FILE --eval '(indent-region (point-min) (point-max))' \
  -f save-buffer

It would have worked but it had several drawbacks as a general solution, for it:

  1. modifies a file in-place;
  2. can't read from the stdin;
  3. doesn't work w/ multiple files forcing a user to use xargs(1) of smthg;
  4. if FILE doesn't exist, Emacs quietly creates an empty file, whereas a user probably expects to see an error message.

Whereas an ideal little script should:

  1. never overwrite the source file;
  2. read the input from the stdin or from its arguments;
  3. spit out the result to the stdout;
  4. handle the missing input reasonably well.

Everything else should be accommodated by the shell, including the item #2 from the 'drawbacks' list above.

Using Emacs for such a task is tempting, for it supports a big number of programming modes, giving us the ability to employ the editor as a universal reindenting tool, for example, in makefiles or Git hooks.

Writing to the stdout

We can slightly modify the above recipe to:

$ emacs -Q -batch FILE --eval '(indent-region (point-min) (point-max))' \
  --eval '(princ (buffer-string))'

In -f stead we are using --eval twice. buffer-string function does exactly what it says: returns the contents of the current buffer as a string.

$ cat 1.html
<p>
hey, <i>
what's up</i>?
</p>

$ emacs -Q -batch 1.html --eval '(indent-region (point-min) (point-max))' \
  --eval '(princ (buffer-string))'
Indenting region...
Indenting region...done
<p>
  hey, <i>
    what's up</i>?
</p>

The "Indenting region" lines come from Emacs message function (which the progress reporter uses). In the batch mode message prints the lines to the stderr.

The solution also address the item #3 from the "drawbacks" list--Emacs doesn't create an empty file on the invalid input, although it doesn't indicate the error properly, i.e., it fails w/ the exit code 0.

Processing all files at once

If you know what are you doing & the files you're going to process are under Git, overwriting the source it not a big deal. Perhaps for a quick hack this script will do:

:; exec emacs -Q --script "$0" -- "$@" # -*- emacs-lisp -*-

(defun indent(file)
  (set-buffer "*scratch*")
  (if (not (file-directory-p file))
      (when (and (file-exists-p file) (file-writable-p file))
        (message "Indenting `%s`" file)

        (find-file file)
        (indent-region (point-min) (point-max))
        (save-buffer))))

(setq args (cdr argv))                  ; rm --
(dolist (val args)
  (indent val))

If you put the src above into a file named emacs-indent-overwrite & add executable bits to it, then the shell thinks it's a usual sh-script that doesn't have a shebang line. A colon in sh is a noop, but on stumbling upon exec the shell replaces itself with the command

emacs -Q --script emacs-indent-overwrite -- arg1 arg2 ...

When Emacs reads the script, it doesn't expect it to be a sh one, but hopefully the file masks itself as a true Elisp, for in Elisp a symbol whose name starts with a colon is called a keyword symbol. : is a symbol w/o a name (a more usual constant would be :omg) that passes the check because it satisfies keywordp predicate:

ELISP> (keywordp :omg)
t
ELISP> (keywordp :)
t

Everything else in the file is usual Elisp code. Here's how it works:

$ emacs-indent-overwrite src/*
Indenting ‘src/1.html‘
Indenting region...
Indenting region...done
Indenting ‘src/2.html‘
Indenting region...
Indenting region...done

The only thing worth mentioning about the script is why indent procedure has (set-buffer "*scratch*") call. It's an easy way to switch the current directory to the directory from which the script was started. This is required, for find-file modifies the default directory of the current buffer (via modifying a buffer-local default-directory variable). The other way is to modify args w/ smthg like

(setq args (mapcar 'expand-file-name args))

& just pass the file argument as a full path.

Reading from the stdin

There is next to none info about the stdin in the Emacs Elisp manual. The section about minibuffers hints us that for reading from the standard input in the batch mode we ought to seek out for functions that normally read kbd input from the minibuffer.

One of such functions is read-from-minibuffer.

$ cat read-from-minibuffer
:; exec emacs -Q --script "$0"
(message "-%s-" (read-from-minibuffer ""))

$ printf hello | ./read-from-minibuffer
-hello-
$ printf 'hello\nworld\n' | ./read-from-minibuffer
-hello-

The function only read the input up to the 1st newline which it also impertinently ate. This leaves us w/ a dilemma: if we read multiple lines in a loop, should we unequivocally assume that the last line contained the newline?

read-from-minibuffer has another peculiarity: on receiving the EOF character it raises an unhelpful error:

$ ./read-from-minibuffer
^D
Error reading from stdin
$ echo $?
255

The simplest Elisp cat program must watch out for that:

$ ./cat.el < cat.el
:; exec emacs -Q --script "$0"
(while (setq line (ignore-errors (read-from-minibuffer "")))
  (princ (concat line "\n")))

Next, if we read the stdin & print the result to the stdout, our "ideal" reindent script cannot rely on find-file anymore, unless we save the input in a tmp file. Or we can leave out "heavy" find-file altogether & just create a temp buffer & copy the input there for the further processing. The latter implies we must manually set the proper major mode for the buffer, otherwise indent-region won't do anything good.

set-auto-mode function does the major mode auto-detection. One of the 1st hints it looks for is the file extension, but the stdin has none. We can ask the user to provide one in the arguments of the script.

:; exec emacs -Q --script "$0" -- "$@" # -*- emacs-lisp -*-

(defun indent(mode text)
  (with-temp-buffer
    (set-visited-file-name mode)
    (insert text)
    (set-auto-mode t)
    (message "`%s` major mode: %s" mode major-mode)

    (indent-region (point-min) (point-max))
    (buffer-string)))

(defun read-file(file)
  (with-temp-buffer
    (insert-file-contents file)
    (buffer-string)))

(defun read-stdin()
  (let (line lines)
    (while (setq line (ignore-errors (read-from-minibuffer "")))
      (push line lines))
    (push "" lines)
    (mapconcat 'identity (reverse lines) "\n")
    ))

 

(setq args (cdr argv))                  ; rm --

(setq mode (car args))
(if (equal "-" mode)
    (progn
      (setq mode (nth 1 args))
      (if (not mode) (error "No filename hint argument, like .js"))
      (setq text (read-stdin)))
  (setq text (read-file mode)))

(princ (indent mode text))

If the 1st argument to the script is '-', we look for the hint in the next argument & then start reading the stdin. It the 1st arg isn't '-' we assume it's a file that we insert into a temp buffer & return its contents as a string.

The

(mapconcat 'identity (reverse lines) "\n")

line in read-stdin procedure is an equivalent of

['world', 'hello'].reverse().join("\n")

in JavaScript.

Some examples:

$ printf "<p>\nhey, what's up?\n</p>" | ./emacs-indent - .html
‘.html‘ major mode: html-mode
Indenting region...
Indenting region...done
<p>
  hey, what's up?
</p>

$ printf "<p>\nhey, what's up?\n</p>" | ./emacs-indent - .txt
‘.txt‘ major mode: text-mode
Indenting region...
Indenting region...done
<p>
hey, what's up?
</p>

$ ./emacs-indent src/2.html 2>/dev/null
<p>
  not much
</p>
-1:-- 'indent-region' in the Emacs batch mode (Post ag (noreply@blogger.com))--L0--C0--April 02, 2018 01:14 PM

Bryan Murdock: Fixing xref-find-references

I was annoyed that xref-find-references searched more than the files in the TAGS file (it seems to search everything below the top level directory of the TAGS file?) so I went looking for a fix. I found that apparently my emacs knowledge is out of date (the fact that it's xref now and not tags was my first clue). I couldn't find any way to customize xref-find-references. Instead I found people referring to project, ivy, helm, confirm, The Silver Searcher (ag), ripgrep (rg), and dumb jump. I didn't go all the way and get into project or ivy or any of the others, but I did download ag and rg and tried them from a command-line outside of emacs and saw exactly what I was expecting xref-find-references to do. I figured all I needed was to replace xref-find-references with one of those. I got ag.el installed and working before any of the ripgrep packages (there's both rg.el and ripgrep.el) and then struggled to remap M-? to call ag-project instead of xref-find-references. The thing that finally worked was remapping commands. Here's the magic:

(define-key global-map [remap xref-find-references] 'ag-project)

And actually, to work completely like xref-find-references I added one option to the ag command, --word-regexp, like so (oh, I also removed --stats which is there by default in ag.el):

(setq ag-arguments (list "--word-regexp" "--smart-case"))

Much better. Are all those other packages worth digging into? I'm not particularly unhappy with ido.
-1:-- Fixing xref-find-references (Post Bryan (noreply@blogger.com))--L0--C0--March 30, 2018 05:49 PM

Jonas Bernoulli: Magit 2.12 released

I am excited to announce the release of Magit version 2.12, consisting of 610 commits since the last feature release six months ago.
-1:-- Magit 2.12 released (Post)--L0--C0--March 29, 2018 07:30 PM

Alex Bennée: Solving the HKG18 puzzle with org-mode

One of the traditions I like about Linaro’s Connect event is the conference puzzle. Usually set by Dave Piggot they provide a challenge to your jet lagged brain. Full disclosure: I did not complete the puzzle in time. In fact when Dave explained it I realised the answer had been staring me in the face. However I thought a successful walk through would make for a more entertaining read 😉

First the Puzzle:

Take the clues below and solve them. Once solved, figure out what the hex numbers mean and then you should be able to associate each of the clue solutions with their respective hex numbers.

Clue Hex Number
Lava Ale Code 1114DBA
Be Google Roe 114F6BE
Natural Gin 114F72A
Pope Charger 121EE50
Dolt And Hunk 12264BC
Monk Hops Net 122D9D9
Is Enriched Tin 123C1EF
Bran Hearing Kin 1245D6E
Enter Slim Beer 127B78E
Herbal Cabbages 1282FDD
Jan Venom Hon Nun 12853C5
A Cherry Skull 1287B3C
Each Noun Lands 1298F0B
Wave Zone Kits 12A024C
Avid Null Sorts 12A5190
Handcars All Trim 12C76DC

Clues

It looks like all the clues are anagrams. I was lazy and just used the first online anagram solver that Google pointed me at. However we can automate this by combining org-mode with Python and the excellent Beautiful Soup library.

from bs4 import BeautifulSoup
import requests
import re

# ask internet to solve the puzzle
url="http://anagram-solver.net/%s" % (anagram.replace(" ", "%20"))
page=requests.get(url)

# fish out the answers
soup=BeautifulSoup(page.text)
answers=soup.find("ul", class_="answers")
for li in answers.find_all("li"):
    result = li.text
    # filter out non computer related or poor results
    if result in ["Elmer Berstein", "Tim-Berners Lee", "Babbage Charles", "Calude Shannon"]:
        continue
    # filter out non proper names
    if re.search("[a-z] [A-Z]", result):
        break

return result

So with :var anagram=clues[2,0] we get

Ada Lovelace

I admit the “if result in []” is a bit of hack.

Hex Numbers

The hex numbers could be anything. But lets first start by converting to something else.

Hex Prompt Number
1114DBA 17911226
114F6BE 18151102
114F72A 18151210
121EE50 19000912
12264BC 19031228
122D9D9 19061209
123C1EF 19120623
1245D6E 19160430
127B78E 19380110
1282FDD 19410909
12853C5 19420101
1287B3C 19430204
1298F0B 19500811
12A024C 19530316
12A5190 19550608
12C76DC 19691228

The #+TBLFM: is $1='(identity remote(clues,@@#$2))::$2='(string-to-number $1 16)

This is where I went down a blind alley. The fact all they all had the top bit set made me think that Dave was giving a hint to the purpose of the hex number in the way many cryptic crosswords do (I know he is a fan of these). However the more obvious answer is that everyone in the list was born in the last millennium.

Looking up Birth Dates

Now I could go through all the names by hand and look up their birth dates but as we are automating things perhaps we can use computers for what they are good at. Unfortunately there isn’t a simple web-api for looking up this stuff. However there is a project called DBpedia which takes Wikipedia’s data and attempts to make it semantically useful. We can query this using a semantic query language called SparQL. If only I could call it from Emacs…

PREFIX dbr: <http://dbpedia.org/resource/>
PREFIX dbo: <http://dbpedia.org/ontology/>
PREFIX dbp: <http://dbpedia.org/property/>

select ?birthDate where {
  { dbr:$name dbo:birthDate ?birthDate }
  UNION
  { dbr:$name dbp:birthDate ?birthDate }
}

So calling with :var name="Ada_Lovelace" we get

"birthDate"
1815-12-10
1815-12-10

Of course it shouldn’t be a surprise this exists. And in what I hope is a growing trend sparql-mode supports org-mode out of the box. The $name in the snippet is expanded from the passed in variables to the function. This makes it a general purpose lookup function we can use for all our names.

There are a couple of wrinkles. We need to format the name we are looking up with underscores to make a valid URL. Also the output spits out a header and possible multiple birth dates. We can solve this with a little wrapper function. It also introduces some rate limiting so we don’t smash DBpedia’s public SPARQL endpoint.

;; rate limit
(sleep-for 1)
;; do the query
(let* ((str (s-replace-all '((" " . "_") ("Von" . "von")) name))
       (ret (eval
             (car
              (read-from-string
               (format "(org-sbe get-dob (name $\"%s\"))" str))))))
  (string-to-number (replace-regexp-in-string "-" "" (car (cdr (s-lines ret))))))

Calling with :var name="Ada Lovelace" we get

18151210

Full Solution

So now we know what we are doing we need to solve all the puzzles and lookup the data. Fortunately org-mode’s tables are fully functional spreadsheets except they are not limited to simple transformations. Each formula can be a fully realised bit of elisp, calling other source blocks as needed.

Clue Solution DOB
Herbal Cabbages Charles Babbage 17911226
Be Google Roe George Boole 18151102
Lava Ale Code Ada Lovelace 18151210
A Cherry Skull Haskell Curry 19000912
Jan Venom Hon Nun John Von Neumann 19031228
Pope Charger Grace Hopper 19061209
Natural Gin Alan Turing 19120623
Each Noun Lands Claude Shannon 19160430
Dolt And Hunk Donald Knuth 19380110
Is Enriched Tin Dennis Ritchie 19410909
Bran Hearing Kin Brian Kernighan 19420101
Monk Hops Net Ken Thompson 19430204
Wave Zone Kits Steve Wozniak 19500811
Handcars All Trim Richard Stallman 19530316
Enter Slim Beer Tim Berners-Lee 19550608
Avid Null Sorts Linus Torvalds 19691228

The #+TBLFM: is $1='(identity remote(clues,@@#$1))::$2='(org-sbe solve-anagram (anagram $$1))::$3='(org-sbe frob-dob (name $$2))

The hex numbers are helpfully sorted so as long as we sort the clues table by the looked up date of birth using M-x org-table-sort-lines we are good to go.

You can find the full blog post in raw form here.

-1:-- Solving the HKG18 puzzle with org-mode (Post Alex)--L0--C0--March 26, 2018 10:19 AM

Flickr tag 'emacs': url

kabads posted a photo:

url

-1:-- url (Post kabads (nobody@flickr.com))--L0--C0--March 23, 2018 06:07 PM

emacsninja: Bridging the Ancient and the Modern

I tried out some new social networks lately. Mastodon I quite like (it’s like what I’ve wanted Twitter to be), Discord, not so sure. So, if you’ve wondered about my reduced presence on IRC, that’s why.

Writing an IRC bot is one of the classic programming exercises that can be done in pretty much every programming language offering you some way to open TCP sockets and manipulate strings. I started doing one in Emacs Lisp long time ago, although not from scratch (rather by leveraging an existing IRC client) and wondered whether there is anything to learn from doing the equivalent with a “modern” IM platform like Discord. Could it be still be done from scratch? What else is different about it?

First I had to find a meaningful thing for the bot to do. I chose Eliza, the classic demonstration of a chatter bot that managed fooling people into having prolonged conversations with them. The version I’m familiar with is M-x doctor which is part of Emacs. So, first of all, I wrote some code to interface with that command in a REPL-style fashion. A companion shell script boots up Emacs in batch mode for interfacing with the doctor from the shell. Much like the original mode, you terminate your input by pressing RET twice. This is an intentional design decision to allow for multi-line input as seen on Discord (but absent from IRC, where you could get away with making it single-line input).

I briefly entertained the thought of writing the rest of the bot from scratch in Emacs Lisp, but abandoned it after learning that I’d need to use websockets with zlib compression to subscribe and respond to incoming messages. While there is an existing library for websocket support, I’d rather not figure out its nitty-gritty details, let alone with the lack of zlib compression. It doesn’t help that Discord’s official API docs are inconclusive and fail answering questions such as how you can set your current status (and more importantly, why it fails getting updated). So, an officially recommended Discord library it is.

The choice on which one it’s going to be depended on whether the programming language it’s implemented with allowed me to communicate with my shell script. I tried out discord.js first, battled a fair bit with Node.js, but gave up eventually. There doesn’t seem to be a way to spawn a child process and read from / write to its stdout / stdin pipes as you see fit. Instead you can only add a callback for the process output and good luck if you want to figure out what piece of output corresponds to the input you wrote earlier. This is why I went for discordrb instead, wrote some glue code for subprocess communication and started figuring out their API to react to incoming messages.

There are a few lessons to be learned from their API:

  • Allow adding as many event handlers as you want for specific events, with convenience options for narrowing down to commonly needed situations (like messages starting with a prefix)
  • Inside these event handlers, provide an object containing all context you’d need, including the information who to respond to
  • Keep the bot alive when an event handler errors out

Now, to test the bot I needed to unleash it on a server. It turns out that unlike on IRC bot accounts are handled specially. You must:

  • Register an application and obtain an ID for authorization purposes
  • Enable a bot account and obtain an authorization token
  • Generate an URL for inviting the bot to a server
  • Share that URL with someone wielding the necessary permissions
  • Hope for the best and wait for them to invite the bot

This meant that I had to create my own test server to check whether my code worked at all. For this reason I haven’t been able to get it running on the server I was invited on. If you want to, you can try it on your own server, the sources are on GitHub as always.

-1:-- Bridging the Ancient and the Modern (Post Vasilij Schneidermann)--L0--C0--March 23, 2018 08:53 AM

Bryan Murdock: It Is Time To Replace Passwords With Keys

It is time to stop using passwords. The Troy Hunt article, Passwords Evolved: Authentication Guidance for the Modern Era just got passed around the office again and wow, what a mess we are in. Instead of passwords we should switch to using public-key cryptography like TLS and ssh use.

"But people can barely manage passwords, there's no way they can manage keys!"

Wrong. People can't barely manage passwords, they can't manage them at all. Read the Troy Hunt article where he says, "Embrace Password Managers." We've given up on humans managing passwords and we are now relying on software to do it. We use software that securely stores our passwords and synchronizes them over the internet so that we have those passwords on all our devices. Often the software generates the passwords for us too. Guess what we could do with keys? The exact same thing.

Now read the Troy Hunt article again and this time think about how it's not just end-users that can't manage passwords, but how developers also cannot manage passwords. There are UI problems, there are concerns with code injection attacks, there are problems with hashing, salting, and storing passwords and with protecting those stored passwords from thieves. Those problems all go away if developers only have to store public keys. No hashing, no salting, no secrets. Think about it. As a user the only thing you'll give to a website to authenticate is your public key. You don't have to trust them with any secrets at all. From the point of view of a developer, you don't have to worry about keeping your customers secrets safe anymore. What a relief! As for UI, if we do it right websites and apps don't need any UI at all for authentication. What is the UI for TLS? The lock symbol that the browser displays. That's it! Websites you visit authenticate themselves to you using strong public-key cryptography behind the scenes, under the covers. It could be the same with user authentication.

Now read the article one more time and think about how not only are users and developers unable to deal with passwords, but security experts can't either. They can't agree on what makes a good secure password, what format it should be in. Special characters? Random strings of characters? Long passphrases of regular words? What's easier to create? Easier to type? Easier to memorize? Should a website show the password as users type it or not? How often should we change passwords? If we are giving up on memorizing and using password managers, does any of that matter? Maybe?

Now think about public-key cryptography. Security experts generally all agree on what makes good public-key cryptography, what format the keys should be in and what length. That was all hashed out years ago. True, there might be disagreement on when stronger keys should be used, whether to use RSA or ECC, and if ECC which curves to use and so on, but regular users relying on key manager software don't have to be involved in those discussions. They don't have to worry anymore about whether they should use a special character or how long of a password to use or if passwords made up of song lyrics are a good idea or not. The discussions on key size or which ECC curve to use raises the debate way beyond trying to figure out what is user friendly but still defeats rainbow tables and script kiddies. It takes the debate up to the level of wondering which nation state might attack you. If we eliminate human involvement in coming up with authentication tokens and remove script kiddies from the attack surface altogether that's a *huge* improvement over passwords.

"OK, but if we use public-key cryptography we also need to use full PKI like TLS."

Do we? PKI provides identity confirmation and key revocation. Do we have identity confirmation for account passwords today? When I create an account with, say, Amazon do they verify that I really am who I say I am? Nope. Not at all. They don't care one bit about who I really am. They just want my money. What about key revocation? Do we have password revocation today? Again, other than manually logging into a website and changing your password, we don't, and we can easily duplicate that manual process with keys in order to get us started. If we don't have full PKI with our public-key authentication we are no worse off than we are today.

The great thing about switching to public-key cryptography is that someday we could add in some sort of Let's Encrypt-like easy-to-use PKI if we want, which would take us light-years beyond where we are with passwords today. We aren't going to get there though if we don't take the first step of ditching passwords.

Getting rid of password authentication and using public-key cryptography instead will make user authentication easier for users, developers, and security experts, and it will make us all more secure.
-1:-- It Is Time To Replace Passwords With Keys (Post Bryan (noreply@blogger.com))--L0--C0--March 23, 2018 04:46 AM

Jonas Bernoulli: Ghub 2.0 and Glab 2.0 released

I am excited to announce the release of Ghub v2.0 and Glab v2.0.
-1:-- Ghub 2.0 and Glab 2.0 released (Post)--L0--C0--March 20, 2018 07:10 PM

Raimon Grau: fixing indentation of lua (busted) in emacs. A nasty hack

In general, indentation is not an issue in emacs.

But there are some exceptions.  For example, in Lua, one of the de facto testing libraries is busted, which tries to mimick rspec in many aspects.

A typical busted test looks like this:
Lua mode tends to indent code in a very lisp-y way (which I personally like) by aligning parameters to the same function using as a "starting point" the offset of the first parameter.  In this case, there's also an opened function that gets indented in addition to that base column.
This is unacceptable in most codebases, so I had to write some hack for those particular cases.

As the indentation code for lua-mode is quite complex and this is an exception to the general rule, I wrote this very ugly hack, that seems to solve my problem at hand.
As with all defadvice uses, it looks like a hack because it is a big ugly hack, but at least it lets me deal with it, and move on with my tasks without manually reindenting stuff.



Another +1 for emacs hackability :)
-1:-- fixing indentation of lua (busted) in emacs. A nasty hack (Post Raimon Grau (noreply@blogger.com))--L0--C0--March 19, 2018 11:04 AM

(or emacs: Using exclusion patterns when grepping

Git

I like Git. A lot. After years of use it has really grown on me. It's (mostly) fast, (often) reliable, and (always) distributed. For me, all properties are important, but being able to do git init to start a new project in seconds is the best feature.

When it comes to working with Git day-to-day, a nice GUI can really make a difference. In Emacs world, of course it's Magit. Outside of Emacs (brr), git-cola looks to be the most promising one. If you're aware of something better, please share - I'm keeping a list of suggestions for my non-Emacs using colleagues.

Ivy integration for Git

The main two commands in ivy that I use for Git are:

  • counsel-git: select a file tracked by Git
  • counsel-rg: grep for a line in all files tracked by Git, using ripgrep as the backend.

There are many alternatives to counsel-rg that use a different backend: counsel-git-grep, counsel-ag, counsel-ack, counsel-pt. But counsel-rg is the fastest, especially when I have to deal with Git repositories that are 2Gb in size (short explanation: it's a Perforce repo with a bunch of binaries, because why not; and I'm using git-p4 to interact with it).

Using .ignore with ripgrep

Adding an .ignore file to the root of your project can really speed up your searches. In my sample project, I went from 10k files to less than 500 files.

Example content:

/TAGS
*.min.js*
/Build/Output/
/ThirdParty/

As you can see, both file patterns and directories are supported. One other nifty thing that I discovered only recently is that you can use ripgrep as the backed for counsel-git in addition to counsel-rg. Which means the same .ignore file is used for both commands. Here's the setting:

(setq counsel-git-cmd "rg --files")

And here's my setting for counsel-rg:

(setq counsel-rg-base-command
      "rg -i -M 120 --no-heading --line-number --color never %s .")

The main difference in comparison to the default counsel-rg-base-command is -M 120 which means: truncate all lines that are longer than 120 characters. This is really helpful when Emacs is accepting input from ripgrep: a megabyte long line of minified JS is not only useless since you can't see it whole, but it will also likely hang Emacs for a while.

Outro

I hope you found these bits of info useful. Happy hacking!

-1:-- Using exclusion patterns when grepping (Post)--L0--C0--March 04, 2018 11:00 PM

Tom Tromey: Emacs JIT Calling Convention

I’ve been working a bit more on my Emacs JIT, in particular on improving function calling.  This has been a fun project so I thought I’d talk about it a bit.

Background

Under the hood, the Emacs Lisp implementation has a few different ways to call functions.  Calls to or from Lisp are dispatched depending on what is being called:

  • For an interpreted function, the arguments are bound and then the interpreter is called;
  • For a byte-compiled function using dynamic binding, the arguments are bound and then the bytecode interpreter is called;
  • For a byte-compiled function using lexical binding, an array of arguments is passed to the bytecode interpreter;
  • For a function implemented in C (called a “subr” internally), up to 8 arguments are supported directly — as in, C functions of the form f(arg,arg,...); for more than that, an array of arguments is passed and the function itself must decide which slot means what.  That is, there are exactly 10 forms of subr (actually there are 11 but the one missing from this description is used for special forms, which we don’t need to think about here).

Oh, let’s just show the definition so you can read for yourself:

union {
Lisp_Object (*a0) (void);
Lisp_Object (*a1) (Lisp_Object);
Lisp_Object (*a2) (Lisp_Object, Lisp_Object);
Lisp_Object (*a3) (Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a4) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a5) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a6) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a7) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*a8) (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
Lisp_Object (*aUNEVALLED) (Lisp_Object args);
Lisp_Object (*aMANY) (ptrdiff_t, Lisp_Object *);
} function;

Initial Approach

Initially the JIT worked like a lexically-bound bytecode function: an array of arguments was passed to the JIT-compiled function.  The JIT compiler emitted a bunch of code to decode the arguments.

For Lisp functions taking a fixed number of arguments, this wasn’t too bad — just moving values from fixed slots in the array to fixed values in the IR.

Handling optional arguments was a bit uglier, involving a series of checks and branches, so that un-bound arguments could correctly be set to nil.  These were done something like:

if nargs < 1 goto nope1
arg0 = array[0]
if nargs < 2 goto nope2
arg1 = array[1]
goto first_bytecode
nope1: arg0 = nil
nope2: arg1 = nil
first_bytecode: ...

&rest arguments were even a bit worse, requiring a call to create a list.  (This, I think, can’t be avoided without a much smarter compiler, one that would notice when reifying the list could be avoided.)

Note that calling also has to use the fully generic approach: we make a temporary array of arguments, then call a C function (Ffuncall) that does the dispatching to the callee.  This is also a source of inefficiency.

Today

Recently, I changed the JIT from this approach to use the equivalent of the subr calling convention.  Now, any function with 8 or fewer (non-&rest) arguments is simply an ordinary function of N arguments, and we let the already-existing C code deal with optional arguments.

Although this often makes the generated assembly simpler, it won’t actually perform any better — the same work is still being done, just somewhere else.  However, this approach does use a bit less memory (most JIT-compiled functions are shorter); and it opens the door to an even bigger improvement.

The Future

What I’m implementing now is an approach to removing most of the overhead from JIT-compiled function calls.

Now, ideally what I’d like is to have every call site work “like C”: move the arguments to exactly where the callee expects them to be, and then call.  However, while looking at this I found some problems that make it tricky:

  • We still need to be able to call Lisp functions from C, so we’re limited to, at best, subr-style calling conventions;
  • While &rest arguments are straightforward (in our simple compiler, somebody just has to make the list); &optional arguments don’t have a good C-like analog.  The callee could push extra arguments, but…
  • In Lisp, a function can be redefined at any time, and it is fine to change the function’s signature.

Consider this example:

(defun callee (x &optional y) (list x y))
(defun caller (callee 23))
(defun callee (x) (list x))

Now, if we compiled caller with a direct call, it would turn out like (callee 23 nil).  But then, after the redefinition, we’d have to recompile caller.  Note this can go the other way as well — we could redefine callee to have more optional arguments, or even more fixed arguments (meaning that the call should now throw an exception).

Recompiling isn’t such a big deal, right?  The compiler is set up very naively: it just compiles every function that is invoked, and in this mode “recompilation” is equivalent to “just abandon the compiled code”.

Except… what do you do if caller is being run when callee is redefined?  Whoops!

Actually, of course, this is a known issue in JIT compilation, and one possible solution is “on-stack replacement” (“OSR”) — recompiling a function while it is running.

This to me seemed like a lot of bookkeeping, though: keeping a list of which functions to compile when some function was redefined, and figuring out a decent way to implement OSR.

The Plan

Instead I came up a with a simpler approach, involving — you guessed it — indirection.

On the callee side, I am going to keep the subr calling convention that is in place today.  This isn’t ideal in all cases, but it is reasonable for a lot of code.  Instead, all the changes will take place at spots where the JIT emits a call.

I am planning to have three kinds of function calls in the JIT:

  1. Indirect.  If we see some code where we can’t determine the callee, we’ll emit a call via Ffuncall like we do today.
  2. Fully direct.  There are some functions that are implemented in C, and that I think are unreasonable to redefine.  For these, we’ll just call the C function directly.  Another fully-direct case is where the code dispatches to a byte-code vector coming from the function’s constant pool — here, there’s no possibility to redefine the function, so we can simply always call the JIT-compiled form.
  3. Semi-direct.  This will be the convention used when JIT-compiled code calls via a symbol.

The core idea of a semi-direct call is to have multiple possible implementations of a function:

  • One “true” implementation.  If the function has 8 or fewer arguments (of any kind), it will simply have that many arguments.  The JIT will simply pretend that an optional argument is fixed.  If it has more than 8 arguments, following the subr convention it will just accept an array of arguments.
  • If the function has optional or rest arguments, there will be trampoline implementations with fewer arguments, that simply supply the required number of additional arguments and then call the true implementation.
  • Remember how there are exactly 10 relevant kinds of subr?  Any forms not covered by the above can simply throw an exception.

A vector of function pointers will be attached to each symbol, and so the JIT-compiled code can simply load the function pointer from the appropriate slot (a single load — the nice thing about a JIT is we can simply hard-code the correct address).

Then, when a function is redefined, we simply define any of the trampolines that are required as well.  We won’t even need to define all of them — only the ones that some actually-existing call site has needed.

Of course, no project like this is complete without a rathole, which is why instead of doing this I’m actually working on writing a compiler pre-pass so that the compiler itself can have the appropriate information about the callee at the point of call.  This sub-project turns out to feel a lot like writing a Java bytecode verifier…

Further Future

Currently the JIT is only used for lexically-bound bytecode functions.  That’s a reasonable restriction, I think — so one thing we should do is make sure that more of the Emacs core is using lexical binding.  Currently, only about 1/3 of the Lisp files in Emacs enable this feature; but many more easily could.

Once my current project is done, the JIT will have a decent calling convention by default.  Since we’ll have information about callees at points of call, I think it will be a good time to look into inlining.  This will require tackling recompilation (and perhaps OSR) and having some sort of tiered optimization approach.  There is still a lot for me to learn here — when does it make sense to inline?  And what metrics should I use to decide when some code is hot enough to optimize?  So, good times head even once the current project is done; and BTW if you have a reading list for any of this I would love to hear about it.

Once this is done, well, I have more ideas for even deeper JIT improvements.  Those will have to wait for another post.

-1:-- Emacs JIT Calling Convention (Post tom)--L0--C0--March 03, 2018 09:53 PM

Phil Hagelberg: 186

I've enjoyed writing Lua code for my side projects for the most part. It has a few quirks, but many of them can be solved by a little linting. For a non-homoiconic language there is little in the syntax to object to, and the aggressively minimalist semantics more than make up for the few shortcomings.

Still, I always wondered what it would be like to use a language that compiled to Lua and didn't have problems like the statement vs expression distinction, lack of arity checks, or defaulting to globals. Over the past year or so I looked at several such languages1, but nothing ever stuck. Then a few weeks ago I found Fennel (at the time called "fnl"), and it really resonated with me.

The Microlanguage

snowy sidewalk

The thing that sets Fennel apart is that it is strictly about adding new syntax to Lua and keeping Lua's semantics. This allows it to operate as a compiler which introduces no runtime overhead. Code in Fennel translates trivially to its Lua equivalent:

(let [x (+ 89 5.2)
      f (fn [abc] (print (* 2 abc)))]
  (f x))

... becomes ...

do
  local x = (89 + 5.2000000000000002)
  local function _0_(abc)
      return print((2 * abc))
  end
  local f = _0_
  return f(x)
end

There are a few quirks around introducing temporary variable names in cases where it's unnecessary like the above, but these are only readability concerns and do not affect the performance of the code, since Lua is smart enough to collapse it down. The temporary locals are introduced in order to ensure that every form in Fennel has a value; there are no statements, only expressions. This fixes a common problem in Lua where you can't use an if to calculate the value of an expression, because it's implemented as a statement, so you have to construct complex and/or chains to simulate if expressions. Plus it's just simpler and more consistent to omit statements completely from the language semantics.

The one exception to the no-overhead rule is Fennel's lambda form. Fennel's fn keyword compiles straight to a no-nonsense Lua function with all that implies. But Lua's function has one feature that is quite error-prone: it doesn't check to ensure that it was called with the correct number of arguments. This leads to nil values easily getting propagated all thru the call stack by mistake. Fennel's solution to this is lambda, which includes checks to ensure that this doesn't happen. This function will signal an error when it's called with zero or one argument, but the ?w and ?h arguments are optional:

(lambda [x y ?w ?h]
  (make-thing {:x x :y y
               :width (or ?w 10) :height (or ?h 10)}))

The other main difference between Fennel and Lua is that Fennel takes more care to distinguish between sequential tables and key/value tables. Of course on the Lua runtime there is no difference; only one kind of table exists, and whether it is sequential or not is just a matter of how it's constructed or used. Lua uses {} notation for all tables, but Fennel allows you to construct sequential tables (array-like tables which have consecutive integers as keys) using [] instead. Lua overloads the for keyword to iterate over a numeric range as well as to work with generic iterators like pairs for values in tables. Fennel uses each for the latter, which makes the difference clearer at a glance.

The Compiler

To be clear, these are very small improvements over Lua. Normally I wouldn't consider switching to a new language over such things. But Fennel is unique in its simplicity and lack of overhead, so it doesn't cost much to bring it in. When I found out about Fennel, it was an experimental compiler that had been written over the course of one week in 2016 and then forgotten. But I was impressed with how useful it was after only that week of development. I could see at a glance that in around a thousand lines of code it had a functional compiler that output fairly readable Lua code and fixed the problem of statements.

So I dived into the codebase and started adding a few conveniences, starting with a test case and some static analysis. When Fennel's creator Calvin Rose saw what I was doing, he gave me feedback and started to pick back up on development too. As I got more comfortable with the code I started adding features, like each, when, and comments. Then I started putting it thru the paces by porting some of my existing Lua programs over2. This went much more smoothly than I anticipated. I did find a few compiler bugs, but they were either easy to fix myself or fixed quickly by Calvin Rose once I pointed them out. Once I had a few programs under my belt I wrote up an introductory tutorial to the language that you should read if you want to give it a try.

Tumwater falls, with raging water

But what about the lines?

But one thing really bothered me when writing Fennel programs. When you needed to debug, the Lua output was fairly readable, but you quickly ran into the curse of all source→source compilers: line numbers didn't add up. Some runtimes allow you to provide source maps which change the numbering on stack traces to match the original source code, but Lua runtimes don't offer this. If your compiler emits bytecode instead of source, you can set the line numbers for functions directly. But then you're tied to a single VM and have to sacrifice portability.

So what's a poor compiler to do? This being Fennel, the answer is "the simplest thing possible". In this case, the simplest thing is to track the line number when emitting Lua source, and only emit newlines when it's detected that the current output came from input with a line number greater than the one it's currently on.

For most compilers, this naive approach would quickly fall to pieces. Usually you can't rely on the output being in the same order as the input that generated it3. I honestly did not have high hopes for this when I started working on it. But because of Fennel's 1:1 simplicity and predictability, it actually works surprisingly well.

The future?

At this point I'm pretty happy with where the compiler is, so my own plans are mostly just to write more Fennel code. The upcoming Lisp Game Jam will be a perfect excuse to do just that. I have a few ideas for further compiler improvements, like associative destructuring (sequential destructuring already works great), pattern matching, or even making the compiler self-hosting, but there's nothing quite like getting out there and banging out some programs.


[1] Here's a list of the lisps I found that compile to Lua and my brief impression of them:

Moonlisp
An experimental lisp by the creator of Moonscript. Looks neat, but it requires an alpha build of Moonscript 0.2.0 from 2012 to run.
Hua
Inspired by Hy, this seems to be an improvement over Hy, since the latter inherits some of Python's unfortunate design bugs around scoping. If you're a Hy user, this might be a nice way to trade library availability for speed and consistency, but since the compiler is written in Hy it means you need two runtimes, which complicates deployment.
ClojureScript-Lua
This looked promising when it was first announced, but it was based on a very early version of ClojureScript that was still quirky and didn't have a repl, and was abandoned a few months after it was announced. It has the same problem of requiring a separate runtime for the compiler as Hua, except that runtime needs dramatically greater resources.
scheme.lua
This actually looks pretty nice if you like Scheme. It's pretty immature, but probably wouldn't take that much work to get to a usable state. Personally I find Lua tables to be much more friendly to work with than Scheme's lists, so sticking strictly with Scheme's semantics seems like a step backwards, but I know some people like it.
l2l
I actually did use this in my game. But since I tried it the compiler has been more or less completely rewritten. The unique thing about the new l2l compiler is that it allows you to mix Lua code and lisp code in the same file. I found it rather difficult to follow code that does this, but it's an interesting idea. The readme for l2l includes some very apt Taoist quotations, which earns it extra points in my book.
Urn
I saved the best for last. Urn is a very impressive language with a smart compiler that has great error messages, pattern matching, and tree-shaking to strip out code that's not used. The main reason I decided not to use Urn is that it wants to do a lot of optimization and analysis up-front and sacrifices some interactivity and reloading features to achieve it. As one who prizes interactivity above all other considerations, I found it a poor fit. Urn wants to be its own language that just uses Lua as a runtime, and that's great, but right now I'm looking for something that just takes Lua and makes the syntax nicer.

[2] My first program was a 1-page Pong, which is kind of the "hello world" of games. Then I ported the user-level config from Polywell, my Emacs clone, over to Fennel. This has made Polywell seem a lot more Emacsy than it used to be.

[3] Of course, once macros are introduced to the picture you can write code where this guarantee no longer applies. Oddly enough I'm not particularly interested in complex macros beyond things like pattern matching, so I don't see it being a problem for code I write, but it's worth noting that there are complications.

-1:-- 186 (Post Phil Hagelberg)--L0--C0--March 02, 2018 11:35 PM

Rafal Spacjer: Why is my Emacs slow?

Emacs is amazing text editor (let’s treat it as a fact ;) ), but it can become clunky in some situations. For example one day I was working with a big JavaScript file and after a while I’ve notice that even moving a cursor down a few line was slow (to the point where I couldn’t do my work). I’ve stared wondering what is going one. Thankfully Emacs, as usually, has a solution for that. You can run profiler that measures usage of cpu and/or memory of all functions run by Emacs. To do that we can run command (M-x):

profiler-start

then you need to choose what you want to profile. You can type:

  • mem - to profile only memory
  • cpu - to profile only usage of the processor
  • cpu+mem - to profile both memory and the processor

while the profiler is running you can display its result by invoking:

profiler-report

command.

To stop profiling you should run:

profiler-stop

The profiler report looks like this:

Emacs profiler

Every line in this report represents invoked function. On the right side of the line we can spot information about how much processor or memory was used by this function. The plus (+) sign near the function name indicate that this item can be expanded to show functions called by the upper function. You can expand such function by the enter key. By pressing enter again you can collapse the whole hierarchy. If you want to see whole call stack you can use C-u RET key stroke.

There are three additional key bindings:

  • j to go to the function’s definition
  • d to see the function’s documentation.
  • = to compare two profiling reports

Using this technique I was able to spot that the main slowness in my big JavaScript file was caused by the line numbering feature (global-linum-mode function). When I disabled it I no longer have problems with the performance.

Happy profiling!

-1:-- Why is my Emacs slow? (Post)--L0--C0--March 02, 2018 09:00 PM

Sanel Zukan: Extending org-mode Easy Templates

I frequently use org-mode Easy Templates and I noticed I often miss shortcut for comment block (BEGIN_COMMENT/END_COMMENT) which is very useful from time to time.

Let's add that feature.

(add-to-list 'org-structure-template-alist
             '("C" "#+BEGIN_COMMENT\n?\n#+END_COMMENT" ""))

C is character used to trigger completion by entering <C and pressing TAB - Emacs will then replace it with this block:

#+BEGIN_COMMENT

#+END_COMMENT

? is place where cursor will be set after expation.

Third argument in list I'm adding (empty string) is for Muse-like tags, which I'm not using at all.

Curious fact is that we are not limited to single character to trigger expansion - we can use words too:

(add-to-list 'org-structure-template-alist
             '("comment" "#+BEGIN_COMMENT\n?\n#+END_COMMENT" ""))

Entering <comment and pressing TAB will again expand it to above comment block.

-1:-- Extending org-mode Easy Templates (Post)--L0--C0--March 01, 2018 11:00 PM

Alex Schroeder: Emacs Packages

My Emacs setup is on GitHub but I might have to finally get rid of my special package handling code and move to something else instead.

How about using straight together with use-package?

One of these days!

Tags:

-1:-- Emacs Packages (Post)--L0--C0--February 27, 2018 08:37 AM

Raimon Grau: SF emacs meetup

This week I attended an emacs meetup in SF just by pure chance. I was browsing /r/emacs and there was a comment about the meetup. I found about it just the day before the event.

Being the first time I'm in San Francisco, and the fact that I'll be around just for a couple of weeks (for my new job), it makes it even more surprising that  I was able to go.

We were about 15 people, most from the bay area, and I think myself I was the only foreigner. The meetup topic was "a few of our favourite emacs modes", which unlocked the possibility to talk about helm-dash (not that it's my favourite mode, but is the one I wrote (and I also find it quite helpful)).

So I volunteered and gave a really quick intro to helm-dash.

Others talked about evil, magit, pynt, multiple-cursors (that was nuts!), git-timemachine, use-package, and probably some more that I already forgot.

My discoveries were:

- evil can easily create new text objects.
- learn to use multiple cursors (although I prefer vim's substitutions, mc work better for multiline "macros", and give you more visual feedback than emacs macros)
- pynt and emacs-ipython-notebook . If I ever do python again, I should remember that.
- use-package has more options that one usually thinks. RTFM
- ggtags is worth looking at if your language is not supported by etags/ctags.
- hydra red/blue modes.


Lots of fun during a couple of hours talking about tooling and worfklows with random techies.See you next time I'm around.
-1:-- SF emacs meetup (Post Raimon Grau (noreply@blogger.com))--L0--C0--February 26, 2018 04:23 PM