Over at What keramida said…, Giorgos Keramidas poses an interesting problem. Given the following output from dstat, get rid of the B, k, and M suffixes by converting them into bytes.

----system---- ----total-cpu-usage---- --net/eth0- -dsk/total- sda-
     time     |usr sys idl wai hiq siq| recv  send| read  writ|util
16-05 08:36:15|  2   3  96   0   0   0|  66B  178B|   0     0 |   0
16-05 08:36:16| 42  14  37   0   0   7|  92M 1268k|   0     0 |   0
16-05 08:36:17| 45  11  36   0   0   7|  76M 1135k|   0     0 |   0
16-05 08:36:18| 27  55   8   0   0  11|  67M  754k|   0    99M|79.6
16-05 08:36:19| 29  41  16   5   0  10| 113M 2079k|4096B   63M|59.6
16-05 08:36:20| 28  48  12   4   0   8|  58M  397k|   0    95M|76.0
16-05 08:36:21| 38  37  14   1   0  10| 114M 2620k|4096B   52M|23.2
16-05 08:36:22| 37  54   0   1   0   8|  76M 1506k|8192B   76M|33.6

Keramidas also wants to do a bit of reformatting of the table, which we’ll ignore for this post. He does the conversion with three invocations of replace-regexp but given that he titles his post Powerful Regular Expressions Combined with Lisp in Emacs, I thought it would be an interesting challenge to write some Elisp to get rid of those suffixes and convert the entries into bytes.

Obviously, for a one-off task, Keramidas’ solution is the best but if this is something that has to be done regularly a bit of Elisp is just what’s needed. My solution is a Lisp function that converts the table with a single call. Leave your solution in the comments and I’ll post mine in a few days.

In light of the recent leaks about the NSA's illegal spying, I've decided to go back to using M-x spook output in my email signatures.

cypherpunk anthrax John Kerry rail gun security plutonium Guantanamo wire transfer JPL number key military MD5 SRI FIPS140 Uzbekistan

Every Emacs user knows that he can split the current window horizontally (with C-x 2) and vertically (with C-x 3) as much as he desires to. However, some Emacs users don’t know what to do with the extra windows they’ve created when they do not them.

To delete the selected window, type C-x 0 (delete-window). Once a window is deleted, the space that it occupied is given to an adjacent window (but not the minibuffer window, even if that is the active window at the time). Deleting the window has no effect on the buffer it used to display; the buffer continues to exist, and you can still switch to with C-x b or any other buffer navigation command.

C-x 4 0 (kill-buffer-and-window) is a stronger (and fairly unknown) command; it kills the current buffer and then deletes the selected window (basically it combines C-x k and C-x 0). Obviously it’s a good idea to use it on windows displaying buffers you’re no longer needing.

C-x 1 (delete-other-windows) deletes all the windows, except the selected one; the selected window expands to use the whole frame. (This command cannot be used while the minibuffer window is active; attempting to do so signals an error.) In the era of narrow screens I used that command fairly often when I needed to focus on a particular task. Now I keep my screen split in half vertically 99% of the time, but I still use C-x 1 from time to time when I’m about to resplit my screen in some uncommon way.

Windows displaying help buffers (generally created with commands like C-h ...) warrant a special mention. They can be deleted with a single keystroke - q. That would delete the help window altogether if it was created by the help command, or restore its original content if the window existing beforehand and was reused by the help command.

This post continues a topic that was introduced in smarter open-line few months back.

Often when editing code one wishes to open a line just above the current one, which is properly indented relative to the existing code, and position the cursor at its beginning. Such a feature is present in most IDEs, such as IntelliJ IDEA, Eclipse and NetBeans. It’s sometimes bound to Control+Shift+Enter. Last time I showed you how to implement a similar function called smart-open-line, this time will implement smart-open-line-above. Just add this snippet to your .emacs (or .emacs.d/init.el or whatever):

1
2
3
4
5
6
7
8
9
10

(defun smart-open-line-above ()
  "Insert an empty line above the current line.
Position the cursor at it's beginning, according to the current mode."
  (interactive)
  (move-beginning-of-line nil)
  (newline-and-indent)
  (forward-line -1)
  (indent-according-to-mode))

(global-set-key [(control shift return)] 'smart-open-line-above)

Evaluate the code (or restart Emacs) and you’ll be able to use M-x smart-open-line-above or Control+Shift+Enter (aka C-S-return).

Admittedly this keybinding kind of sucks, so here’s another option for you - M-o (used by default as the prefix for some font setting commands nobody ever uses) for smart-open-line and M-O for smart-open-line-above.

1
2

(global-set-key (kbd "M-o") 'smart-open-line)
(global-set-key (kbd "M-O") 'smart-open-line-above)

Another good option would be to fold the two commands into one and use a prefix argument to trigger the opening a new line above the current one.

This command is part of Prelude(it’s named prelude-smart-open-line-above there).

Infodad posted a photo:

David recently pulled in a couple of changes to make OB-SML happy with Emacs’ package manager, SML-MODE 6.4, and Org Babel. For you reproducible research junkies, this is super-cool. It is already out on Marmalade.

I found another one (I guess I still need to order that book). In using the UVM, I have some sequences randomizing other sub sequences. I really want it to work like this (simplified, non-UVM) example:

class Foo;
   rand int bar;
   function void display();
      $display("bar: %0d", bar);
   endfunction 
endclass

class Bar;
   int bar;
   function void body();
      Foo foo = new();
      bar = 3;
      foo.display();
      assert(foo.randomize() with {bar == this.bar;});
      foo.display();
   endfunction 
endclass

module top;
   Bar bar;
   initial begin 
      bar = new();
      bar.body();
   end 
endmodule 

See the problem there? Here's what prints out when you run the above:

bar: 0
bar: -1647275392

foo.bar is not constrained to be 3 like you might expect. That's because this.bar refers to bar that is a member of class Foo, not bar that's a member of class Bar. As far as I can tell, there is no way to refer to bar that is a member of Bar in the constraint. I guess Foo could have a reference back up to Bar, but that's really awkward. Has anyone else run into this? How do you deal with it?

Shane Celis has announced that Emacsy was accepted as a Google Summer of Code project. Celis describes Emacsy as an embeddable Emacs-like library for non-text applications: Emacs OS without the text editor. The idea is to bring the Emacs way of doing things to other applications. Emacsy will be built on top of Guile Scheme, which is already embeddable in C and C++ applications.

Celis’s post gives a hint of what he’s aiming at but for the real details you should see his Google SoC application. It seems like an interesting project and I’m looking forward to seeing the results. Take a look and see if you don’t agree.

Some years ago, I came across M-x animate-birthday-present (and therefore animate-string and animate-sequence) while reading through the output of M-x apropos-command RET . RET, which lists all of the interactive commands. (Well worth exploring! The Emacs Manual also lists a few unusual things under “Amusements.”) It’s one of my favourite examples of odd things you can find in Emacs, like M-x doctor and M-x tetris. I use animate-string to create the title sequences of Emacs chats like this one with Bastien Guerry.

It turns out that lots of people use the Emacs text editor for animating things. Andrea Rossetti (from Trieste, Italy) e-mailed to share this little thing he put together to simulate typing in Emacs. And, boggle of boggles, someone even taught a course on Emacs Lisp Animations.

Next: Maybe someone can make an onion-skin animation mode to go on top of artist-mode so that we can make Emacs flipbooks?

Read the original or check out the comments on: Animating things in Emacs (Sacha Chua's blog)

commander.el is a command line parser for Emacs. It avoids messing with command-switch-alist (and friends) and instead defines the schema in an elegant API.

Example schema

Here is a (silly) example where numbers can be added and subtracted.

(require 'commander)

(defvar calc-fn nil)

(defun calc (&rest args)
  (if calc-fn
      (message "%s" (apply calc-fn (mapcar 'string-to-int args)))
    0))

(defun add ()
  (setq calc-fn '+))

(defun sub ()
  (setq calc-fn '-))

(commander
 (option "--help, -h" "Show usage information" 'commander-print-usage)
 (option "--add" "Add values" 'add)
 (option "--sub" "Subtract values" 'sub)
 (command "calc [*]" "Calculate these values" 'calc))

Example usage

Add list of values

$ carton exec emacs --script math.el -- calc 1 2 3 4 5 --add

15

Subtract list of values

$ carton exec emacs --script math.el -- calc 1 2 3 4 5 --sub

-13

Show usage information

$ carton exec emacs --script math.el -- --help

USAGE: math.el COMMAND [OPTIONS]

COMMANDS:
 calc <*>            Calculate these values

OPTIONS:
 --sub               Subtract values
 --add               Add values
 -h                  Show usage information
 --help              Show usage information

More information is available at Github: https://github.com/rejeep/commander.el

typester posted a photo:

Over the last week I have revamped my Gnus configuration. I used to look at my e-mail once a day because I felt it was distracting. I now want notification of interesting e-mail every minute.

I will detail how I have setup this feature with my new configuration.

To speed up Gnus you need to set the nnml backend as your primary select method. This is critical as setting up a news server will use the network and slow things down.

I use getmail to fetch e-mail on a maildir directory. This way I don't have to deal with file locks of an mbox file. The getmail configuration is:

[retriever]
type = SimplePOP3SSLRetriever
server = pop.gmail.com
username = ivan.kanis@googlemail.com
password = secret

[destination]
type = Maildir
path = ~/keep/Maildir/

[options]
delete = True

The variables to set in Gnus are:

(setq gnus-select-method '(nnml ""))
(setq mail-sources '((maildir :path "~/keep/Maildir")))

Before I used to do fancy splitting. It turned out to be a pain to maintain. I now use plain splitting with three inboxes: interesting, medium and boring.

(setq
 nnmail-split-methods
 '(("medium" ".*bbdb-info@lists\\.sourceforge\\.net")
   ("medium" ".*stumpwm-devel@nongnu\\.org")
   ("interesting" private-is-email-in-bbdb)
   ("interesting" "X-Ivan: interesting")
   ("boring" "")))

The function private-is-email-in-bbdb will check that the person that sent me an e-mail is in bbdb. If that's the case, the e-mail will land in the "interesting" inbox.

(defun private-is-email-in-bbdb (arg)
  (save-excursion
    (goto-line 1)
    (when (re-search-forward "^From: \\(.*\\)" nil t)
      (let ((e-mail (nth 1 (gnus-extract-address-components (match-string 1)))))
        (bbdb-search (bbdb-records) nil nil e-mail)))))

The "X-Ivan: interesting" header comes from my mail server.

I define a function that will fetch new e-mail:

(defun private-get-mail ()
  (when (get-buffer "*Group*")
    (setq ivan-var-display-message nil)
    (let (state)
      (setq state gnus-plugged)
      (gnus-agent-toggle-plugged t)
      (gnus-group-get-new-news)
      (gnus-agent-toggle-plugged gnus-plugged))
    (setq ivan-var-display-message t)))

(run-at-time t 60 'private-get-mail)

I have gmane news as my secondary select. The function unplugs Gnus so it won't check for news over the network. I think it would be too slow.

The ivan-var-display-message variable is a nasty, nasty hack to hide all the messages issued by Gnus while it fetches e-mail. I have advised the message function like so:

(defadvice message (around ivan-fun-message activate)
  "Shut up message"
  (when ivan-var-display-message
    ad-do-it))

The following function shows the e-mail indicator when there are messages in my interesting inbox (thanks to Julien Danjou):

(defun private-check-email ()
  "Return t when there are interesting e-mails"
  (when (boundp 'gnus-newsrc-alist)
    (dolist (entry gnus-newsrc-alist)
      (let ((group (car entry)))
        (when (string= group "interesting")
          (let ((unread (gnus-group-unread group)))
            (when (and (numberp unread)
                       (> unread 0))
              (return t))))))))

(setq display-time-mail-function 'private-check-email)

And now I have an indicator when there is an interesting mail to read.

Pic from ajc1 on Flikr

brew install gnutls

touch ~/.authinfo

chmod 600 ~/.authinfo

machine smtp.mail.me.com port 587 login YOURNAME@icloud.com password YOURPASSWORD

mdfind -name gnutls-cli 

Version 1.2 of Circe, the Client for IRC in Emacs, has been released.

Read more about Circe on its homepage: https://github.com/jorgenschaefer/circe/wiki

Circe is available from Marmalade.

Changes

• Channel name shortening has been improved a lot, resulting in #emacs and #emacs-circe being shortened to #e and #e-c respectively.
• /WL (who left) will now show the nicks in a single line, making the output a lot more readable.
• New CTCP queries supported: SOURCE and CLIENTINFO.
• New command: /CLEAR. This will delete all text in a chat buffer.
• Lurker handling has been improved a lot and should get confused less often now.
• Query buffers now are renamed to reflect the queried user’s nick when that users is seen to change their nick.
• Non-blocking connects are disabled by default in win32, as that doesn’t work there and caused errors.
• Added a new circe-message-handler-table which can be used instead of circe-receive-message-functions.
• Display handling was rewritten, and circe-display-handler now allows for lists in addition to functions, replacing the old circe-format-strings special case.
• The way Circe displays messages has been changed. The new list circe-message-option-functions is used to collect options. See the docstring for more information.
• Recent channel user handling has been reworked to be more consistent. In the process, circe-parted-users-timeout was renamed to circe-channel-recent-users-timeout.
• Duration strings won’t be empty anymore.
• The active channels are now shown in different places in the mode line to make more sense.
• circe-fool-face is now not bold anymore to avoid unwanted attention when combined with other faces.
• And a number of bugfixes.
• XEmacs is not supported anymore. Sorry. No one here uses it, so the code didn’t work anyhow.

Thanks go to Taylan Ulrich B, John Foerch, Pi and Donald Curtis for their contributions.

After reading this post on the MasteringEmacs blog I gave eshell a try and I like it. On this other post he shows how to implement completion with pcomplete that is automatically used by eshell.

Without further ado, here are completions for git, bzr, and hg. The git completion is basically his completion with some improvements. For example, it completes all git commands by parsing the output of git help --all.

;;**** Git Completion

(defun pcmpl-git-commands ()
  "Return the most common git commands by parsing the git output."
  (with-temp-buffer
    (call-process-shell-command "git" nil (current-buffer) nil "help" "--all")
    (goto-char 0)
    (search-forward "available git commands in")
    (let (commands)
      (while (re-search-forward
	      "^[[:blank:]]+\\([[:word:]-.]+\\)[[:blank:]]*\\([[:word:]-.]+\\)?"
	      nil t)
	(push (match-string 1) commands)
	(when (match-string 2)
	  (push (match-string 2) commands)))
      (sort commands #'string<))))

(defconst pcmpl-git-commands (pcmpl-git-commands)
  "List of `git' commands.")

(defvar pcmpl-git-ref-list-cmd "git for-each-ref refs/ --format='%(refname)'"
  "The `git' command to run to get a list of refs.")

(defun pcmpl-git-get-refs (type)
  "Return a list of `git' refs filtered by TYPE."
  (with-temp-buffer
    (insert (shell-command-to-string pcmpl-git-ref-list-cmd))
    (goto-char (point-min))
    (let (refs)
      (while (re-search-forward (concat "^refs/" type "/\\(.+\\)$") nil t)
	(push (match-string 1) refs))
      (nreverse refs))))

(defun pcmpl-git-remotes ()
  "Return a list of remote repositories."
  (split-string (shell-command-to-string "git remote")))

(defun pcomplete/git ()
  "Completion for `git'."
  ;; Completion for the command argument.
  (pcomplete-here* pcmpl-git-commands)
  (cond
   ((pcomplete-match "help" 1)
    (pcomplete-here* pcmpl-git-commands))
   ((pcomplete-match (regexp-opt '("pull" "push")) 1)
    (pcomplete-here (pcmpl-git-remotes)))
   ;; provide branch completion for the command `checkout'.
   ((pcomplete-match "checkout" 1)
    (pcomplete-here* (append (pcmpl-git-get-refs "heads")
			     (pcmpl-git-get-refs "tags"))))
   (t
    (while (pcomplete-here (pcomplete-entries))))))

;;**** Bzr Completion

(defun pcmpl-bzr-commands ()
  "Return the most common bzr commands by parsing the bzr output."
  (with-temp-buffer
    (call-process-shell-command "bzr" nil (current-buffer) nil "help" "commands")
    (goto-char 0)
    (let (commands)
      (while (re-search-forward "^\\([[:word:]-]+\\)[[:blank:]]+" nil t)
	(push (match-string 1) commands))
      (sort commands #'string<))))

(defconst pcmpl-bzr-commands (pcmpl-bzr-commands)
  "List of `bzr' commands.")

(defun pcomplete/bzr ()
  "Completion for `bzr'."
  ;; Completion for the command argument.
  (pcomplete-here* pcmpl-bzr-commands)
  (cond
   ((pcomplete-match "help" 1)
    (pcomplete-here* pcmpl-bzr-commands))
   (t
    (while (pcomplete-here (pcomplete-entries))))))

;;**** Mercurial (hg) Completion

(defun pcmpl-hg-commands ()
  "Return the most common hg commands by parsing the hg output."
  (with-temp-buffer
    (call-process-shell-command "hg" nil (current-buffer) nil "-v" "help")
    (goto-char 0)
    (search-forward "list of commands:")
    (let (commands
	  (bound (save-excursion
		   (re-search-forward "^[[:alpha:]]")
		   (forward-line 0)
		   (point))))
      (while (re-search-forward
	      "^[[:blank:]]\\([[:word:]]+\\(?:, [[:word:]]+\\)*\\)" bound t)
	(let ((match (match-string 1)))
	  (if (not (string-match "," match))
	      (push (match-string 1) commands)
	    (dolist (c (split-string match ", ?"))
	      (push c commands)))))
      (sort commands #'string<))))

(defconst pcmpl-hg-commands (pcmpl-hg-commands)
  "List of `hg' commands.")

(defun pcomplete/hg ()
  "Completion for `hg'."
  ;; Completion for the command argument.
  (pcomplete-here* pcmpl-hg-commands)
  (cond
   ((pcomplete-match "help" 1)
    (pcomplete-here* pcmpl-hg-commands))
   (t
    (while (pcomplete-here (pcomplete-entries))))))

Yesterday, the first milestone of the Havana developement branch of Ceilometer has been released and is now available for testing and download. This means the first quarter of the OpenStack Havana development has passed!

New features

Ten blueprints have been implemented as you can see on the release page. I'm going to talk through some of them here, that are the most interesting for users.

Ceilometer can now counts the scheduling attempt of instances done by nova-scheduler. This can be useful to eventually bill such information or for audit (implemented by me for eNovance).

People using the HBase backend can now do requests filtering on any of the counter fields, something we call metadata queries, and which was missing for this backend driver. Thanks to Shengjie Min (Dell) for the implementation.

Counters can now be sent over UDP instead of the Oslo RPC mechanism (AMQP based by default). This allows counter transmission to be done in a much faster way, though less reliable. The primary use case being not audit or billing, but the alarming features that we are working on (implemented by me for eNovance).

The initial alarm API has been designed and implemented, thanks to Mehdi Abaakouk (eNovance) and Angus Salkled (RedHat) who tackled this. We're now able to do CRUD actions on these.

Posting of meters via the HTTP API is now possible. This is now another conduct that can be used to publish and collector meter. Thanks to Angus Salkled (RedHat) for implementing this.

I've been working on an somewhat experimental notifier driver for Oslo notification that publishes Ceilometer counters instead of the standard notification, using the Ceilometer pipeline setup.

Sandy Walsh (Rackspace) has put in place the base needed to store raw notifications (events), with the final goal of bringing more functionnalities around these into Ceilometer.

Obviously, all of this blueprint and bugfixes wouldn't be implemented or fixed without the harden eyes of our entire team, reviewing code and advising restlessly the developers. Thanks to them!

Bug fixes

Thirty-one bugs were fixed, though most of them might not interest you so I won't elaborate too much on that. Go read the list if you are curious.

Toward Havana 2

We now have 21 blueprints targetting the Ceilometer's second Havana milestone, with some of them are already started. I'll try to make sure we'll get there without too much trouble for the 18th July 2013. Stay tuned!

I am using the / and the z keys to act as if I pressed the Alt key. Now I have the actual Alt key on my keyboard free for other uses. I have mapped it to be the Hyper key. I then use this modifier so that the keys i, j, k and l will do cursor motion.

First I set up my .xmodmap thus:

keycode 133 = Hyper_L

All keyboard are different. You can find your keycode with the program xev.

Mapping the keys in Emacs, I disable the usual keys so that I get into the habit:

(global-set-key (kbd "H-j") [left])
(global-set-key (kbd "H-k") [down])
(global-set-key (kbd "H-i") [up])
(global-set-key (kbd "H-l") [right])
(global-set-key (kbd "C-b") nil)
(global-set-key (kbd "C-f") nil)
(global-set-key (kbd "C-n") nil)
(global-set-key (kbd "C-p") nil)))

Mapping the keys in xterm with the file .Xresources:

XTerm*VT100.translations: #override \n\
 Hyper<Key>i: string("0x1b") string("[A") \n\
 Hyper<Key>k: string("0x1b") string("[B") \n\
 Hyper<Key>l: string("0x1b") string("[C") \n\
 Hyper<Key>j: string("0x1b") string("[D")

I feel that the motion keys in emacs are awkward. With this hack I will be able to use the same keys for Emacs, vim and the command line. I am not sure it's worth it, time will tell.

Babel is about letting many different languages work together. Programming languages live in blocks inside natural language Org-mode documents. A piece of data may pass from a table to a Python code block, then maybe move on to an R code block, and finally end up embedded as a value in the middle of a paragraph or possibly pass through a gnuplot code block and end up as a plot embedded in the document.

My current approach is to use multiple languages, build scripts, intermediate files to share data, and finally weaving it together inside of LaTeX. The babel way looks intriguing, with excellent support (via Emacs modes) for numerous languages. Very exciting.

• New Exporters: org-mode now uses org-element to parse org files. That's a big big improvement because that allows users to write new exporters relying on a somewhat more abstract and high level parser api than what we had before.

• orgstruct and orgstruct++ got orgstruct-heading-prefix-regexp option to set allowed prefixes and be able to fold parts of non-org files as if they where

(require 'ox-reveal)
(setq org-reveal-root "reveal.js")

(setq orgstruct-heading-prefix-regexp "^;; ")

Ever been annoyed at the lack of reindentation after using sgml-delete-tag?

;; after deleting a tag, indent properly
(defadvice sgml-delete-tag (after reindent activate)
  (indent-region (point-min) (point-max)))

Be annoyed no more!

This blogpost brought to you live from WebRebels 2013.

I already covered the awesomely commented diminish.el. Here's another trick to reduce the cruft in your modeline:

(defmacro rename-modeline (package-name mode new-name)
  `(eval-after-load ,package-name
     '(defadvice ,mode (after rename-modeline activate)
        (setq mode-name ,new-name))))

(rename-modeline "js2-mode" js2-mode "JS2")
(rename-modeline "clojure-mode" clojure-mode "Clj")

With this, I reduce the js2-mode modeline lighter from "JavaScript IDE" to just "JS2".

I stole it from Bodil's .emacs.d and macroified it a little. The first argument is the package name, the second is the mode in question, and the third is the new lighter for the mode.

I’ve picked up an interest in JavaScript and HTML5 this year, and have gone to a bunch of great technical talks in Boston. I brought a camera with me and recorded some of them, so you can see them too if you like. Here they are:

Rick Waldron – The Future of JavaScript

Mike Pennisi – Stress Testing Realtime Node.js Apps

Paul Irish – The Mobile Web Is In Deep Trouble

Daniel Rinehart – Debugging Node.js Applications

Ian Johnson – Prototyping data visualizations in d3.js

Kenneth Reitz – Heroku 101

I think these are world-class talks. But if I hadn’t brought my little camera with me and recorded them, they would be destroyed. No-one else offered to record them, even though they were popular — the Paul Irish talk had 110 people signed up to attend, and more than the same number again waitlisted who couldn’t go because they wouldn’t fit in the room. So there were more people in Boston who didn’t get to see the talk (but wanted to) than who did, even before we start counting the rest of the world’s interest in technical talks.

I’m happy that I’m able to help disseminate knowledge from Boston, which has an abundance of incredibly smart people living here or visiting, to wherever in the world you’re reading from now. But I’m also sad, because there are far more talks that I don’t go to here, and I expect most of those aren’t being recorded.

We’re technologists, right? So this should be easy. It’s not like I went to video camera school:

• The equipment I’m using (Panasonic Lumix G2 camera and Lumix 20mm f/1.7 lens) costs under USD $800. Maybe it could be cheaper; maybe a recent cellphone (HTC One or Galaxy S4?) would be adequate.
• I use a $20 tripod which is half broken.
• I don’t use an external audio recorder (just the camera’s microphone) so the audio is noisier than it could be.
• My camera’s sensor is small so it doesn’t have great low-light performance, and it records 720p instead of 1080p.
• Sometimes the refresh rate/frequency of the projector is out of sync with the camera and there are strobing colors going across the screen in the final video. I don’t think I can do anything about this on the camera’s side?
• I don’t do any editing because I don’t have time; I just upload the raw video file to YouTube and use YouTube’s “crop” feature to trim the start and end, that’s it.

I’d really like to know what the right answer is here. Am I overestimating how important it is to record these, and how privileged I am to be somewhere where there’s an interesting talk happening almost every day? Is owning a device that can record HD video for around 90 mins rare, even amongst well-paid developers and designers? If the presenter just recorded a screencast of their laptop with audio from its microphone, is that good enough or is that too boring for a full-length talk?

Might part of the problem be that people don’t know how to find videos of technical talks (I don’t know how anyone would find these unless they were randomly searching YouTube) so there isn’t as much demand as there should be — is there a popular website for announcing new recordings of tech talks somewhere? Maybe I just need to write up a document that describes how to record talks with a minimum of hassle and make sure people see it? Do we need to make a way for someone to signify their interest in having an upcoming talk be recorded, so that a team of volunteer videographers could offer to help with that?

In this chat, Bastien tells stories about getting started in Emacs, reading his mail/news/blogs in Gnus, and hacking his life with Org. =) Enjoy!

Want just the audio? You can get MP3s or OGG from archive.org.

Read the original or check out the comments on: Emacs Chat: Bastien Guerry (Sacha Chua's blog)

WebRTC is incredibly exciting, and is starting to see significant deployment: it’s available by default in Chrome and Firefox releases now. Most people think of WebRTC as an API for video calling, but there’s a general purpose method for directly sharing data between web browsers (even when they’re behind NAT) in there if you look harder. For example:

• P does peer-to-peer mesh networking in JavaScript.
• TowTruck allows you to add collaboration features (collaborative text editing, text chat, voice chat) to websites.
• PeerCDN forms a network from a site’s visitors, and uses it to offload serving up static content away from the web server and on to the networked peers.
• The Tor Project is interested in using WebRTC to enable volunteers with JavaScript-enabled web browsers to become on-ramps onto the Tor network for users under censorship, as part of the Flash Proxies project. The idea is that censoring organizations may block the public Tor relays directly, but they can’t easily block every random web browser who might route traffic for those relays over WebRTC, especially if each web browser’s proxy is short-lived.

All of this activity means that we might finally be close to solving — amongst other important world problems — the scourge of xkcd.com/949:

xkcd: File Transfer, used under CC-BY-NC 2.5.

I wanted to experiment with WebRTC and understand its datachannels better, and I also felt like the existing code examples I’ve seen are unsatisfying in a specific way: it’s a peer-to-peer protocol, but the first thing you do (for example, on sites like conversat.io) is have everyone go to the same web server to find each other (this is called “signaling” in WebRTC) and share connection information.

If we’re going to have a peer-to-peer protocol, can’t we use it without all visiting the same centralized website first? Could we instead make a WebRTC app that just runs out of a file:/// path on your local disk, even if it means you have to manually tell the person you’re trying to talk to how to connect to you?

It turns out that we can: I’ve created a serverless-webrtc project on GitHub that decouples the “signaling server” exchange of connection information from the WebRTC code itself. To run the app:

• download Firefox Nightly.
• git clone git://github.com/cjb/serverless-webrtc.git
• load file:///path/to/serverless-webrtc/serverless-webrtc.html

You’ll be asked whether you want to create or join a channel, and then you’re prompted to manually send the first party’s “WebRTC offer” to the second party (for example, over an instant message chat) and then to do the same thing with the second party’s “WebRTC answer” reply back. Once you’ve done that, the app provides text chat and file transfer between peers, all without any web server. (A STUN server is still used to find out your external IP for NAT-busting.)

There are open issues that I’d be particularly happy to receive pull requests for:

#1: The code doesn’t work on Chrome yet. Chrome is behind Firefox as far as DataChannels are concerned — Chrome doesn’t yet have support for binary transfers, or for “reliable” (TCP, not UDP) channels (Firefox does). These are both important for file transfers.

#2: Large file transfers often fail, or even hang the browser, but small transfers seem to work every time. I’m not sure whose code is at fault yet.

#3: File transfers should have a progress bar.

Thanks for reading this far! Here’s to the shared promise of actually being able to use the Internet to directly share files with each other some time soon.

Regular expressions are a powerful text transformation tool. Any UNIX geek will tell you that. It’s so deeply ingrained into our culture, that we even make jokes about it. Another thing that we also love is having a powerful extension language at hand, and Lisp is one of the most powerful extension languages around (and of course, we make jokes about that too).

Emacs, one of the most famous Lisp applications today, has for a while now the ability to combine both of these, to reach entirely new levels of usefulness. Combining regular expressions and Lisp can do really magical things.

An example that I recently used a few times is parsing & de-humanizing numbers in dstat output. The output of dstat includes numbers that are printed with a suffix, like ‘B’ for bytes, ‘k’ for kilobytes and ‘M’ for megabytes, e.g.:

----system---- ----total-cpu-usage---- --net/eth0- -dsk/total- sda-
     time     |usr sys idl wai hiq siq| recv  send| read  writ|util
16-05 08:36:15|  2   3  96   0   0   0|  66B  178B|   0     0 |   0
16-05 08:36:16| 42  14  37   0   0   7|  92M 1268k|   0     0 |   0
16-05 08:36:17| 45  11  36   0   0   7|  76M 1135k|   0     0 |   0
16-05 08:36:18| 27  55   8   0   0  11|  67M  754k|   0    99M|79.6
16-05 08:36:19| 29  41  16   5   0  10| 113M 2079k|4096B   63M|59.6
16-05 08:36:20| 28  48  12   4   0   8|  58M  397k|   0    95M|76.0
16-05 08:36:21| 38  37  14   1   0  10| 114M 2620k|4096B   52M|23.2
16-05 08:36:22| 37  54   0   1   0   8|  76M 1506k|8192B   76M|33.6

So if you want to graph one of the columns, it’s useful to convert all the numbers in the same unit. Bytes would be nice in this case.

Separating all columns with ‘|’ characters is a good start, so you can use e.g. a CSV-capable graphing tool, or even simple awk scripts to extract a specific column. ‘C-x r t’ can do that in Emacs, and you end up with something like this:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66B| 178B|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  92M|1268k|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  76M|1135k|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  67M| 754k|   0 |  99M|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 113M|2079k|4096B|  63M|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  58M| 397k|   0 |  95M|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 114M|2620k|4096B|  52M|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  76M|1506k|8192B|  76M|33.6|

The leading and trailing ‘|’ characters are there so we can later use orgtbl-mode, an awesome table editing and realignment tool of Emacs. Now to the really magical step: regular expressions and lisp working together.

What we would like to do is convert text like “408B” to just “408″, text like “1268k” to the value of (1268 * 1024), and finally text like “67M” to the value of (67 * 1024 * 1024). The first part is easy:

M-x replace-regexp RET \([0-9]+\)B RET \1 RET

This should just strip the “B” suffix from byte values.

For the kilobyte and megabyte values what we would like is to be able to evaluate an arithmetic expression that involves \1. Something like “replace \1 with the value of (expression \1)“. This is possible in Emacs by prefixing the substitution pattern with \,. This instructs Emacs to evaluate the rest of the substitution pattern as a Lisp expression, and use its string representation as the “real” substitution text.

So if we match all numeric values that are suffixed by ‘k’, we can use (string-to-number \1) to convert the matching digits to an integer, multiply by 1024 and insert the resulting value by using the following substitution pattern:

\,(* 1024 (string-to-number \1))

The full Emacs command would then become:

M-x replace-regexp RET \([0-9]+\)k RET \,(* 1024 (string-to-number \1)) RET

This, and the byte suffix removal, yield now the following text in our Emacs buffer:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66| 178|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  92M|1298432|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  76M|1162240|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  67M| 772096|   0 |  99M|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 113M|2128896|4096|  63M|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  58M| 406528|   0 |  95M|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 114M|2682880|4096|  52M|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  76M|1542144|8192|  76M|33.6|

Note: Some of the columns are indeed not aligned very well. We’ll fix that later. On to the megabyte conversion:

M-x replace-regexp RET \([0-9]+\)M RET \,(* 1024 1024 (string-to-number \1)) RET

Which produces a version that has no suffixes at all:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66| 178|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  96468992|1298432|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  79691776|1162240|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  70254592| 772096|   0 |  103809024|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 118489088|2128896|4096|  66060288|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  60817408| 406528|   0 |  99614720|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 119537664|2682880|4096|  54525952|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  79691776|1542144|8192|  79691776|33.6|

Finally, to align everything in neat, pipe-separated columns, we enable M-x orgtbl-mode, and type “C-c C-c” with the pointer somewhere inside the transformed dstat output. The buffer now becomes something usable for pretty-much any graphing tool out there:

| time           | cpu | cpu | cpu | cpu | cpu | cpu |      eth0 |    eth0 |  disk |      disk | sda- |
| time           | usr | sys | idl | wai | hiq | siq |      recv |    send |  read |      writ | util |
| 16-05 08:36:15 |   2 |   3 |  96 |   0 |   0 |   0 |        66 |     178 |     0 |         0 |    0 |
| 16-05 08:36:16 |  42 |  14 |  37 |   0 |   0 |   7 |  96468992 | 1298432 |     0 |         0 |    0 |
| 16-05 08:36:17 |  45 |  11 |  36 |   0 |   0 |   7 |  79691776 | 1162240 |     0 |         0 |    0 |
| 16-05 08:36:18 |  27 |  55 |   8 |   0 |   0 |  11 |  70254592 |  772096 |     0 | 103809024 | 79.6 |
| 16-05 08:36:19 |  29 |  41 |  16 |   5 |   0 |  10 | 118489088 | 2128896 |  4096 |  66060288 | 59.6 |
| 16-05 08:36:20 |  28 |  48 |  12 |   4 |   0 |   8 |  60817408 |  406528 |     0 |  99614720 | 76.0 |
| 16-05 08:36:21 |  38 |  37 |  14 |   1 |   0 |  10 | 119537664 | 2682880 |  4096 |  54525952 | 23.2 |
| 16-05 08:36:22 |  37 |  54 |   0 |   1 |   0 |   8 |  79691776 | 1542144 |  8192 |  79691776 | 33.6 |

The trick of combining arbitrary Lisp expressions with regexp substitution patterns like \1, \2 … \9 is something I have found immensely useful in Emacs. Now that you know how it works, I hope you can find even more amusing use-cases for it.

Update: The Emacs manual has a few more useful examples of \, in action, as pointed out by tunixman on Twitter.

Recently on the Clojure mailing list someone started an interesting thread on what motivates you as a programmer. My friend Nelson Morris responded as so:

Contributions and projects start off well, and energy might wane depending on time and life factors. Even contributing to tools used by many of the members of the community like [Leiningen] and Clojars doesn't prevent it. What helps is direct involvement by someone else.

This really resonated with me because it emphasizes that people are more important than programs. For me sharing is the thing that makes programming even worth doing in the first place. So it got me thinking about different technologies and what kind of people they're good for helping.

If you follow my writing it will be obvious that I enjoy working in Emacs and Clojure. While these are among the most powerful, flexible technologies I know of, collaborating with others on tools for Emacs and Clojure basically limits me to working with professional programmers, because both environments are very poor from a beginner's perspective. If I'm working solo or on a team of seasoned hackers, I'll definitely be most effective with Clojure. If my primary goal is to interact with the widest group of programmers possible, I would use Ruby as it's the most commonly-used language I can bring myself to use. But if I want to reach out to people who don't already spend all day thinking about functions and data structures, well that's another thing entirely.

This is particularly relevant for me personally as a father. I'm taking an active role in the education of my sons, and of course I think technical literacy must be an important part of it. But when you look at how computers used in traditional educational settings, you're much more likely to see computers programming children than children programming computers. So I've been looking for ways to foster technical skills and encourage algorithmic thinking in engaging ways that can keep the attention of my five-year-old.

In the book Mindstorms, Seymour Papert describes the shift from concrete reasoning to formal reasoning as one of the main transitions children undergo as they learn to think like adults. One of the design goals of the Logo system he created was to provide transitional concepts to bridge the gap between the two.

Children interact with Logo by giving commands to an onscreen object known as the turtle. While the turtle lives in the abstract world of geometry comprised of points and lines, children are able to identify with it since they tell it to move in ways which they can relate to—it has a heading and position, and it turns and moves forward and backwards just like they do. Because the turtle's movements on the screen are isomorphic to their own physical movements, it gives them a model to help them grasp abstract geometrical concepts though they're only used to thinking in concrete terms. And Piagetian learning—ambient, natural learning which children are so adept at doing without study—is all a matter of building models of the world.

Most people know Logo from its original triangle-turtle-centric incarnation, but in Mindstorms Papert describes Logo as more of an educational philosophy than any single program, language, or implementation. A more recent version of Logo is Scratch, a drag-and-drop visual programming environment from MIT's Media Lab targeting school children. Since my older son is an early reader he's been able to construct simple scripts (with some guidance) for the characters within Scratch, watching them interact with each other and even in some cases the outside world.

While it's been lots of fun to come up with ideas and talk through how we'd bring them to life on the screen, one of the most rewarding parts is watching his problem-solving abilities develop. Papert talks about how children are often afraid to try things for fear of failure, but Scratch teaches that debugging is a normal part of making things work. Rather than "does it work", the question becomes "how can we make it work?" This was demonstrated the other day (outside the context of Scratch) when he was putting together some Snap Circuits:

Of course the goal is not to produce "little programmers". It's primarily about developing the ability to think systematically, but it extends beyond that into getting them thinking about thinking itself. In some sense once they're in the habit of asking the right epistemological questions, the parent or teacher almost just needs to get out of the way and let them explore. At that point the process of discovering a topic with someone is much more rewarding than telling facts at them.

One of my recent innocent tweet about Gerrit vs Github triggered much more reponses and debate that I expected it to. I realize that it might be worth explaining a bit what I meant, in a text longer than 140 characters.

I'm having a hard time now contributing to projects not using Gerrit. Github isn't that good.

— Julien Danjou (@juldanjou) May 8, 2013

The problems with Github pull-requests

I always looked at Github from a distant eye, mainly because I always disliked their pull-request handling, and saw no value in the social hype it brings. Why?

One click away isn't one click effort

The pull-request system looks like an incredible easy way to contribute to any project hosted on Github. You're a click away to send your contribution to any software. But the problem is that any worthy contribution isn't an effort of a single click.

Doing any proper and useful contribution to a software is never done right the first time. There's a dance you will have to play. A slowly rhythmed back and forth between you and the software maintainer or team. You'll have to dance it until your contribution is correct and can be merged.

But as a software maintainer, not everybody is going to follow you on this choregraphy, and you'll end up with pull-request you'll never get finished unless you wrap things up yourself. So the gain in pull-requests here, isn't really bigger than a good bug report in most cases.

This is where the social argument of Github isn't anymore. As soon as you're talking about projects bigger than a color theme for your favorite text editor, this feature is overrated.

Contribution rework

If you're lucky enough, your contributor will play along and follow you on this pull-request review process. You'll make suggestions, he will listen and will modify his pull-request to follow your advice.

At this point, there's two technics he can use to please you.

Technic #1: the Topping

Github's pull-requests invite you to send an entire branch, eclipsing the fact that it is composed of several commits. The problem is that a lot of one-click-away contributors do not masterize Git and/or do not make efforts to build a logical patchset, and nothing warns them that their branch history is wrong. So they tend to change stuff around, commit, make a mistake, commit, fix this mistake, commit, etc. This kind of branch is composed of the whole brain's construction process of your contributor, and is a real pain to review. To the point I quite often give up.

A typical case: 3 commits to build a 4 lines long file.

Without Github, the old method that all software used, and that many software still use (e.g. Linux), is to send a patch set over e-mail (or any other medium like Gerrit). This method has one positive effect, that it forces the contributor to acknowledge the list of commits he is going to publish. So, if the contributor he has fixup commits in his history, they are going to be seen as first class citizen. And nobody is going to want to see that, neither your contributor, nor the software maintainers. Therefore, such a system tend to push contributors to write atomic, logical and self-contained patchset that can be more easily reviewed.

Technic #2: the History Rewriter

This is actually the good way to build a working and logical patchset using Git. Rewriting history and amending problematic patches using the famous git rebase --interactive trick.

The problem is that if your contributor does this and then repush the branch composing your pull-request to Github, you will both lose the previous review done, each time. There's no history on the different versions of the branch that has been pushed.

In the old alternative system like e-mail, no information is lost when reworked patches are resent, obviously. This is far better because it eases the following of the iterative discussions that the patch triggered.

Of course, it would be possible for Github to enhance this and fix it, but currently it doesn't handle this use case correctly..

Exercise for the doubtful readers: good luck finding all revisions of my patch in the pull-request #157 of Hy.

A quick look at OpenStack workflow

It's not a secret for anyone that I've been contributing to OpenStack as a daily routine for the last 18 months. The more I contribute, the more I like the contribution workflow and process. It's already well and longly described on the wiki, so I'll summarize here my view and what I like about it.

Gerrit

To send a contribution to any OpenStack project, you need to pass via Gerrit. This is way simpler than doing a pull-request on Github actually, all you have to do is do your commit(s), and type git review. That's it. Your patch will be pushed to Gerrit and available for review.

Gerrit allows other developers to review your patch, add comments anywhere on it, and score your patch up or down. You can build any rule you want for the score needed for a patch to be merged; OpenStack requires one positive scoring from two core developers before the patch is merged.

Until a patch is validated, it can be reworked and amended locally using Git, and then resent using git review again. That simple. The historic and the different version of the patches are available, with the whole comments. Gerrit doesn't lose any historic information on your workflow.

Finally, you'll notice that this is actually the same kind of workflow projects use when they work by patch sent over e-mail. Gerrit just build a single place to regroup and keep track of patchsets, which is really handy. It's also much easier for people to actually send patch using a command line tool than their MUA or git send-email.

Gate testing

Testing is mandatory for any patch sent to OpenStack. Unit tests and functionnals test are run for each version of each patch of the patchset sent. And until your patch passes all tests, it will be impossible to merge it. Yes, this implies that all patches in a patchset must be working commits and can be merged on their own, without the entire patchset going in! With such a restricution, it's impossible to have "fixup commits" merged in your project and pollute the history and the testability of the project.

Once your patch is validated by core developers, the system checks that there is not any merge conflicts. If there's not, tests are re-run, since the branch you are pushing to might have changed, and if everything's fine, the patch is merged.

This is an uncredible force for the quality of the project. This implies that no broken patchset can ever sneak in, and that the project pass always all tests.

Conclusion: accessibility vs code review

In the end, I think that one of the key of any development process, which is code review, is not well covered by Github pull-request system. It is, along with history integrity, damaged by the goal of making contributions easier.

Choosing between these features is probably a trade-off that each project should do carefully, considering what are its core goals and the quality of code it want to reach.

I tend to find that OpenStack found one of the best trade-off available using Gerrit and plugging testing automation via Jenkins on it, and I would probably recommend it for any project taking seriously code reviews and testing.

This week, I re-implemented my main website. The text on the old one needed to be revised heavily (in fact I removed almost all of it except a short bio and some longer articles) and I wanted to experiment with a more colorful look. I need to revise the stylesheet a bit more to give a better experience on small screens (mobile), but so far I’m pleased with the result.

The new site html is generated completely by emacs’ org-mode project publishing functionality (the old one used a custom bunch of perl scripts generating from HTML snippets and perl POD documents). This hopefully means it’ll be easier to add and revise content, and it also provides nice syntax highlighting for any code snippets I put in my articles.

The whole switchover was pretty smooth, especially once I wrote a bit of elisp to roughly prepare conversion from POD formatting to org:


(defun pod2org
(point mark)
"Rougly convert region from pod syntax to org-mode"
(interactive "r")
(replace-regexp "C<\\([^>]+\\)>" "=\\1=" nil point mark)
(replace-regexp "I<\\([^>]+\\)>" "/\\1/" nil point mark)
(replace-regexp "B<\\([^>]+\\)>" "*\\1*" nil point mark)
(replace-regexp "^=head1 \\(.*\\)" "* \\1" nil point mark)
(replace-regexp "^=head2 \\(.*\\)" "** \\1" nil point mark)
(replace-regexp "^=head3 \\(.*\\)" "*** \\1" nil point mark)
(replace-regexp "^=head4 \\(.*\\)" "**** \\1" nil point mark)
(replace-regexp "^=head5 \\(.*\\)" "***** \\1" nil point mark))


Very basic, but pretty useful.

Now, I’m considering replacing WordPress for this blog with org-mode too. That probably requires a bit more investigation. I want to keep at least the tags/categories feature and related rss feeds, and I’m not sure if that’s available for org at the moment.