Irreal: org-goto

I learned a really useful Org mode command from Artur Malabarba's first anniversary post: org-goto. The idea is that you want to navigate to somewhere else in the current Org buffer. You could fold the buffer and navigate to the proper subtree but often you want to leave the current subtree unfolded as well as the subtree you're navigating to. The org-goto command, bound to 【Ctrl+c Ctrl+j】 by default, allows you to do just that. An copy of the buffer is created with an overlay displaying the buffer in overview mode. You can navigate within that buffer and press 【Return】 on the desired subtree which will take you back to the original buffer with the point at the new subtree.

That may not seem too useful but see Malabarba's post for a compelling use case. In my case, it's useful because of the large Org files I use to store my data. When I load such a file via a bookmark or it gets reloaded during a sync from git, it will be completely folded. The files are too large to set them to be completely unfolded so I have to have some way of finding the proper place. I used to call org-sparse-tree and then do a regular expression search for the proper heading. With org-goto, I can simply display the headers, navigate to the desired one, and press 【Return】 to navigate to the proper heading that will be conveniently unfolded. Very handy.

It's a bit hard to explain how org-goto works so you should experiment with it a bit to see how you can fold it into your workflow. If you use Org files to organize your data as I do, this command is very likely to be a win for you.

-1:-- org-goto (Post jcs)--L0--C0--July 01, 2015 01:43 PM

sachachua: Emacs Hangout June 2015

Times may be off by a little bit, sorry!

Boo, I accidentally browsed in the Hangouts window before copying the text chat, so no copy of the text chat this time… =|

The post Emacs Hangout June 2015 appeared first on sacha chua :: living an awesome life.

-1:-- Emacs Hangout June 2015 (Post Sacha Chua)--L0--C0--June 30, 2015 07:50 PM

Irreal: Making My Org Agenda More Useful

I've written before (1, 2, 3) about my efforts to implement a system like the one Karl Voit has for managing personal data. Voit uses his Memacs system to automatically collect data from a variety of sources and add them to Org files so that they will appear in his Org agenda. Unfortunately, Memacs was written with the Linux/Android ecosystem in mind so much of it needs a bit work to be usable in the OS X/iOS world.

Nevertheless, many of the ideas (and probably some of the scripts) from Memacs can be used with Org mode on any platform. I keep most of my personal data in 5 Org files: journal, chart (a lightweight quantified self), b-ideas (a queue of ideas and notes for blog posts), todo, and notes. I have capture templates for each of these making it easy to add data. My latest effort to make this data more accessible and useful was to modify the capture templates to add an active time stamp to each entry, causing it to appear in that day's agenda. For example, here is the capture template for my journal:

("j" "Journal" entry (file+datetree "~/org/journal.org")
         "* %<%R: >%? %^g\n%t")

The %t at the end causes an active date stamp to be added to each journal entry. Thereafter, the headline and tags from the entry will appear in the agenda and selecting that headline will take me directly to the entry.

Here's an example of one day's agenda. If I want to know what happened on that day, I can jump directly to it in my agenda to find out. If I want to see where I've had dinner, I can bring up all the entries with a tag of dinner. It's a tremendously powerful system that comes almost for free just by arranging for those active date stamps.

day-agenda.jpg

It could be better though. That's where Memacs comes in. It automatically extracts data from SMS, the Phone's log, posts, emails, and other data sources. All of the data magically appears in the agenda without having to explicitly capture it. My next step is to adapt some of Voit's Memacs scripts for my own use.

All this seems pretty geeky but I use the system every day for practical tasks. This morning, for example, I used it to extract data for my tax files. If, as often happens, I find I need to make an update to a blog post, there's a link to the source file right in the agenda and clicking on it brings up the proper file. As you can see in the example, I also add technical items to my journal. If I need to export an Org file containing citations to docx and don't remember how, I can search for the docx tag to find the journal entry that contains the details.

Org is tremendously powerful and it pays huge dividends for the effort you expend learning it. One of the nice things about Org is you don't have to learn everything at once. I keep learning new bits as I find a need for them.

-1:-- Making My Org Agenda More Useful (Post jcs)--L0--C0--June 30, 2015 02:16 PM

Pragmatic Emacs: Run shell command on buffer or region

Emacs has tonnes of features built in, and is infinitely expandable with emacs lisp, but sometimes there is already a shell command that can do the job you want more easily. You can use M-! (wich runs shell-command) to run a shell command with the contents of the current buffer passed to stdin (standard input for the command), or M-| (which runs shell-command-on-region) to do the same for the currently selected text in the buffer.

The output of the shell command is then shown as a new buffer in emacs.

As a trivial example, suppose a buffer contained the following, and we wanted to sort the lines (N.B. this is easy to do in emacs)

bbb
aaa
ddd
ccc

Hitting M-! prompts for a shell command, and we then enter sort to call the standard unix/linux sort command. We now get a buffer called *Shell Command Output* containing

aaa
bbb
ccc
ddd

Since I am not the world’s greatest lisp programmer, I use this quite often for things that could probably be accomplished in emacs. For example, I wanted to pull out all of the xmlUrl addresses from a large file that looked like this

<opml version="1.0">
    <head>
        <title>Ben subscriptions in feedly Cloud</title>
    </head>
    <body>
        <outline text="daily" title="daily">
            <outline type="rss" text="Information Is Beautiful" title="Information Is Beautiful" xmlUrl="http://feeds.feedburner.com/InformationIsBeautiful" htmlUrl="http://www.informationisbeautiful.net"/>
            <outline type="rss" text="Planet Emacsen" title="Planet Emacsen" xmlUrl="http://planet.emacsen.org/atom.xml" htmlUrl="http://planet.emacsen.org/"/>
            <outline type="rss" text="What If?" title="What If?" xmlUrl="http://what-if.xkcd.com/feed.atom"/>
        </outline>
    </body>
</opml>

I couldn’t come up with a way to do this internally in emacs, but it is easy for me in perl, so I just ran M-! and then entered

perl -ne '/xmlUrl=(".+?")/; print "$1\n"' | uniq

to get a buffer containing just

"http://feeds.feedburner.com/InformationIsBeautiful"
"http://planet.emacsen.org/atom.xml"
"http://what-if.xkcd.com/feed.atom"

I’d be curious if anyone has a nice way to do this in emacs alone!

-1:-- Run shell command on buffer or region (Post Ben Maughan)--L0--C0--June 30, 2015 12:46 PM

Ryan Rix: KDE ActivityManager in Emacs

Today I whipped up a small Emacs minor-mode to interface with KDE's ActivityManager system. It's my first minor-mode and it's janky as fuck right now, but I'm going to expand on it to eventually be able to filter, for example, to just buffers that are linked to your current activity, pushing me towards a long-standing goal of mine to create a system which *flows* with what I'm doing, rather than forcing me in to its workflow.
-1:-- KDE ActivityManager in Emacs (Post)--L0--C0--June 30, 2015 12:00 AM

(or emacs: Context aware hydra

In this post, I'll demonstrate a snippet that came up this week in a conversation over John Kitchin's post. While the following snippet isn't inherently useful, it might help a creative user to understand Hydra a bit more and accomplish more advanced things.

This hydra demonstrates that the hint and even the key bindings aren't set in stone: you can modify them depending on the current state of your Emacs, in this case the current line number.

(defhydra hydra-vi (:hint nil)
  "vi"
  ("j" next-line)
  ("k" previous-line)
  ("n" next-line)
  ("p" previous-line))

(setq hydra-vi/hint
  '(if (evenp (line-number-at-pos))
    (prog1 (eval
            (hydra--format nil '(nil nil :hint nil)
                           "\neven: _j_ _k_\n" hydra-vi/heads))
      (define-key hydra-vi/keymap "n" nil)
      (define-key hydra-vi/keymap "p" nil)
      (define-key hydra-vi/keymap "j" 'hydra-vi/next-line)
      (define-key hydra-vi/keymap "k" 'hydra-vi/previous-line))
    (prog1 (eval
            (hydra--format nil '(nil nil :hint nil)
                           "\nodd: _n_ _p_\n" hydra-vi/heads))
      (define-key hydra-vi/keymap "j" nil)
      (define-key hydra-vi/keymap "k" nil)
      (define-key hydra-vi/keymap "n" 'hydra-vi/next-line)
      (define-key hydra-vi/keymap "p" 'hydra-vi/previous-line))))

The first statement is one of the most elementary defhydra calls. The only extra thing is that it sets the :hint to nil.

The defhydra statement generates a bunch of function and variable definitions. You can examine them closely by evaluating:

(macroexpand
 '(defhydra hydra-vi (:hint nil)
   "vi"
   ("j" next-line)
   ("k" previous-line)
   ("n" next-line)
   ("p" previous-line)))

Just paste that code into *scratch*, and press C-j. If you want pretty output (153 lines of code instead of 26), turn on lispy-mode and press E instead of C-j.

Anyway, among these defined variables is hydra-vi/hint which is evaluated each time to display the hint. So now we can just redefine hydra-vi/hint to make it so that on even lines n calls next-line, while on odd lines it's j, with the appropriate changes in the doc. The change in bindings, modifying hydra-vi/keymap - also one of the defined variables, needs to be a side-effect, since hydra-vi/hint is expected to evaluate to a string.

Just to give you some idea of how it could be used: you can have a context-aware "open" command that, for instance, delegates to open-in-pdf-tools or open-in-firefox or open-in-emacs when it detects that the point is on a link. And of course all these commands would have their own key binding that works only if the command makes sense.

This approach is described on the wiki, in case you read this post much later and want to see an up-to-date code, or even update it yourself. In case something cool comes out of this snippet, I can try to implement a more palatable API for defhydra, most likely an option to supply a function name in the docstring argument position.

-1:-- Context aware hydra (Post)--L0--C0--June 29, 2015 10:00 PM

Endless Parentheses: Fine-tuning subtree visibility in org-mode

Org is one of those packages that you can use for a lifetime and still not know all of its features. One of the first things you learn is how to use the #+STARTUP header to define the initial visibility of headlines when you first open an org-mode file. But did you know you can also use that on a per headline basis?

Besides the usual 4 attributes you can chose from for the header (overview, content, showall and showeverything), you can also assign similar options to the VISIBILITY property of any headline. The specific options are folded, children, content, and all, and their names are fairly self-explanatory.

As an example, remember this screenshot of my init.org from the birthday post?

init-org-1.png

To achieve that, you just had to set the * init.el headline to have :VISIBILITY: children, along with the usual startup header you see in the image. I was almost implementing this myself, when I decided to Google first and wasn’t surprised to find org-mode already had it.

Comment on this.

-1:-- Fine-tuning subtree visibility in org-mode (Post)--L0--C0--June 29, 2015 12:00 AM

Pragmatic Emacs: Prettier text replacement with anzu

We recently introduced basic text replacement, but it is easy to make things a bit snazzier. You can preview the replacement by installing the anzu package, and then adding the following to your emacs config file:

(require 'anzu)
(global-anzu-mode)
(global-set-key (kbd "M-%") 'anzu-query-replace)
(global-set-key (kbd "C-M-%") 'anzu-query-replace-regexp)

This is illustrated in the example below from my post on renaming multiple files in dired – note how when I replace “foo” with “bar” you see the replacement text previewed next to the original text

dired-rename.gif

If you use my recommended setup, prelude, this behaviour is the default,so you shouldn’t need to do anything.

-1:-- Prettier text replacement with anzu (Post Ben Maughan)--L0--C0--June 28, 2015 09:12 PM

Ryan Rix: The Beginnings of a Systems Integration Hydra

Over on Hardcore Freestyle Emacs I'm beginning to take a look at what sort of systems-level integration I want to build in to my Emacs workflow; the end goal, naturally is to run nearly everything inside of Emacs; having to open up a terminal to do things like this throughout the day is a major pain in the butt and it'd be nice to not have to do that, even though XMonad does make that fairly simple.
-1:-- The Beginnings of a Systems Integration Hydra (Post)--L0--C0--June 28, 2015 12:00 AM

Endless Parentheses: How I blog: One year of posts in a single org file

When this blog was conceived, I decided that I wanted it to be entirely contained in a single org file, and that this would also be my Emacs init file. On the blog’s very first post I explained how to implement the latter, an init file that also serves other purposes. Today, Endless Parentheses turns 1 year old, and it’s time to explain the former, how to turn a file into a blog.

As is usually the case with first birthdays, the child has no clue of what’s going on, and it’s really just an excuse to indulge the parents into praising themselves for keeping the kid alive 12 whole months. As such, this will not be a productive post. Expect no code snippets, no Emacs news, and no productivity tips, as I release myself from my usual shackles and go off on a tangent for a change. Still, just in case someone else likes the idea, I’ve pushed the code to Github.

Before anything else, it should go without saying that the content of the posts is written in org-mode. The engine I use for exporting is a large wrapper around ox-jekyll, and the posts are all pushed to Github and rendered by their built-in Jekyll support.

Why keep a blog in a single file?

First of all because org, once you learn its knobs and bobs, is just plain powerful. This is all the more true because the post contents are also in org format, so you’re effectively removing one layer of distance between you and the content. For example, if I want to reschedule a post, I just find it with C-c C-j and hit C-c C-s; whereas, if the posts were separate files, I’d have to find it in dired, then visit it, and then hit C-c C-s.

This is a small difference, but it applies all around. If I want to link to a previous post, I find it with C-c C-j and then move back with C-u C-SPC, all without leaving the buffer. When I look at the posts list, the tags are listed right beside the title, I don’t have to open a file to see them.

Now let me be honest with you. I didn’t predict these advantages before I started the blog, so it’s not why I chose this approach. Rather, it was linked to the fact that I wanted to blog about all these elisp snippets I had built over years and accumulated in my init file.

You see, if posts were separate files I would have to copy the snippets to a separate org file, and then write about them there, and then export them to Jekyll. In this scenario, I just know I would eventually change some snippet (a healthy init file is a fluid creature) and forget to update the corresponding org file, and the thought of leaving out-dated code lying around sent a chill through my spine. Not to mention, this whole flow of “init file → org post → jekyll post” has one layer too many for my taste, and redundancy is an evil I slay with a chainsaw.

How it works

First of all, here’s what I see when Emacs starts up.

init-org-1.png

If you read (and remember) the first post, you’ll know that the actual init file is composed of code blocks inside that init.el headline. Everything else is text. Also, notice that last SEQ_TODO header. It specifies that headlines in this file have three possible TODO states, TODO, READY and DONE.

init-org-meta-binds.png

Every post is marked with one of these states, and that is what defines them. TODO is something I plan on writing about; READY is, well, ready to publish; and DONE is published.

This three-states setup has several uses, mainly:

  • I can review my entire history and my future schedule with a custom agenda command, leveraging all the features of org-agenda.
  • Whenever I change a snippet, and the headline above it is marked DONE, I’m immediately reminded to update the post (as simple as C-c b). So I’ll never leave outdated code around.
  • When Monday arrives sooner than expected, and I didn’t write anything new, I can issue C-c / T READY and get an org-sparse-tree of all READY posts.

init-org-ready-state.png

See how practical that is? This file is not just the blog and the init file, it’s also the future posts queue and the “vague ideas” list. All without having to do manual maintenance. Of course, the headline is automatically marked DONE when posted.

Then there are a few more advantages that arise simply from the fact we’re using org-mode.

  • I can physically move posts around with M-up/down and M-S-right/left, to whatever order makes more sense, instead of being constrained by alphabetical or historical order of files.
  • By nesting several posts under a headline, any tags or properties applied to the parent are inherited by the posts. (Killing redundancy, remember?)

    init-org-news.png

  • The C-c C-j (org-goto) command makes it a breeze to jump around.
  • It feels more Emacsy than anything else you could possibly do.

Last, but not least

This part applies to any of the org blogging methods (not just my setup), but still, the org-export engine is extensive and even more so if you know where to hack it (org links, in particular, are extremely versatile).

With a single C-c b, Emacs will spell-check the contents, export to Jekyll, clean up some links, move the file to the right directory, bring up magit-status, and even save a commit message to the kill-ring. You could even have it commit and push automatically too, if you’re a bit of a thrill seeker.

Good night, and thanks for coming

Well, now that I’ve finished monologuing, the cake is completely gone, there’s more candy on the floor than on the tables, and I think I hear a child crying somewhere, so it’s clearly time to wind down the party. Thanks for reading, and I mean that generally. Thanks for commenting too. Thanks for the emails and the tweets. Thanks for the bug reports and the pull requests, and any other form of positive interaction. This was a fun year.

Oh, and thanks for coming to the party. Little EP totally didn’t notice that you didn’t bring a gift. (Though he might notice next year… just saying.)

Comment on this.

-1:-- How I blog: One year of posts in a single org file (Post)--L0--C0--June 26, 2015 12:00 AM

Alex Schroeder: Emacs Live

I’ve been trying to switch from my own Emacs setup to Emacs Live. Some things are great. Some things… I don’t know. They need a lot of getting used to.

I’ve created my own pack in ~/elisp/alex-pack.

In ~/.emacs-live.el:

(live-add-packs '(~/elisp/alex-pack))

Paredit is messing with how I navigate using Control + Arrow Keys.

In my own pack, ~/elisp/alex-pack/init.el:

(dolist (x '(scheme emacs-lisp lisp clojure))
  (let ((hook (intern (concat (symbol-name x) "-mode-hook"))))
    (set hook (delq 'enable-paredit-mode (symbol-value hook)))))

Delete Trailing Whitespace Mode deleted some non-breaking spaces in my SVG document. This may be an indication that I should write better SVG.

In my own pack, ~/elisp/alex-pack/init.el:

(setq before-save-hook
      (delq 'live-clenup-whitespace before-save-hook))

Changing how C-h works. Help is now on M-h. I’ll adapt. Except I’ve found modes where M-h does describe-mode and that confuses me. I think I’m going to stop using the bindings pack provided by Emacs Live.

In ~/.emacs-live.el:

(setq live-packs (delete "stable/bindings-pack" live-packs))

Ispell no longer shows any content in the *Choices* buffer/window. What’s up?

I have trouble accessing the file open history using <up> or M-p. Similarly, I can’t edit the filename using <left> or <right>. What’s up? I think I’m going to disable Ido mode for find-file.

In my own pack, ~/elisp/alex-pack/init.el:

(ido-mode 'buffers)

I like the some of the fontification, but it turns out that I’m starting to confuse the Emacs frame with terminal windows. So I think I’m going to stop using the colour pack provided by Emacs Live.

In ~/.emacs-live.el:

(setq live-packs (delete "stable/colour-pack" live-packs))

Tags: RSS

-1:-- Emacs Live (Post)--L0--C0--June 24, 2015 04:15 PM

Got Emacs?: EmacsConf 2015 (Announcement)

This should be interesting in that there's enough buzz for a Emacs conference.  And they're looking for speakers. This is in San Franscisco, CA.
-1:-- EmacsConf 2015 (Announcement) (Post sivaram (noreply@blogger.com))--L0--C0--June 23, 2015 05:59 AM

(or emacs: New video demo - counsel-load-theme

Without further ado, here's the video link.

The code

(defun counsel--load-theme-action (x)
  "Disable current themes and load theme X."
  (condition-case nil
      (progn
        (mapc #'disable-theme custom-enabled-themes)
        (load-theme (intern x))
        (when (fboundp 'powerline-reset)
          (powerline-reset)))
    (error "Problem loading theme %s" x)))

;;;###autoload
(defun counsel-load-theme ()
  "Forward to `load-theme'.
Usable with `ivy-resume', `ivy-next-line-and-call' and
`ivy-previous-line-and-call'."
  (interactive)
  (ivy-read "Load custom theme: "
            (mapcar 'symbol-name
                    (custom-available-themes))
            :action #'counsel--load-theme-action))

It looks almost trivial, the main idea is to disable all current themes and load the new one. Additionally, try to reset the powerline, since it has to match the mode-line face, which most themes customize.

The Interface

The interface of ivy-read is the same as the built-in completing-read in first two arguments. The difference is that it also accepts a callback through the :action argument. This callback will make the completion engine aware of what needs to be done with the completion result. The presence of the callback allows these completion engine features to work:

  • ivy-resume will resume the last completion. Very useful if you change your mind on the candidate, or want to examine a related candidate.
  • ivy-next-line-and-call selects the next matching candidate and executes the callback for it.
  • ivy-previous-line-and-call selects the previous matching candidate and executes the callback for it.

I like to think of ivy-resume as a DEL or <left> for completion. As you can erase or go back one character with the same DEL binding, regardless of the last character inputted (a or B etc), in the same way you can call the completion again with the same <f6> binding, regardless of the command that required completion (counsel-git-grep or counsel-load-theme or counsel-load-library etc). ivy-resume isn't bound by default, since it needs to be a global binding. I just use this in my config:

(global-set-key [f6] 'ivy-resume)

For the functions that execute the callback while changing the candidate, the idea is:

  • C-M-n (ivy-next-line-and-call) corresponds to C-n (ivy-next-line),
  • C-M-p (ivy-previous-line-and-call) corresponds to C-n (ivy-previous-line).

I've also showed off a generic ivy feature: M-j will yank the word at point into the minibuffer. Think of it as the mirror of C-w in isearch. It could not be C-w, since I like C-w being bound to kill-region.

The command/insert mode split

counsel-load-theme.png

Finally, I show off the equivalent hydra-based modal selection method. So instead of pressing C-M-n C-M-n C-M-n C-M-p C-M-p, you can press the equivalent C-o c jjjkk. Luckily, you don't need to remember a lot of bindings for this hydra mode: just press C-o and read the hints. I'll just list the exit points, since that's usually the more important stuff:

  • To exit the "command-mode" completely, press o or the standard C-g.
  • To exit the "command-mode" with the current candidate, press f or d.
  • To exit the "command-mode" and once again edit the minibuffer, press i.

You might ask why f and d do the same. They actually mirror C-j (ivy-alt-done) and C-m (ivy-done). And if you ask what the difference between those two is, the answer is that ivy-alt-done will not exit the completion when selecting directories during file name completion. It may be possible to extend this to other types of completion where it makes sense to select something but not to exit the minibuffer.

Outro

If you're using ivy-mode, make sure to try the new features: the action-using commands should work for any command that starts with counsel-. For other commands, like package-install, you can only select one candidate with C-m.

Also do try C-M-n with counsel-find-file: you'll be able to cycle through all files in a directory without exiting the completion. Same goes for ivy-switch-buffer, which should probably be renamed to counsel-switch-buffer for consistency.

-1:-- New video demo - counsel-load-theme (Post)--L0--C0--June 22, 2015 10:00 PM

Grant Rettke: How to backward delete a word in Emacs

M-DEL

-1:-- How to backward delete a word in Emacs (Post Grant)--L0--C0--June 21, 2015 08:39 PM

Rubén Berenguel: Using QGIS to create a custom map

I’ve always loved maps. I guess it goes well with liking drawing and maths, there’s a very deep connection (the pun is intended) between the subjects. As a map lover, when we decided to relocate to a somewhat more countryside town, I wanted better-looking maps to wander around the area. I checked government-issued maps, but they were either too large (scale 1:25000) or didn’t show the area I was interested (for the 1:10000 maps.) Thus I did what any other nerd would do: roll my own map as I wanted it.

The end result looks like this (downscaled to 20%)

The real map is 1.4 × 1.2 metres

Getting the data

My first idea was to follow the steps of Dominik Schwarz in his quest for printing a huge map of the world, and harvest images from Google Maps and get away with it. But one of the fundamental parts of my map was supposed to be having all walking routes visible and available. Google Maps won’t show that, or at least not everything. Specially among vines or forests. What’s the point of living close to the Pantà de Foix (the Foix Dam) and not being able to walk there?

Castellet, just beside Pantà de Foix
Hence, I turned to our trusty government, and in particular to the Catalan Cartographic and Geologic Institute (from now on the ICC, which is the Catalan acronym.) They have a powerful online map application, with good quality satellite imagery, elevation data, toponymy… All is available online. And for free download! You only need to register, and seem to be limited to one download at a time. No big issue, although I needed to get 16 satellite tiles.

There is a huge amount of data available for download. Since my knowledge of technical map stuff ended at knowing that the technical term for what I wanted to do is GIS , I wasn’t really sure what files I needed and what I could do with them on a Mac without having to pay for a dedicated app.

The first step was finding a free app that could “do maps.” After some Googling it was pretty clear it had to be QGIS (formerly known as Quantum GIS) which is free, open-source and available cross-platform. It was quite straightforward to install on Mac, just needing some extra libraries to work with the satellite image files I was using. Once I had it, the real “nightmare” started, where I had to decide what kind of files I needed and what they were good for. This is where ICC helped, since they have documentation for all their file types, not only explaining what they have but also what they are useful for.

For a starter I downloaded all satellite imagery with a scale of 1:2500, generated on 2014. Recent enough. This was the easy one:
  • Download each file
  • Unzip it
  • In QGIS, Layer→Add Layer→Add Raster Layer (or Shift-Cmd-R)
Since the files are already geolocated data (the format is SID, which is why I needed a special plugin to handle it) in just a few minutes I had a satellite view of the area, automatically placed and multi-scale. Not bad for a few minutes of downloading and goofing around.

Then the real work started: I wanted street names, road names and features. How? I’m pretty sure there are many ways, but the one I found most straightforward was:
  • Download Topography-base maps, with a scale of 1:2500 in SHP (Shapefile) format
  • In QGIS, Layer→Add Layer→Add Vector Layer (or Shift-Cmd-V)
  • Select full folder, Arc/Info binary format and browse to it
  • Select the files you really want
ICC provides a lot of files for each map tile at that resolution. Lots of stuff I didn’t really want. I chose only toponymy, elevation lines, hydrography and roads. Why?
  • I needed toponymy layers to set labels to features
  • Elevation looks very cool (even if I disabled it for the final generated map)
  • With hydrography I could change the colour from the satellite imagery to a more bluish tone in water areas (since the Foix dam is quite greenish)
  • Roads was the main reason I was doing this

Adding some text

Once I had everything in place I wondered: Where are the labels?
  • Double-click a toponymy layer
  • Labels→Label this layer with…
Here some of the huge power of QGIS started to appear. Most “variables” in the map can be controlled via either fields in the data layer OR functions applied to these fields. So, I could easily label by a data column, like TEXT_TOP (which had the common name of the feature, like Pantà de Foix.) But I also could set the size of the font to the one defined in the datafile. And I could even choose the font according to the feature type (or in my case, the font defined in the data file: the data provided by the ICC had font size and font family for the features) with a function!

CASE 
WHEN "FONTNAME" LIKE 'Times New Roman' THEN 'Georgia'
WHEN "FONTNAME" LIKE'Arial' THEN 'Futura'
WHEN "FONTNAME" LIKE 'Arial Narrow' THEN 'Futura'
WHEN "FONTNAME" LIKE 'Courier New' THEN 'Inconsolata'
END

The language seems to be QGIS specific, but has SQL-like LIKE. Single quotes denote strings, double quotes denote fields. Important to remember, since it’s very easy to mix one with the other without realising. I set all labels to show, even colliding labels. I didn't want to miss a castle or church because it was too close to a large street. I'd rather have overlapping labels in this case.

Then I decided I wanted something more: I wanted icons to show what each feature was. Icons like what you have in a normal map, where churches are indicated by a cross and castles by a tower. The layer I was working with (the toponymy layer) only had type codes for its features, so I couldn’t really tell what a 23 was. Church? Castle? Farm?

Symbols…? Labels...?

To do this, I needed to download an auxiliary file (which the ICC already told me about when I clicked Shapefile as type of download) having the list of code together with human-readable definitions. It is loaded as a single-file Shapefile layer, but has no “visible data.” Instead you need some magic.
  • Open a toponymy layer
  • Select Joins
  • Add a join layer with codis_topo (the source, in this case the layer I just added) and selecting as join field the type code (in this case, CODI_TOP)
  • I disabled caching and added a custom name to the newly created column, the join

A join
With this at least I’d know, since the type was now human-readable. I still didn’t know how to add icons, but I could edit the name and add a letter to denote what it is (or I could even add some text.) The labelling function is then

CASE 
WHEN "coditopo_03ca_CONCEPTE" LIKE '%capella%' THEN CONCAT( '† ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE '%monestir%' THEN CONCAT( '† ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE '%monestir%' THEN CONCAT( '† ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE '%tedral%' THEN CONCAT( '† ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE 'Església' THEN CONCAT( '† ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE 'Convent' THEN CONCAT( '† ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE 'Temple' THEN CONCAT( '† ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE 'Arc' THEN CONCAT( 'fl ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE '%ducte' THEN CONCAT( 'fl ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE 'Font, bullidor, deu' THEN CONCAT( 'ƒ ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE '%toll,%' THEN CONCAT( '≈ ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE '%astell,%' THEN CONCAT( 'Ħ ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE 'Talaia, torre de defensa, torre de guaita' THEN CONCAT( 'Ħ ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE '%rica (en%' THEN CONCAT( 'Ħ ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE 'Muralla' THEN CONCAT( '= ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE 'GR (Sender de Gran Recorregut)' THEN CONCAT( 'GR: ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE 'PR (Sender de Petit Recorregut)' THEN CONCAT( 'PR: ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE 'Pou%' THEN CONCAT( 'ů ', "TEXT_TOP" )
WHEN "coditopo_03ca_CONCEPTE" LIKE '%pont%' THEN CONCAT( 'π ', "TEXT_TOP" )
ELSE "TEXT_TOP"
END

This, basically adds a weird symbol to the text, so, for instance, bridges have a π symbol in front. Not great, but I actually printed it like this for a starter and it looks pretty decent. Here you can see a zoom in and the general view:

The general view

A zoom
In the zoom you can see the glyphs for a fountain (Font d’Horta and Font de Mas Carlús), a church (Sant Pere) an aqueduct (de les Aigües de Vilanova i la Geltrú) and a bridge (de la font d’Horta.) This was nice but non-optimal. I wanted symbols! So, how do you get symbols if you have a toponymy layer with lines, not feature points?

Symbols!!

First, duplicate the layers holding the labels. It can be done without duplicating, but it introduces a layer of complication for the human… So, if your computer doesn’t mind the extra layers, it is easier for you as human to have a separate layer for the labels and a layer for the icons.

Once you have duplicated the layer, change its style to rule-based, and add marks following rules with the add sign. For instance, this is the rule I use for Castell (castle):

CASE
WHEN "coditopo_03ca_CONCEPTE" LIKE 'Castell,%' THEN 1
ELSE 0
END

with SVG marker svg/fortress2.svg


NOTE: if you are using only one condition, for rule-based markers you can just keep it at

"coditopo_03ca_CONCEPTE" LIKE 'Castell,%'

since it will just return 0 or 1 as needed. You'll only need CASE when you have several types going to the same marker. Since I just copy-pasted most of my code I didn't mind the extra cruft. Thanks to Nathan Woodrow, one of QGIS' developers for pointing it out on twitter.

To add this as image you need to select Marker, then SVG marker. I found very nice SVG icons here, so I use this opportunity as credit (since the printed map is just for my personal use.) And indeed, you need to add as many rules as icons you have. That’s life. A GIS professional probably knows other ways, I’m just a newbie. Once you have all the rules you want, you can copy-paste them in all the layers. Actually, I think all this map business would have been easier merging all vector layers before doing anything else, but since I didn’t do it from the beginning it was hard to do it now. You can find how to merge vector layers in this StackOverflow question.
This is how Style should look like with rule-based

And this is what a specific rule looks like
 After adding markers you’ll need to tweak them. I did one part in SVG (adding stroke to the shapes and changing colours directy in the file so I didn’t need to bother in QGIS) and some others relating to placement (middle point and then changing size and baseline of the icon.) I also did some changes editing the file directly in Emacs, since QGIS files are just XML and replacing is much faster without clicking anything, once you know how the fields are named. The end result? Here you can see the same area as above with icons.

A zoom in with markers in place


Nice, don’t you think?

So, how do you print it?

This is actually very very easy. To print you need a New print composer from the Project menu. This is a set of rules to print some map. So, you choose the paper size and then add pieces to the layout. The basics are a map, a scale and some text, or at least this is the bare minimum I wanted. 

Once you have the map area you can change scale, drag around and change many things, it’s awesome. I decided to use 1:10000 since fitting the whole map in the map area I chose was 1:9556 and I’d rather have a round number and lose some map around the borders. Ready to export to PDF and bring to a printing place. You can also generate a JPG file, but beware of trying to open a 176 Mb with an iPad (even if it is an Air.) It won’t zoom. On the other hand, for my Mac it was much easier to open and zoom/navigate a huge JPG rather than a huge PDF file. Also keep in mind large maps take a long time to export: if you are making tests disable as many layers as you can really manage.

The map composer reminds me of Scribus
Since I wanted a portable version of the map to have on my iPad Air, I decided to split it using ImageMagick in 9 tiles I could easily zoom in any app. IM has the nice “crop” feature that can generate tiles, like:

convert -crop 4x3 MEGAMAP.jpg megamap/tiles%03d.jpg

The problem is that this takes way, way longer than expected, because the image is too big. The fastest way is instead to use the stream command to generate raw RGB data and convert that to JPG. So, repeating 9 times stuff like:

stream -extract 5566x3937+11132+7874 MEGAMAP.jpg t9.rgb
convert -depth 8 -size 5566x3937 rgb:t9.rgb t9.jpeg

Since I only needed to do it this time I didn’t bother to code it as a bash for loop or anything. Whereas convert was taking more than 12 hours to do anything, stream took 3 minutes, counting the time I needed to compute the pixel shifts and the conversions. And probably even the time it took me to create a index.html with the pieces to zoom.

What’s the takeaway of this post? Doing maps is fun, and using QGIS is not that hard as it may seem. And yes, I have an awesome big map now.
-1:-- Using QGIS to create a custom map (Post Rubén Berenguel (noreply@blogger.com))--L0--C0--June 19, 2015 11:13 AM

Julien Danjou: Timezones and Python

Recently, I've been fighting with the never ending issue of timezones. I never thought I would have plunged into this rabbit hole, but hacking on OpenStack and Gnocchi I felt into that trap easily is, thanks to Python.

“Why you really, really, should never ever deal with timezones”

To get a glimpse of the complexity of timezones, I recommend that you watch Tom Scott's video on the subject. It's fun and it summarizes remarkably well the nightmare that timezones are and why you should stop thinking that you're smart.

The importance of timezones in applications

Once you've heard what Tom says, I think it gets pretty clear that a timestamp without any timezone attached does not give any useful information. It should be considered irrelevant and useless. Without the necessary context given by the timezone, you cannot infer what point in time your application is really referring to.

That means your application should never handle timestamps with no timezone information. It should try to guess or raises an error if no timezone is provided in any input.

Of course, you can infer that having no timezone information means UTC. This sounds very handy, but can also be dangerous in certain applications or language – such as Python, as we'll see.

Indeed, in certain applications, converting timestamps to UTC and losing the timezone information is a terrible idea. Imagine that a user create a recurring event every Wednesday at 10:00 in its local timezone, say CET. If you convert that to UTC, the event will end up being stored as every Wednesday at 09:00.

Now imagine that the CET timezone switches from UTC+01:00 to UTC+02:00: your application will compute that the event starts at 11:00 CET every Wednesday. Which is wrong, because as the user told you, the event starts at 10:00 CET, whatever the definition of CET is. Not at 11:00 CET. So CET means CET, not necessarily UTC+1.

As for endpoints like REST API, a thing I daily deal with, all timestamps should include a timezone information. It's nearly impossible to know what timezone the timestamps are in otherwise: UTC? Server local? User local? No way to know.

Python design & defect

Python comes with a timestamp object named datetime.datetime. It can store date and time precise to the microsecond, and is qualified of timezone "aware" or "unaware", whether it embeds a timezone information or not.

To build such an object based on the current time, one can use datetime.datetime.utcnow() to retrieve the date and time for the UTC timezone, and datetime.datetime.now() to retrieve the date and time for the current timezone, whatever it is.

>>> import datetime
>>> datetime.datetime.utcnow()
datetime.datetime(2015, 6, 15, 13, 24, 48, 27631)
>>> datetime.datetime.now()
datetime.datetime(2015, 6, 15, 15, 24, 52, 276161)


As you can notice, none of these results contains timezone information. Indeed, Python datetime API always returns unaware datetime objects, which is very unfortunate. Indeed, as soon as you get one of this object, there is no way to know what the timezone is, therefore these objects are pretty "useless" on their own.

Armin Ronacher proposes that an application always consider that the unaware datetime objects from Python are considered as UTC. As we just saw, that statement cannot be considered true for objects returned by datetime.datetime.now(), so I would not advise doing so. datetime objects with no timezone should be considered as a "bug" in the application.

Recommendations

My recommendation list comes down to:

  1. Always use aware datetime object, i.e. with timezone information. That makes sure you can compare them directly (aware and unaware datetime objects are not comparable) and will return them correctly to users. Leverage pytz to have timezone objects.
  2. Use ISO 8601 as input and output string format. Use datetime.datetime.isoformat() to return timestamps as string formatted using that format, which includes the timezone information.

In Python, that's equivalent to having:

>>> import datetime
>>> import pytz
>>> def utcnow():
return datetime.datetime.now(tz=pytz.utc)
>>> utcnow()
datetime.datetime(2015, 6, 15, 14, 45, 19, 182703, tzinfo=<UTC>)
>>> utcnow().isoformat()
'2015-06-15T14:45:21.982600+00:00'


If you need to parse strings containing ISO 8601 formatted timestamp, you can rely on the iso8601, which returns timestamps with correct timezone information. This makes timestamps directly comparable:

>>> import iso8601
>>> iso8601.parse_date(utcnow().isoformat())
datetime.datetime(2015, 6, 15, 14, 46, 43, 945813, tzinfo=<FixedOffset '+00:00' datetime.timedelta(0)>)
>>> iso8601.parse_date(utcnow().isoformat()) < utcnow()
True


If you need to store those timestamps, the same rule should apply. If you rely on MongoDB, it assumes that all the timestamp are in UTC, so be careful when storing them – you will have to normalize the timestamp to UTC.

For MySQL, nothing is assumed, it's up to the application to insert them in a timezone that makes sense to it. Obviously, if you have multiple applications accessing the same database with different data sources, this can end up being a nightmare.

PostgreSQL has a special data type that is recommended called timestamp with timezone, and which can store the timezone associated, and do all the computation for you. That's the recommended way to store them obviously. That does not mean you should not use UTC in most cases; that just means you are sure that the timestamp are stored in UTC since it's written in the database, and you check if any other application inserted timestamps with different timezone.

OpenStack status

As a side note, I've improved OpenStack situation recently by changing the oslo.utils.timeutils module to deprecate some useless and dangerous functions. I've also added support for returning timezone aware objects when using the oslo_utils.timeutils.utcnow() function. It's not possible to make it a default unfortunately for backward compatibility reason, but it's there nevertheless, and it's advised to use it. Thanks to my colleague Victor for the help!

Have a nice day, whatever your timezone is!

-1:-- Timezones and Python (Post Julien Danjou)--L0--C0--June 16, 2015 05:39 PM

Bozhidar Batsov: CIDER 0.9

CIDER 0.9 is finally out and it’s our best release yet (believe it or not)! It took a lot more time than I originally anticipated, but at least we managed to ship in time for EuroClojure!

There are a ton of important changes and new features in 0.9 and now I’ll go quickly through some of them.

Debugger

Believe it or not CIDER now has a debugger! This was like the most requested feature ever, so I’m sure at least some of you are excited. The debugger was developed by the awesome Artur Malabarba. He even wrote a post about it and I guess you should read it.

Dependency isolation

CIDER’s dependencies will no longer affect your projects (read this as introduce dependency conflicts in them). All the dependencies are now isolated using source rewriting (simply put – they live in different namespaces than the original libraries). This magic is done by mranderson. Thanks to Benedek Fazekas for creating this small but super helpful tool!

Rich code completion

Completion candidates are now annotated with information about the namespace and the type of the thing being completed. It’s pretty neat.

The screenshot above features company-mode. The annotations are not supported in auto-complete-mode (that’s a limitation of AC, not a limitation of CIDER).

Misc additions

Here’s a short list of other important additions:

  • Support for Piggieback 0.2
  • New code formatting commands (based on cljfmt)
  • New EDN data formatting commands

Changes

There were also a few important changes. Most notably we had to kill source-tracking code evaluation, as it wasn’t playing nice with ClojureScript. This was also a hacky solution and I still hope than one day this will be properly supported in nREPL itself. In simple terms – var definitions evaluated by themselves won’t have any location metadata set for them, which will make it impossible to go their definition. You can also help out by voicing your support for this nREPL ticket’s patch to be merged.

You’ll also notice that some commands that didn’t prompt for confirmation in the past do so now (e.g. find-var). This was done mostly for consistency with Emacs’s own commands that do similar things. The behavior is configurable via cider-prompt-for-symbol. If a ton of people dislike the new defaults reverting them is on the table.

All the Gory Details

There were truly a ton of changes and there’s little point in me repeating them here. If you want to know everything have a look at the release notes.

The Road Ahead

Going forward our top priority will be merging some functionality from refactor-nrepl and clj-refactor into CIDER itself. Think of things like find-usages, extract-definition, etc. Refining the debugger will be another top priority.

We’ll also try to do some important internal changes:

Depending of how well we progress on those tasks the next release will be either 0.10 or 1.0. I won’t make any commitments about its release date (but judging from past it will likely be 3 to 6 months from now).

If you’re wondering why things are moving more slowly lately, here’s the answer for you – I’ve been super busy since the beginning of the year and haven’t had much time for open-source projects. I’m hoping this will change, but only time will tell. I’m very happy that a lot of people contributed to the development of CIDER 0.9. The project definitely doesn’t have a bus factor of one. :–)

Special thanks to Michael Griffiths who did a ton of great work on this release. You rock!

P.S. Recently I talked on the Cognicast about CIDER (in general and 0.9 in particular). You might find this episode interesting.

P.P.S. I’m rarely on IRC these days. #cider on slack and our gitter channel are the official CIDER chats as far as I’m concerned.

-1:-- CIDER 0.9 (Post)--L0--C0--June 16, 2015 12:59 PM

sachachua: Using your own Emacs Lisp functions in Org Mode table calculations: easier dosage totals

UPDATE 2015-06-17: In the comments below, Will points out that if you use proper dates ([yyyy-mm-dd] instead of yyyy-mm-dd), Org will do the date arithmetic for you. Neato! Here’s what Will said:

Hi Sacha. Did you know you can do date arithmetic directly on org’s inactive or active timestamps? It can even give you an answer in fractional days if the time of day is different in the two timestamps:

| Start                  | End                    | Interval |
|------------------------+------------------------+----------|
| [2015-06-16 Tue]       | [2015-06-23 Tue]       |        7 |
| <2015-06-13 Sat>       | <2015-06-15 Mon>       |        2 |
| [2015-06-10 Wed 20:00] | [2015-06-17 Wed 08:00] |      6.5 |
#+TBLFM: $3=$2 - $1 

Here’s my previous convoluted way of doing things… =)
—-

I recently wrote about calculating how many doses you need to buy using an Org Mode table. On reflection, it’s easier and more flexible to do that calculation using an Emacs Lisp function instead of writing a function that processes and outputs entire tables.

First, we define a function that calculates the number of days between two dates, including the dates given. I put this in my Emacs config.

(defun my/org-days-between (start end)
  "Number of days between START and END.
This includes START and END."
  (1+ (- (calendar-absolute-from-gregorian (org-date-to-gregorian end))
         (calendar-absolute-from-gregorian (org-date-to-gregorian start)))))

Here’s the revised table. I moved the “Needed” column to the left of the medication type because this makes it much easier to read and confirm.

| Needed | Type         | Per day |      Start |        End | Stock |
|--------+--------------+---------+------------+------------+-------|
|     30 | Medication A |       2 | 2015-06-16 | 2015-06-30 |     0 |
|      2 | Medication B |     0.1 | 2015-06-16 | 2015-06-30 |   0.2 |
#+TBLFM: @2$1..@>$1='(ceiling (- (* (my/org-days-between $4 $5) (string-to-number $3)) (string-to-number $6)))

C-c C-c on the #+TBLFM: line updates the values in column 1.

@2$1..@>$1 means the cells from the second row (@2) to the last row (@>) in the first column ($1).  '  tells Org to evaluate the following expression as Emacs Lisp, substituting the values as specified ($4 is the fourth column’s value, etc.).

The table formula calculates the value of the first column (Needed) based on how many you need per day, the dates given (inclusive), and how much you already have in stock. It rounds numbers up by using the ceiling function.

Because this equation uses the values from each row, the start and end date must be filled in for all rows. To quickly duplicate values downwards, set org-table-copy-increment to nil, then use S-return (shift-return) in the table cell you want to copy. Keep typing S-return to copy more.

This treats the calculation inputs as strings, so I used string-to-number to convert some of them to numbers for multiplication and subtraction. If you were only dealing with numbers, you can convert them automatically by using the ;N flag, like this:

| Needed | Type         | Per day | Days | Stock |
|--------+--------------+---------+------+-------|
|      6 | Medication A |       2 |    3 |     0 |
|      1 | Medication B |     0.1 |    3 |   0.2 |
#+TBLFM: @2$1..@>$1='(ceiling (- (* $3 $4) $5)));N

The post Using your own Emacs Lisp functions in Org Mode table calculations: easier dosage totals appeared first on sacha chua :: living an awesome life.

-1:-- Using your own Emacs Lisp functions in Org Mode table calculations: easier dosage totals (Post Sacha Chua)--L0--C0--June 16, 2015 12:00 PM

Grant Rettke: Eclim: The power of Eclipse in your favorite editor

Eclim provides the ability to access Eclipse code editing features (code completion, searching, code validation, and many more) via the command line or a local network connection, allowing those features to be integrated with your favorite editor. Eclim provides an integration with Vim, but third party clients have been created to add eclim support to other editors as well (emacs, sublime text 2, textmate).

Via emacslife.

-1:-- Eclim: The power of Eclipse in your favorite editor (Post Grant)--L0--C0--June 13, 2015 05:46 PM

Alex Schroeder: AKICK on the Emacs Channel

Here’s how to define two commands to use ChanServ’s AKICK command on Freenode. The commented commands are what I had before. The benefit of the AKICK command is the timeout.

;; rcirc /kickban

(eval-after-load 'rcirc
  '(defun-rcirc-command kickban (input)
     "Kickban a nick for two hours."
     (interactive "s")
     ;; (rcirc-send-string process (format "MODE %s +b %s!*@*" target input))
     ;; (rcirc-send-string process (format "KICK %s %s kickban!" target input))
     (rcirc-send-privmsg process "chanserv" (format "AKICK %s ADD %s !T 120 banned for 2h -- kensanata" target input))))

;; rcirc /unban

(eval-after-load 'rcirc
  '(defun-rcirc-command unban (input)
     "Unban a nick."
     (interactive "s")
     ;; (rcirc-send-string process (format "MODE %s -b %s!*@*" target input))
     (rcirc-send-privmsg process "chanserv" (format "AKICK %s DEL %s!*@*" target input))))

Tags: RSS RSS RSS

-1:-- AKICK on the Emacs Channel (Post)--L0--C0--June 11, 2015 12:36 PM

Ivan Kanis: someday

someday

appt improvements

add popup. redo multi appointment patch without refactoring

-1:-- someday (Post Ivan Kanis)--L0--C0--June 11, 2015 03:04 AM

Ivan Kanis: maybe

maybe

climb old man of hoy

write clippy.el

i♥emacs emacs for beginners

</example>

The first section is where I capture stuff. My weekly review is a natural process. I start at the top and type 'w' to refile. Using speed keys really help. If you don't know about it check out the documentation in the function org-speed-command-help.

Then I go through each section and see what needs to be done. I tried to be ruthless and put stuff in someday or maybe so that my list is current and feels fresh.

I then finish with things that are in someday (one day I will do it) to maybe (who knows?).

That pretty much sums up how I implement GTD with org mode.

-1:-- maybe (Post Ivan Kanis)--L0--C0--June 11, 2015 03:04 AM

Michael Fogleman: 2014: A Year of Emacs

My Emacs "birthday" is January 21, 2014. That's right-- I just started using Emacs this year! While it's not quite my one-year birthday, I thought I'd take advantage of the annual review blogging tradition to do a retrospective on my first year in Emacs. Note: much of the following chronology takes place during an extended romp through Asia.

I installed Arch Linux in the Fall of 2013. When using Linux, you often need to edit configuration files. The friend who helped me set me up with Vim. I learned the basics--how to insert, save, and quit--and not much else.

Arch on a bench

In January, I decided it might be fun to give Emacs a try. Here is a picture of Marina Bay in Singapore, taken just before or while I did the Emacs tutorial:

Marina Bay in Singapore, just before or during my Emacs birth

As a life-long user of keyboard shortcuts, I appreciated Emacs' notation for key bindings. The tutorial begins with basic cursor controls; easy enough, but nothing exciting. I could see the line, sentence, and buffer equivalents being useful. But my jaw dropped when it taught me C-t, transpose-chars. I knew that mastering just that command would probably save me a lot of pain and typos, and that was probably the least of what Emacs could do.

While the Emacs tutorial probably isn't meant to take more than an hour or two on the first try, it took me probably twice as long. This was because I made Anki cards. Some people say that Emacs' learning curve is steep. For myself, I've found Emacs' learning curve to be gentle, but (pleasantly) endless. Then again, it was probably making Anki cards that made the learning curve less steep for me.

A variation of this card was probably one of my first Emacs cards:

Anki card: transpose-chars 1

Cloze deletion cards like this one are very convenient for this kind of thing. One reason is that it's easy to add new, related information later:

Anki cards transpose-chars 2

In February, the battery on my little Arch laptop died. I spent most of the month without a laptop, waiting for a new battery to arrive by mail. I borrowed my friend and travel partner's machine to review my Anki cards. Still, I did manage to have a little time to develop my Emacs-fu.

In Hanoi, the hotel room we stayed in had a (virus-ridden) Windows machine. What else could I do but install Emacs? At this time, my Emacs configuration was not that complex, so it wasn't worth adapting. As it turned out, that time with a bare Emacs proved especially helpful for mastering some of the key Emacs concepts that I hadn't quite made use of yet; in particular, I got the hang of using multiple buffers in one session. By the the end of February, my new battery had arrived, and I was back in the Emacs saddle.

In March, I converted my my Emacs configuration to an Org-literate format, and started tracking it with Git. This was really the first project that I used Git for, and it gave me an opportunity to learn Git.

The other highlight of March was starting to learn a little Elisp. I found Robert Chassell's "An Introduction to Programming in Emacs Lisp" especially helpful. Chassell's overview and Org-mode's fantastic documentation helped me to write my first significant piece of Elisp (some org-capture templates).

April started off with a bang when my Emacs configuration was featured in a major motion picture, Tron Legacy. But that wasn't nearly as exciting as making my first major mode, tid-mode.

In late June, Artur Malabarba launched his Emacs blog, Endless Parentheses. One of his early posts was about hungry-delete. I was excited about it, but found that it did not work with transient-mark-mode. Artur encouraged me to write a fix. I was very excited when Nathaniel Flath, the maintainer, agreed to merge my patches into hungry-delete. At about the same time, Artur posted an adapted version of my narrow-or-widen-dwim function to Endless Parentheses. Sacha Chua, Nathaniel Flath, and some other Emacs Lisp hackers also offered suggestions.

At the end of July, I started learning Clojure, and added an initial CIDER configuration. While reading the Clojure Cookbook, I decided to hack out a function that turned the pages. Ryan Neufeld, one of the co-authors, merged my pull request, and a while later, I saw my function mentioned on Planet Clojure.

In October, I purchased a Mac for a Clojure contracting job. (Knowing Emacs helped me get the job!) Adding the :ensure keyword to my use-package declarations made the Emacs configuration portion of my OS X set-up process near-instant and pain-free. It was a really cool feeling to be using the same Emacs configuration on two different machines. By the end of the month, both were running on the newly-released Emacs 24.4.

Once Emacs 24.4 was out, Artur posted an Emacs 25 wishlist to Endless Parentheses. I responded with this tweet:

This tweet indirectly led to the happy occasion of the Emacs Hangouts.

That just about brings us to this post. In retrospect, I can't believe it's been less than a year, or that I've learned so much. Thanks to everyone who has helped me so far on the endless path of mastering Emacs, especially Ben, Eric, Sacha Chua, Artur Malabarba, and Bozhidar Batsov.

Joseph Campbell was a Lisp guru

I want to close this post out with Sacha's tweet about 2015:

Like Sacha, I want to learn how to write and run tests in Emacs. I want to learn more about org-mode's features and possible workflows. More broadly, I'd like to make more posts about Emacs, and find other ways to contribute to the Emacs community. I'd like to learn more about Emacs core development. In terms of contrib, I'd like to help accelerate the release of Magit's "next" branch. Above all, I'm sure I'll keep tweaking my configuration.

-1:-- 2014: A Year of Emacs (Post Michael Fogleman)--L0--C0--June 11, 2015 03:02 AM

Michael Fogleman: Emacs can keep in touch

I've wanted to show a specific Emacs workflow that I've established at the Emacs Hangouts that I've attended, but thought it might require too much explanation for an Emacs Hangout. Hence, this post. You might be interested in this workflow if you regularly edit a specific text file in a particular way, run an external program with the same parameters. You might also get some ideas if you're an Emacs-wielding programmer, and wish Emacs had some particular functionality, but would prefer to use Bash or another language to give it that functionality.

Some time ago, my friend Ben started working on a side project called "Keep In Touch." The premise: you have friends, family, and acquaintances that you'd like to stay in touch with. You maintain a list of those people, how often you'd like to contact them, and when you last contacted them. Keep In Touch can then figure out who you are out of touch with. For example, if you want to talk to someone every 30 days, but it's been 31 days, then you are out of touch with them, and the program tells you so. Simple!

Since I started using the program, I've fallen in love with using it. It has helped me to stay in touch with more people, more often. Moreover, I re-implemented Keep in Touch in Clojure as an exercise. And while Keep in Touch provides a serviceable CLI, I've worked out a simple Emacs interface for maintaining the database and viewing the program's output.

It all starts with telling Emacs where it can find the database:

(setq keepintouch-datafile "~/Dropbox/keepintouch.data")

Next, I needed a way to tell Emacs that I had contacted someone. The result is an interactive function that asks you who you contacted. It ordinarily assumes that you contacted the person today, but with a prefix argument, it asks you for an alternate date. Then it opens the file and makes the appropriate changes.

(defun keptintouch (arg)
  "Request a contact in a keepintouch.data file, and update their last
  contacted date (either today, or, if a prefix is supplied, a user-supplied date.)"
  (interactive "P")
  (let ((contact (read-string "Who did you contact? "))
        (date (if (equal arg nil)
                  (format-time-string "%Y/%m/%d")
                (read-string "When did you contact them? (year/month/date): "))))
    (save-excursion
      (find-file keepintouch-datafile)
      (goto-char (point-min))
      (search-forward contact)
      (forward-line -1)
      (beginning-of-line)
      (kill-line)
      (insert date)
      (save-buffer)
      (switch-to-buffer (other-buffer))
      (kill-buffer (other-buffer)))
    (message "%s was contacted." contact)))

"keptintouch" uses read-string to ask you who contacted, and, if need be, when you contacted them. It uses format-time-string to format today's date in the appropriate format.

The body of the function works almost like a keyboard macro. Often, you can start writing an Elisp function by hitting "C-h k" (describe-key) and finding out what functions are called when you execute the steps as keystrokes. Another way to do this quickly might be to make the keyboard macro, run edit-last-kbd-macro, and use the function annotations to read their documentation or make pseudo-code.

One reason that this is a function instead of a keyboard macro is that it uses some buffer management functions. (All interactive functions are of course accessible to keyboard macros through keyboard strokes, which can really save you some time in one-off situations!) It opens my Keep In Touch database in a new buffer, finds the contact, edits their last contacted date, saves, closes, and switches the buffers. If all went well, it reports success with message. If for some reason the contact is non-existent, search-forward will throw an error. (If that happens, or I want to view or edit the datafile manually, I find it with ido-recentf-open.)

All of this is wrapped in save-excursion, which is kind of like the shell commands popd and pushd except for buffers. In other words, it does something, and then returns the user to what they were doing before they executed the function.

Now that I had a way to maintain the database, I needed a way to run Keep in Touch and view its output, a sorted backlog of people that you are out of touch with.

The Clojure version of Keep in Touch is designed to be exported to a .jar file. Running the resulting program from the command line looks like this:

# note the special location of the .jar
cd target/uberjar
java -jar keepintouch-0.1.0-SNAPSHOT-standalone.jar ~/Dropbox/keepintouch.data schedule backlog

I decided to look at shell-command's documentation to find a way to run this command and put the results into an output buffer. As it turns out, shell-command is that function. As expected, it takes an argument for the desired shell-command. But it also allows for two optional arguments: the name of an output buffer, and the name of an error buffer.

(defun keptintouch-backlog ()
  "Create a buffer with Keep In Touch backlog."
  (interactive)
  (let ((src "~/src/keepintouch/clj/keepintouch")
        (jar "-jar target/uberjar/keepintouch-0.1.0-SNAPSHOT-standalone.jar")
        (cur default-directory)) 
    (cd src)
    (shell-command
     (concat "java " jar " " keepintouch-datafile " schedule backlog")
     "*Keep In Touch Backlog*")
    (cd cur)))

You'll see that the concat will expand to the shell command that runs the .jar, and that it is followed by an appropriate buffer name.

All of this Lisp makes its way to my finger with two key-binds (by way of use-package's bindkey.el):

(bind-keys ("C-c k" . keptintouch)
           ("C-c K" . keptintouch-backlog))

One variable, two functions, and two keybinds are all it took to make my life easier and get back to staying in touch with friends. You can eat your Emacs cake, and have defun, too.

-1:-- Emacs can keep in touch (Post Michael Fogleman)--L0--C0--June 11, 2015 03:02 AM

Timo Geusch: Improved syntax highlighting with org2blog and custom CSS

As mentioned in an earlier post, I changed my blogging workflow to org2blog for writing and editing posts in Emacs and only push them up to my WordPress blog when the posts are almost done. I still do the final editing in WordPress so I can tweak the SEO settings and all that, but the… Read More »

The post Improved syntax highlighting with org2blog and custom CSS appeared first on The Lone C++ Coder's Blog.

-1:-- Improved syntax highlighting with org2blog and custom CSS (Post Timo Geusch)--L0--C0--June 10, 2015 10:08 AM

linek posted a photo:

-1:--  (Post linek (nobody@flickr.com))--L0--C0--June 08, 2015 07:39 PM

Julien Danjou: Get back up and try again: retrying in Python

I don't often write about tools I use when for my daily software development tasks. I recently realized that I really should start to share more often my workflows and weapons of choice.

One thing that I have a hard time enduring while doing Python code reviews, is people writing utility code that is not directly tied to the core of their business. This looks to me as wasted time maintaining code that should be reused from elsewhere.

So today I'd like to start with retrying, a Python package that you can use to… retry anything.

It's OK to fail

Often in computing, you have to deal with external resources. That means accessing resources you don't control. Resources that can fail, become flapping, unreachable or unavailable.

Most applications don't deal with that at all, and explode in flight, leaving a skeptical user in front of the computer. A lot of software engineers refuse to deal with failure, and don't bother handling this kind of scenario in their code.

In the best case, applications usually handle simply the case where the external reached system is out of order. They log something, and inform the user that it should try again later.

In this cloud computing area, we tend to design software components with service-oriented architecture in mind. That means having a lot of different services talking to each others over the network. And we all know that networks tend to fail, and distributed systems too. Writing software with failing being part of normal operation is a terrific idea.

Retrying

In order to help applications with the handling of these potential failures, you need a plan. Leaving to the user the burden to "try again later" is rarely a good choice. Therefore, most of the time you want your application to retry.

Retrying an action is a full strategy on its own, with a lot of options. You can retry only on certain condition, and with the number of tries based on time (e.g. every second), based on a number of tentative (e.g. retry 3 times and abort), based on the problem encountered, or even on all of those.

For all of that, I use the retrying library that you can retrieve easily on PyPI.

retrying provides a decorator called retry that you can use on top of any function or method in Python to make it retry in case of failure. By default, retry calls your function endlessly until it returns rather than raising an error.

import random
from retrying import retry
 
@retry
def pick_one():
if random.randint(0, 10) != 1:
raise Exception("1 was not picked")


This will execute the function pick_one until 1 is returned by random.randint.

retry accepts a few arguments, such as the minimum and maximum delays to use, which also can be randomized. Randomizing delay is a good strategy to avoid detectable pattern or congestion. But more over, it supports exponential delay, which can be used to implement exponential backoff, a good solution for retrying tasks while really avoiding congestion. It's especially handy for background tasks.

@retry(wait_exponential_multiplier=1000, wait_exponential_max=10000)
def wait_exponential_1000():
print "Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards"
raise Exception("Retry!")


You can mix that with a maximum delay, which can give you a good strategy to retry for a while, and then fail anyway:

# Stop retrying after 30 seconds anyway
>>> @retry(wait_exponential_multiplier=1000, wait_exponential_max=10000, stop_max_delay=30000)
... def wait_exponential_1000():
... print "Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards"
... raise Exception("Retry!")
...
>>> wait_exponential_1000()
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/local/lib/python2.7/site-packages/retrying.py", line 212, in call
raise attempt.get()
File "/usr/local/lib/python2.7/site-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/local/lib/python2.7/site-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "<stdin>", line 4, in wait_exponential_1000
Exception: Retry!


A pattern I use very often, is the ability to retry only based on some exception type. You can specify a function to filter out exception you want to ignore or the one you want to use to retry.

def retry_on_ioerror(exc):
return isinstance(exc, IOError)
 
@retry(retry_on_exception=retry_on_ioerror)
def read_file():
with open("myfile", "r") as f:
return f.read()


retry will call the function passed as retry_on_exception with the exception raised as first argument. It's up to the function to then return a boolean indicating if a retry should be performed or not. In the example above, this will only retry to read the file if an IOError occurs; if any other exception type is raised, no retry will be performed.

The same pattern can be implemented using the keyword argument retry_on_result, where you can provide a function that analyses the result and retry based on it.

def retry_if_file_empty(result):
return len(result) <= 0
 
@retry(retry_on_result=retry_if_file_empty)
def read_file():
with open("myfile", "r") as f:
return f.read()


This example will read the file until it stops being empty. If the file does not exist, an IOError is raised, and the default behavior which triggers retry on all exceptions kicks-in – the retry is therefore performed.

That's it! retry is really a good and small library that you should leverage rather than implementing your own half-baked solution!

-1:-- Get back up and try again: <em>retrying</em> in Python (Post Julien Danjou)--L0--C0--June 02, 2015 10:21 AM

Eric James Michael Ritz: Firestarter: Automated Testing and More in Emacs

Like most of my Emacs-related posts, today’s is a short example of a useful package, one called Firestarter. It’s a recent addition to the plethora of available Elisp packages out there, not even two months old at this time of writing. Firestarter’s GitHub page provides a nice, terse summary of its purpose:

firestarter lets you execute all kinds of things after saving a buffer. You can run shell commands, execute interactive Emacs commands and evaluate arbitrary pieces of Emacs Lisp. Support for shell commands includes asynchronous process control, format chars in command strings and report of process on common process termination states.

I want to show you some simple ways in which I’ve found Firestarter useful.

My first application of Firestarter was to automate testing for PHP Mode. We have a collection of unit tests for PHP Mode which use ERT, the Emacs Lisp Regression Testing library that comes with Emacs. Once we have loaded our file of tests we can run them via M-x ert-run-tests-interactively. I like to run our unit tests whenever I’m making changes to PHP Mode; it is hardly a chore to run ERT manually, but I prefer to automate things whenever I can. And if I’m being honest, there have been times where I’ve written some patches in the late hours of the night, half-asleep, and lacking the self-discipline in that scenario to make sure all the tests pass. With Firestarter I can ensure the tests always run by adding a file-local variable to the PHP Mode sources, which looks like this:


;; Local Variables:
;; firestarter: ert-run-tests-interactively
;; End:

With this variable in the source files, and with firestarter-mode enabled, Firestarter will run that command any time I save those files. This is valuable not because it saves me any time, but because it prevents me from forgetting to run the unit tests after any modifications, a safety net, if you will.

Firestarter can do more than run Emacs commands. I can also use it to run shell commands. Here’s a simple, practical example: The top-level directory of the game I’m currently working on has a file of directory-level variables which looks like this:

((nil
  (firestarter-type . failure)
  (firestarter . "make game")))

In my first example firestarter had a symbol for its value, in which case Firestarter runs the Emacs command named by that symbol. But when given a string, as in this example, Firestarter treats it as a shell command. So now whenever I introduce game-breaking bugs add a new feature and save my work, Firestarter automatically runs make game, which in turn invokes GNU Make to rebuild the game. The other variable, firestarter-type, lets me control Firestarter’s output based on the results of make game. By giving it the value failure I will only see the output from make game if something goes wrong. A successful rebuild of the game will happen silently.

And that’s all for my simple introduction. Firestarter offers more customization than what I’ve demonstrated here, such as its format string variables which you can use to create more complex commands. If you’re looking for an easy way to automatically do something whenever you save a buffer or file then check out Firestarter. As I said, it hasn’t been out for long, but it has already become an important piece of my Emacs toolbox.


-1:-- Firestarter: Automated Testing and More in Emacs (Post ericjmritz)--L0--C0--June 01, 2015 04:18 PM

Chris Ball: Announcing GitTorrent: A Decentralized GitHub

(This post is an aspirational transcript of the talk I gave to the Data Terra Nemo conference in May 2015. If you’d like to watch the less eloquent version of the same talk that I actually gave, the video should be available soon!)

I’ve been working on building a decentralized GitHub, and I’d like to talk about what this means and why it matters — and more importantly, show you how it can be done and real GitTorrent code I’ve implemented so far.

Why a decentralized GitHub?

First, the practical reasons: GitHub might become untrustworthy, get hacked — or get DDOS’d by China, as happened while I was working on this project! I know GitHub seems to be doing many things right at the moment, but there often comes a point at which companies that have raised $100M in Venture Capital funding start making decisions that their users would strongly prefer them not to.

There are philosophical reasons, too: GitHub is closed source, so we can’t make it better ourselves. Mako Hill has an essay called Free Software Needs Free Tools, which describes the problems with depending on proprietary software to produce free software, and I think he’s right. To look at it another way: the experience of our collaboration around open source projects is currently being defined by the unmodifiable tools that GitHub has decided that we should use.

So that’s the practical and philosophical, and I guess I’ll call the third reason the “ironical”. It is a massive irony to move from many servers running the CVS and Subversion protocols, to a single centralized server speaking the decentralized Git protocol. Google Code announced its shutdown a few months ago, and their rationale was explicitly along the lines of “everyone’s using GitHub anyway, so we don’t need to exist anymore”. We’re quickly heading towards a single central service for all of the world’s source code.

So, especially at this conference, I expect you’ll agree with me that this level of centralization is unwise.

Isn’t Git already decentralized?

You might be thinking that while GitHub is centralized, the Git protocol is decentralized — when you clone a repository, your copy is as good as anyone else’s. Isn’t that enough?

I don’t think so, and to explain why I’d like you to imagine someone arguing that we can do without BitTorrent because we have FTP. We would not advocate replacing BitTorrent with FTP, and the suggestion doesn’t even make sense! First — there’s no index of which hosts have which files in FTP, so we wouldn’t know where to look for anything. And second — even if we knew who owned copies of the file we wanted, those computers aren’t going to be running an anonymous FTP server.

Just like Git, FTP doesn’t turn clients into servers in the way that a peer-to-peer protocol does. So that’s why Git isn’t already the decentralized GitHub — you don’t know where anything’s stored, and even if you did, those machines aren’t running Git servers that you’re allowed to talk to. I think we can fix that.

Let’s GitTorrent a repo!

Let’s jump in with a demo of GitTorrent – that is, cloning a Git repository that’s hosted on BitTorrent:

1  λ git clone gittorrent://github.com/cjb/recursers
2  Cloning into 'recursers'...
3
4  Okay, we want to get: 5fbfea8de70ddc686dafdd24b690893f98eb9475
5
6  Adding swarm peer: 192.34.86.36:30000
7
8  Downloading git pack with infohash: 9d98510a9fee5d3f603e08dcb565f0675bd4b6a2
9
10 Receiving objects: 100% (47/47), 11.47 KiB | 0 bytes/s, done.
11 Resolving deltas: 100% (10/10), done.
12 Checking connectivity... done.

Hey everyone: we just cloned a git repository over BitTorrent! So, let’s go through this line by line.

Lines 1-2: Git actually has an extensible mechanism for network protocols built in. The way it works is that my git clone line gets turned into “run the git-remote-gittorrent command and give it the URL as an argument”. So we can do whatever we want to perform the actual download, and we’re responsible for writing git objects into the new directory and telling Git when we’re done, and we didn’t have to modify Git at all to make this work.

So git-remote-gittorrent takes it from here. First we connect to GitHub to find out what the latest revision for this repository is, so that we know what we want to get. GitHub tells us it’s 5fbfea8de...

Lines 4-6: Then we go out to the GitTorrent network, which is a distributed hash table just like BitTorrent’s, and ask if anyone has a copy of commit 5fbdea8de... Someone said yes! We make a BitTorrent connection to them. The way that BitTorrent’s distributed hash table works is that there’s a single operation, get_nodes(hash) which tells you who can send you content that you want, like this:

get_nodes('5fbfea8de70ddc686dafdd24b690893f98eb9475') =
  [192.34.86.36:30000, ...]

Now, in standard BitTorrent with “trackerless torrents”, you ask for the files that you want by their content, and you’d get them and be happy. But a repository the size of the Linux kernel has four million commits, so just receiving the one commit 5fbdea8de.. wouldn’t be helpful; we’d have to make another four million requests for all the other commits too. Nor do we want to get every commit in the repository every time we ‘git pull’. So we have to do something else.

Lines 8-12: Git has solved this problem — it has this “smart protocol format” for negotiating an exchange of git objects. We can think of it this way:

Imagine that your repository has 20 commits, 1-20. And the 15th commit is bbbb and the most recent 20th commit is aaaa. The Git protocol negotiation would look like this:

1> have aaaa
2> want aaaa
2> have bbbb

Because of the way the git graph works, node 1> here can look up where bbbb is on the graph, see that you’re only asking for five commits, and create you a “packfile” with just those objects. Just by a three-step communication.

That’s what we’re doing here with GitTorrent. We ask for the commit we want and connect to a node with BitTorrent, but once connected we conduct this Smart Protocol negotiation in an overlay connection on top of the BitTorrent wire protocol, in what’s called a BitTorrent Extension. Then the remote node makes us a packfile and tells us the hash of that packfile, and then we start downloading that packfile from it and any other nodes who are seeding it using Standard BitTorrent. We can authenticate the packfile we receive, because after we uncompress it we know which Git commit our graph is supposed to end up at; if we don’t end up there, the other node lied to us, and we should try talking to someone else instead.

So that’s what just happened in this terminal. We got a packfile made for us with this hash — and it’s one that includes every object because this is a fresh clone — we downloaded and unpacked it, and now we have a local git repository.

This was a git clone where everything up to the actual downloading of git objects happened as it would in the normal GitHub way. If GitHub decided tomorrow that it’s sick of being in the disks and bandwidth business, it could encourage its users to run this version of GitTorrent, and it would be like having a peer to peer “content delivery network” for GitHub, falling back to using GitHub’s servers in the case where the commits you want aren’t already present in the CDN.

Was that actually decentralized?

That’s some progress, but you’ll have noticed that the very first thing we did was talk to GitHub to find out which hash we were ultimately aiming for. If we’re really trying to decentralize GitHub, we’ll need to do much better than that, which means we need some way for the owner of a repository to let us know what the hash of the latest version of that repository is. In short, we now have a global database of git objects that we can download, but now we need to know what objects we want — we need to emulate the part of github where you go to /user/repo, and you know that you’re receiving the very latest version of that user’s repo.

So, let’s do better. When all you have is a hammer, everything looks like a nail, and my hammer is this distributed hash table we just built to keep track of which nodes have which commits. Very recently, substack noticed that there’s a BitTorrent extension for making each node be partly responsible for maintaining a network-wide key-value store, and he coded it up. It adds two more operations to the DHT, get() and put(), and put() gives you 1000 bytes per key to place a message into the network that can be looked up later, with your answer repeated by other nodes after you’ve left the network. There are two types of key — the first is immutable keys, which work as you might expect, you just take the hash of the data you want to store, and your data is stored with that hash as the key.

The second type of key is a mutable key, and in this case the key you look up is the hash of a public key to a crypto keypair, and the owner of that keypair can publish signed updates as values under that key. Updates come with a sequence number, so anytime a client sees an update for a mutable key, it checks if the update has a newer sequence number than the value it’s currently recorded, and it checks if the update is signed by the public key corresponding to the hash table key, which proves that the update came from the key’s owner. If both of those things are true then it’ll update to this newer value and start redistributing it. This has many possible uses, but my use for it is as the place to store what your repositories are called and what their latest revision is. So you’d make a local Git commit, push it to the network, and push an update to your personal mutable key that reflects that there’s a new latest commit. Here’s a code description of the new operations:

// Immutable key put
hash(value) = put({
  value: 'some data'
})

// Mutable key put
hash(key) = put({
  value: 'some data',
  key: key,
  seq: n
})

// Get
value = get(hash)

So now if I want to tell someone to clone my GitHub repo on GitTorrent, I don’t give them the github.com URL, instead I give them this long hex number that is the hash of my public key, which is used as a mutable key on the distributed hash table.

Here’s a demo of that:

λ git clone gittorrent://81e24205d4bac8496d3e13282c90ead5045f09ea/recursers

Cloning into 'recursers'...

Mutable key 81e24205d4bac8496d3e13282c90ead5045f09ea returned:
name:         Chris Ball
email:        chris@printf.net
repositories: 
  recursers: 
    master: 5fbfea8de70ddc686dafdd24b690893f98eb9475

Okay, we want to get: 5fbfea8de70ddc686dafdd24b690893f98eb9475

Adding swarm peer: 192.34.86.36:30000

Downloading git pack with infohash: 9d98510a9fee5d3f603e08dcb565f0675bd4b6a2

Receiving objects: 100% (47/47), 11.47 KiB | 0 bytes/s, done.
Resolving deltas: 100% (10/10), done.
Checking connectivity... done.

In this demo we again cloned a Git repository over BitTorrent, but we didn’t need to talk to GitHub at all, because we found out what commit we were aiming for by asking our distributed hash table instead. Now we’ve got true decentralization for our Git downloads!

There’s one final dissatisfaction here, which is that long strings of hex digits do not make convenient usernames. We’ve actually reached the limits of what we can achieve with our trusty distributed hash table, because usernames are rivalrous, meaning that two different people could submit updates claiming ownership of the same username, and we wouldn’t have any way to resolve their argument. We need a method of “distributed consensus” to give out usernames and know who their owners are. The method I find most promising is actually Bitcoin’s blockchain — the shared consensus that makes this cryptocurrency possible.

The deal is that there’s a certain type of Bitcoin transaction, called an OP_RETURN transaction, that instead of transferring money from one wallet to another, leaves a comment as your transaction that gets embedded in the blockchain forever. Until recently you were limited to 40 bytes of comment per transaction, and it’s been raised to 80 bytes per transaction as of Bitcoin Core 0.11. Making any Bitcoin transaction on the blockchain I believe currently costs around $0.08 USD, so you pay your 8 cents to the miners and the network in compensation for polluting the blockchain with your 80 bytes of data.

If we can leave comments on the blockchain, then we can leave a comment saying “Hey, I’d like the username Chris, and the hash of my public key is <x>“, and if multiple people ask for the same username, this time we’ll all agree on which public key asked for it first, because blockchains are an append-only data structure where everyone can see the full history. That’s the real beauty of Bitcoin — this currency stuff is frankly kind of uninteresting to me, but they figured out how to solve distributed consensus in a robust way. So the comment in the transaction might be:

@gittorrent!cjb!81e24205d4bac8496d3e13282c90ead5045f09ea

(@service!username!pubkey)

It’s interesting, though — maybe that “gittorrent” at the beginning doesn’t have to be there at all. Maybe this could be a way to register one username for every site that’s interested in decentralized user accounts with Bitcoin, and then you’d already own that username on all of them. This could be a separate module, a separate software project, that you drop in to your decentralized app to get user accounts that Just Work, in Python or Node or Go or whatever you’re writing software in. Maybe the app would monitor the blockchain and write to a database table, and then there’d be a plugin for web and network service frameworks that knows how to understand the contents of that table.

It surprised me that nothing like this seems to exist already in the decentralization community. I’d be happy to work on a project like this and make GitTorrent sit on top of it, so please let me know if you’re interested in helping with that.

By the way, username registration becomes a little more complicated than I just said, because the miners could see your message, and decide to replace it before adding it to the blockchain, as a registration of your username to them instead of you. This is the equivalent of going to a domain name registrar and typing the domain you want in their search box to see if it’s available — and at that moment of your search the registrar could turn around and register it for themselves, and then tell you to pay them a thousand bucks to give it to you. It’s no good.

If you care about avoiding this, Bitcoin has a way around it, and it works by making registration a two-step process. Your first message would be asking to reserve a username by supplying just the hash of that username. The miners don’t know from the hash what the username is so they can’t beat you to registering it, and once you see that your reservation’s been included in the blockchain and that no-one else got a reservation in first, you can send on a second comment that says “okay, now I want to use my reservation token, and here’s the plain text of that username that I reserved”. Then it’s yours.

(I didn’t invent this scheme. There’s a project called Blockname, from Jeremie Miller, that works in exactly this way, using Bitcoin’s OP_RETURN transaction for DNS registrations on bitcoin’s blockchain. The only difference is that Blockname is performing domain name registrations, and I’m performing a mapping from usernames to hashes of public keys. I’ve also just been pointed at Blockstore, which is extremely similar.)

So to wrap up, we’ve created a global BitTorrent swarm of Git objects, and worked on user account registration so that we can go from a user experience that looks like this:

git clone gittorrent://github.com/cjb/foo

to this:

git clone gittorrent://81e24205d4bac8496d3e13282c90ead5045f09ea/foo

to this:

git clone gittorrent://cjb/foo

And at this point I think we’ve arrived at a decentralized replacement for the core feature of GitHub: finding and downloading Git repositories.

Closing thoughts

There’s still plenty more to do — for example, this doesn’t do anything with comments or issues or pull requests, which are all very important aspects of GitHub.

For issues, the solution I like is actually storing issues in files inside the code repository, which gives you nice properties like merging a branch means applying both the code changes and the issue changes — such as resolving an issue — on that branch. One implementation of this idea is Bugs Everywhere.

We could also imagine issues and pull requests living on Secure Scuttlebutt, which synchronizes append-only message streams across decentralized networks.

I’m happy just to have got this far, though, and I’d love to hear your comments on this design. The design of GitTorrent itself is (ironically enough) on GitHub and I’d welcome pull requests to make any aspect of it better.

I’d like to say a few thank yous — first to Feross Aboukhadijeh, who wrote the BitTorrent libraries that I’m using here. Feross’s enthusiasm for peer-to-peer and the way that he runs community around his “mad science” projects made me feel excited and welcome to contribute, and that’s part of why I ended up working on this project.

I’m also able to work on this because I’m taking time off from work at the moment to attend the Recurse Center in New York City. This is the place that used to be called “Hacker School” and it changed its name recently; the first reason for the name change was that they wanted to get away from the connotations of a school where people are taught things, when it’s really more like a retreat for programmers to improve their programming through project work for three months, and I’m very thankful to them for allowing me to attend.

The second reason they decided to change their name because their international attendees kept showing up at the US border and saying “I’m here for Hacker School!” and.. they didn’t have a good time.

Finally, I’d like to end with a few more words about why I think this type of work is interesting and important. There’s a certain grand, global scale of project, let’s pick GitHub and Wikipedia as exemplars, where the only way to have the project be able to exist at global scale after it becomes popular is to raise tens of millions of dollars a year, as GitHub and Wikipedia have, to spend running it, hoarding disks and bandwidth in big data centers. That limits the kind of projects we can create and imagine at that scale to those that we can make a business plan for raising tens of millions of dollars a year to run. I hope that having decentralized and peer to peer algorithms allows us to think about creating ambitious software that doesn’t require that level of investment, and just instead requires its users to cooperate and share with each other.

Thank you all very much for listening.

(You can check out GitTorrent on GitHub, and discuss it on Hacker News. You could also follow me on Twitter.)

-1:-- Announcing GitTorrent: A Decentralized GitHub (Post cjb)--L0--C0--May 29, 2015 04:23 PM

punchagan: Say Howdy with Emacs!

Staying in touch with people is something I'm not very good at. Since I am not on popular (among my friends/family) networks – FB and Whatsapp – I don't even see random updates from people, to get some sense of being in touch.

I recently read some old posts by Sacha Chua and was inspired by how much code she had for contact management in her old blog posts. I was inspired by this post in particular to try and be more meticulous about how I stay in touch with people. Michael Fogleman blogged about his contact management work-flow using keepintouch. It seemed to do most of what I wanted, but I wanted this to be integrated with my org-contacts-db and I felt having native elisp code would make it easier to hook up email, chat, etc. to this.

I ended up writing a small utility called howdy to help me keep in touch with people. It currently has only a couple of features:

  • M-x howdy lets me update the last contacted timestamp for a contact.
  • Shows me contacts that I'm out of touch in the agenda, once I add the following snippet to an agenda file.
    * Howdy
      %%(howdy-howdy)
    

I also have a few hooks to hook up jabber messages and email to update the db. I've added them to howdy-hooks.el in case anybody else wants to use them. They can also be used as examples to write other hooks. Feel free to contribute other hooks or suggest improvements. The library also ships with a modest test suite, that will hopefully make it easier for others to contribute.

I'm looking forward to experimenting with this over the next few weeks and improving it. Hopefully, it'll help me keep in touch, better than I do now.

-1:-- Say Howdy with Emacs! (Post punchagan)--L0--C0--May 28, 2015 01:09 PM