Tom Breton: Emtest

Emtest

Previously

Some years back, I wrote a testing framework for emacs called Emtest. It lives in a repo hosted on Savannah, mirrored here, doc'ed here.

Cucumber

Recently a testing framwork called Cucumber came to my attention. I have multiple reactions to it:

But they left important parts unadopted

But they didn't really adopt table testing in its full power. There are a number of things I have found important for table-driven testing that they apparently have not contemplated:

N/A fields
These are unprovided fields. A test detects them, usually skipping over rows that lack a relevant field. This is more useful than you might think. Often you are defining example inputs to a function that usually produces output (another field) but sometimes ought to raise error. For those cases, you need to provide inputs but there is nothing sensible to put in the output field.
Constructed fields
Often you want to construct some fields in terms of other fields in the same row. The rationale above leads directly there.
Constructed fields II
And often you want to construct examples in terms of examples that are used in other tests. You know those examples are right because they are part of working tests. If they had some subtle stupid mistake in them, it'd have already shown up there. Reuse is nice here.
Persistent fields
This idea is not originally mine, it comes from an article on Gamasutra1. I did expand it a lot, though. The author looked for a way to test image generation (scenes) and what he did was at some point, capture a "good" image the same image generator. Then from that point on, he could automatically compare the output to a known good image.
  • He knew for sure when it passed.
  • When the comparison failed, he could diff the images and see where and how badly; it might be unnoticeable dithering or the generator might have omitted entire objects or shadows.
  • He could improve the reference image as his generator got better.

I've found persistent fields indispensable. I use them for basically anything that's easier to inspect that it is to write examples of. For instance, about half of the Klink tests use it.

They didn't even mention me

AFAICT neither Cucumber nor Gherkin credits me at all. Maybe they're honestly unaware of the lineage of the ideas they're using. Still, it gets tiresome not getting credit for stuff that AFAICT I invented and gave freely to everybody in the form of working code.

They don't use TESTRAL or anything like it.

TESTRAL is the format I defined for reporting tests. Without going into great detail, TESTRAL is better than anything else out there. Not just better than the brain-dead ad hoc formats, but better than TestXML.

BDD is nice

Still, I think they have some good ideas, especially regarding Behavior Driven Development. IMO that's much better than Test-Driven Development2.

In TDD, you're expected to test down to the fine-grained units. I've gone that route, and it's a chore. Yes, you get a nice regression suite, but pretty soon you just want to say "just let me write code!"

In constrast, where TDD is bottom-up, BDD is top-down. Your tests come from use-cases (which are structured the way I structure inline docstrings in tests, which is nice, and just how much did you Cucumber guys borrow?) BDD looks like a good paradigm for development.

Not satisfied with Emtest tables, I replaced them

But my "I was first" notwithstanding, I'm not satisfied with the way I made Emtest do tables. At the time, because nobody anywhere had experience with that sort of thing, I adopted the most flexible approach I could see. This was tag-based, an idea I borrowed from Carsten Dominick's org-mode3.

However, over the years the tag-based approach has proved too powerful.

  • It takes a lot of clever code behind the scenes to make it work.
  • Maintaining that code is a PITA. Really, it's been one of the most time-consuming parts of Emtest, and always had the longest todo list.
  • In front of the scenes, there's too much power. That's not as good as it sounds, and led to complex specifications because too many tags needed management.
  • Originally I had thought that a global tag approach would work best, because it would make the most stuff available. That was a dud which I fixed that years ago.

So, new tables for Emtest

So this afternoon I coded a better table package for Emtest. It's available on Savannah right now; rather, the new Emtest with it is available. It's much simpler to use:

emt:tab:make
define a table, giving arguments:
docstring
A docstring for the entire table.
headers
A list of column names. For now they are simply symbols, later they may get default initialization forms and other help
rows
The remaining arguments are rows. Each begins with a namestring.
emt:tab:for-each-row
Evaluate body once for each row, with the row bound to var-sym
emt:tab
Given a table row and a field symbol, get the value of the respective field

I haven't added Constructed fields or Persistent fields yet. I will when I have to use them.

Also added foreign-tester support

Emtest also now supports foreign testers. That is, it can communicate with an external process running a tester, and then report that tester's results and do all the bells and whistles (persistence, organizing results, expanding and collapsing them, point-and-shoot launching of tests, etc) So the external tester can be not much more than "find test, run test, build TESTRAL result".

It communicates in Rivest-style canonical s-expressions, which is as simple a structured format as anything ever. It's equally as expressive as XML and there exist interconverters.

I did this with the idea of using it for the Functional Reactive Programming stuff I was talking about before, if in fact I make a test implementation for it (Not sure).

And renamed to tame the chaos

At one time I had written Emtest so that the function and command prefixes were all modular. Originally they were written-out, like emtest/explorer/fileset/launch. That was huge and unwieldy, so I shortened their prefixes to module unique abbreviations like emtl:

But when I looked at it again now, that was chaos! So now

  • Everything the user would normally use is prefixed emtest
    • Main entry point emtest
    • Code-editing entry point emtest:insert
    • "Panic" reset command emtest:reset
    • etc
  • Everything else is prefixed emt: followed by a 2 or 3 letter abbreviation of its module.

I haven't done this to the define and testhelp modules, though, since the old names are probably still in use somewhere.

Footnotes:

1 See, when I borrow ideas, I credit the people it came from, even if I have improved on it. Can't find the article but I did look; it was somewhat over 5 years ago, one of the first big articles on testing there.

2 Kent Beck's. Again, crediting the originator.

3 Again credit where it's due. He didn't invent tags, of course, and I don't know who was upstream from him wrt that.

-1:-- Emtest (Post Tehom (noreply@blogger.com))--L0--C0--May 19, 2012 02:52 AM

Raimon Grau: imap for emacs (insert templates on emacs)

You just miss good old :imap for your temporary configuration of your editor (remember vim?).  When you know you'll be going to insert similar strings over and over.

And you love the way vim allowed you to do crazy stuff inside a map.  But now you're in emacs, so let's see how we can scratch your itch.

My concrete case was inserting strings like (?* ?x) where x is a parameter.

  •  Skeletons. Skeletons can do that and much more. it's a classical template inserter.  For example, you can setup a skeleton of the desired output and interleave str symbol, that will be replaced for the input you'enter.
     
    Skeleton can do more sophisticated things, like inserting multiple lines while user keeps entering data.  I can imagine myself using this when entering lots of data with some desired format.

  • Yasnippets. Yasnippets are skeletons on steroids. Sad thing is that there's no direct way to bypass the file used to define snippets. I guess you can open a new buffer, put it into yasnippet-mode, and c-c c-c to activate the binding for just this session, but maybe it's too much of a hassle.

  • defun+insert.  The dumb way to do it, but the most straightforward if you tend to forget rarely used functionalities like skeletons.  Just define an interactive function that inserts your shit there, and call it at will.  That's the option I go when I don't wanna spend a minute looking for skeleton docs.
Then to call the functions/skeletons, you have different ways, if you want to global-set-key, or even just set the key in insert mode (using evil), reminding of imap style,

I'd love to hear other solutions on that same issue. Any suggestions?
-1:-- imap for emacs (insert templates on emacs) (Post Raimon Grau (noreply@blogger.com))--L0--C0--May 18, 2012 02:59 PM

Ivan Kanis: GTD Weekly Review

GTD enforces a weekly review. I used to do it with great zeal and it would take about two hours. I did it in two phases: refile things in my "in" box to folders and then go through each folders and classify action in sub folders.

I have streamlined the process the following ways. First I am using only folders. This saves classifying time.

Secondly I am using less next actions. I have about 6 next actions actions ongoing. I used to have one for each sub sections and was faced with a mind numbing list of next actions.

Lastly I am doing personal and professional actions on different days. Doing it all at once was taking too much time. The context didn't feel right either. I didn't feel like doing professional stuff at home and vice versa.

I am also breaking the two minutes rule. If I feel like doing it I do it straight away and then get back to reviewing later.

All in all I must be spending 30 minutes a week on reviewing. I feel like I am wasting less time and staying more in focus.

-1:-- GTD Weekly Review (Post Ivan Kanis)--L0--C0--May 17, 2012 12:00 AM

Phil Hagelberg: in which three programming methods are compared

There are, roughly speaking, three ways to develop large user-facing programs, which we will refer to here as 0) the Unix way, 1) the Emacs way, and 2) the wrong way.

The Unix way has been expounded upon at length many times. It consists of many small programs which communicate by sending text over pipes or using the occasional signal. If you can get away with using this model, the simplicity and universality it offers is very compelling. You hook into a rich ecosystem of text-based processes with a long history of well-understood conventions. Anyone can tie into it with programs written in any language. But it's not well-suited for everything: sometimes the requirement of keeping each part of the system in its own process is too high a price to pay, and sometimes circumstances require a richer communication channel than just a stream of text.

This is where the Emacs way shines. A small core written in a low-level language implements a higher-level language in which most of the rest of the program is implemented. Not only does the higher-level language ease the development of the trickier parts of the program, but it also makes it much easier to implement a good extension system since extensions are placed on even ground with the original program itself. I wrote about this in an earlier post on the live-development model Emacs offers:

If you have to use some tacked-on "plugin mechanism" to customize it, then you’re going to be limited at the very least by the imagination of the author of the plugin mechanism; only the things he thought you would want to do with it are doable. But if you’re using the exact same tools as the original authors were using to write the program in the first place, you can bet they put all their effort into making that a seamless, powerful experience, and you'll be able to access things on an entirely new level.

-in which a subject is attempted to be approached objectively, though such a thing is actually impossible

It's worth noting that this is the model under which Mozilla is developed. The core Mozilla platform is implemented mostly in a gnarly mash of C++, but applications like Firefox and Conkeror are primarily written in JavaScript, as are extensions. Following the Emacs way accounted for Firefox's continuing popularity even back when it was getting trounced by competitors in terms of JavaScript performance. Chrome's extension mechanism is laughably simplistic in comparison.

Finally for completeness sake, the wrong way is simply to write a large monolithic application in a low-level language, usually C++. Often half-hearted attempts at extension mechanisms are bolted on to programs developed this way, (usually in order to check off another box on a features list) but they are invariably frustrating and primitive and don't end up offering extension developers the same access to program internals that the developers of the original program itself have.

The Unix way makes particularly explicit the notion of composing small programs, but the Emacs way shines when a single runtime process plays host to a number of independent programs that can interact with each other gracefully. For instance, the Magit version control interface can run in the same Emacs instance as a SLIME session controlling a lisp project. They coexist in a complimentary way and compose together without interference. So rather than saying there are three ways to write large user-facing programs, it might be more accurate to say that there are zero good ways to write large user-facing programs and two ways to compose a number of small programs into a coherent system.

This is especially interesting to me right now since it has come to my attention that when it was rewritten in the transition from version 2 to version 3, GNOME has switched to the second way via an embedded JavaScript runtime, which means things are about to get very interesting.

-1:-- in which three programming methods are compared (Post Phil Hagelberg)--L0--C0--May 16, 2012 04:07 PM

Alex Schroeder: MANPATH

I’m using Emacs on Mac OSX.

Apparently the correct solution for using man and all the related tools is to make sure your /etc/man.conf file is correct. Mine was missing the following line:

MANPATH	/usr/local/man

You can ignore the rest of this page. :)

Thank you, Phil Hudson.

In my /.bashrc:

# MANPATH
# there's no MANPATH by default, and manpath(1) just prints /usr/share/man
if [ -z "$MANPATH" ]; then
    export MANPATH=/opt/local/man:/usr/local/man:/usr/local/share/man:/usr/X11R6/man:/usr/share/man
fi

In my /.emacs:

;; man
(unless (getenv "MANPATH")
  (setenv "MANPATH"
	  (with-temp-buffer
	    (insert-file-contents-literally "~/.bashrc")
	    (when (re-search-forward "MANPATH=\\(.*\\)" nil t)
	      (match-string 1)))))

And finally my little rebinding of C-h f for Perl mode works for modules as well:

(add-hook 'cperl-mode-hook
	  (lambda ()
	    (local-set-key (kbd "C-h f") 'cperl-perldoc)))

This calls perldoc which in turn calls man which uses MANPATH.

Tags: RSS

-1:-- MANPATH (Post)--L0--C0--May 16, 2012 11:59 AM

Ivan Kanis: Reverse Engineering The Sentinel

The Sentinel is a game I was very fond of in the 80s. I have started reverse engineering the ZX Spectrum version. An old friend of mine has helped me find the map generation at 0x6100-0x6ff.

-1:-- Reverse Engineering The Sentinel (Post Ivan Kanis)--L0--C0--May 16, 2012 12:00 AM

sachachua: Org-mode and habits

Org Mode is a personal information manager for the Emacs text editor. People have contributed a ton of useful features to it over the years, and the development shows no sign of slowing down. One of the features I’ve been playing around with is the ability to track habits.

Org habits are recurring tasks. For example, everyday, I want to:

  • take my vitamins
  • capture a quick note about the day, and
  • plan the next day

Every week, I want to:

  • talk to my mom
  • check the org-mode mailing list
  • write a weekly review and plan the next week
  • clear and reorganize my belt bag
  • clear my inbox
  • write a bunch of blog posts
  • back up my computer

Once a month, I want to:

  • update the topical index for my blog
  • review and uninstall programs
  • balance my books and update my budget
  • review the past month and plan the next
  • check the library for new books

Org habits let me manage my task list without cluttering future days with tasks. The Org agenda view displays habits that are due today, indicating consistency with colour. In particular, it shows overdue days in red, so you can get the Seinfeld-esque pleasure/commitment-device of not breaking the chain.

Here’s a view from Sunday:

2 days-agenda (W19-W20):
Sunday     13 May 2012
               8:00...... ----------------
              10:00...... ----------------
              12:00...... ----------------
              14:00...... ----------------
              15:57...... now - - - - - - - - - - - - - - - - - - - - - - - - -
              16:00...... ----------------
              18:00...... ----------------
  organizer:  22:00...... TODO Capture a one-sentence note                                           !      
  organizer:  22:00...... TODO Plan the next day                                    ** ***** ** *****!      
  organizer:  Scheduled:  TODO Make a list of recipes I want to learn
  organizer:  Scheduled:  TODO Write a bunch of blog posts             :writing:
  organizer:  Scheduled:  TODO Set up WordPress as my backup system
Monday     14 May 2012 W20
  organizer:  Scheduled:  TODO Build Emacs interface so that I can have Org automatically switch my tasks

To use Org habits, customize org-modules and enable the habit module. To set something as a habit, use C-c C-x p (org-set-property) to set the STYLE property to habit. For more information, you should definitely check out the Org manual’s section on habits.

Yay Emacs and the people who contribute to it!

Read the original or check out the comments on: Org-mode and habits (Sacha Chua's blog)

-1:-- Org-mode and habits (Post Sacha Chua)--L0--C0--May 15, 2012 12:00 PM

Got Emacs?: Some funny programming quotes

I'm auditing the CS212 course on udacity and I see this page.  My favourite funny and deadpan ones are.

"It scarcely needs mentioning that, when you have a decorator inside a decorator, it's an interior decorator." -- Neale Morison

On A**N + B**N == C**N and N > 1:
"Now, if you can find a solution for this when N is greater than 2, I want you to let me know." -- Peter Norvig


-1:-- Some funny programming quotes (Post sivaram (noreply@blogger.com))--L0--C0--May 13, 2012 03:53 PM

Bryan Murdock: Mercurial Now Has commit --amend

Mercurial 2.2.1 is out, and, among other new features and improvements, the commit command now has a --amend option. Git has had this for a while. Before 2.2 you could get the same functionality in Mercurial by using the mq extension, but it took at least 3 commands (qimport, qrefresh, qfinish). It's nice that you can do it with just one command now. Mercurial's relatively new phases come into play with --amend in that, by default, they will prevent you from amending a commit that has been pushed to or pulled from a remote repository. It's a nice little safety net to have, and of course you can override that behavior if you need.

It's nice to see this incredibly capable and easy-to-use tool get even better.
-1:-- Mercurial Now Has commit --amend (Post Bryan (noreply@blogger.com))--L0--C0--May 10, 2012 03:32 PM

sachachua: Maintaining a manual topical index for my blog using Emacs

I’ve been blogging for almost ten years. I started with notes from my university classes and snippets of open source code, and became comfortable enough to share decisions I’m puzzling through and things I’m learning about life. There’s a lot of stuff in my archive, and I want to be able to review things again.

Categories would probably make this easier, but I use categories liberally and sometimes inconsistently. I use them like tags, quick keywords that I add so that people might explore a category and bump into other posts. I probably should split it out so that I assign posts to one category and leave everything else as tags. Someday.

In the meantime, it’s easy enough to maintain a manual topical index of my blog posts, and it’s a good opportunity to review what I’ve been writing as well.

I use Emacs Org Mode to manage a large text file divided into headings. Every month, I copy a list of titles into my topical index. I hacked Org-friendly output into my WordPress theme – you can see April’s blog posts as an example (sachachua.com/blog/2012/04/?org=1). I manually organize the list items under different headings, splitting off new headings when I can see a pattern. Working with two windows viewing the same buffer makes it easy to move information around, and org-refile is handy too. I use a checklist structure so that Org can automatically update the number of posts under each heading (C-u M-x org-update-statistics-cookies). When I’m happy with the structure, I use org-publish-current-file to publish it using the settings I’ve configured. The files are in my public Dropbox folder, so they’re automatically published to the Web. It takes me about 10 minutes to add a month of posts to my index and publish the page.

I like seeing how much I’ve written about different topics, and it encourages me to write and organize more posts. Maybe the index might be handy for other people too!

Read the original or check out the comments on: Maintaining a manual topical index for my blog using Emacs (Sacha Chua's blog)

-1:-- Maintaining a manual topical index for my blog using Emacs (Post Sacha Chua)--L0--C0--May 09, 2012 12:00 PM

Aidan McQuay: Bloging from Emacs!

wooho it works!

-1:-- Bloging from Emacs! (Post Aidan McQuay)--L0--C0--May 09, 2012 08:51 AM

Phil Hagelberg: in which korean hardware pleases

I just got a new laptop, and while I wouldn't post about it normally I've had a number of people ask me what I thought of it. For the last five years or so I've been a Thinkpad user exclusively, but the latest models of their lightweight X series have offered disappointing screens with 1366x786 being the only resolution option. Since my dignity prevents me from buying a laptop with fewer vertical pixels than my phone, I broadened my search this time around and came across the 13-inch Samsung Series 9, aka NP900X3B.

zuse

At first the most striking thing about it is its thin profile, but once you've started using it you realize it's the screen that really sets it apart. Samsung is the largest manufacturer of LCDs in the world, so it makes sense that the screen would be the distinguishing factor on their high-end models. The resolution is 1600x900, which I couldn't find in anything else smaller than 14 inches, but it's the 400 cd/m2 of brightness that really sets it apart. With summer right around the corner, this perfect for working outdoors.

While I'm pretty thrilled with the new machine, the thin profile forces some difficult compromises with the keyboard. It's of the standard chicklet design, which is a drag coming from the Thinkpad. The X200s Thinkpad I was using previously had a very comfortable response and depth to it that you don't find with chicklet keybords. The Samsung also lacks a trackpoint, drainage tray, and dedicated volume buttons. The X200s isn't appreciably heavier than the Samsung; (only ½ a pound more) but it is over twice as thick. Personally if it's light enough I don't find thickness to be a problem; I'd much rather have a comfortable keyboard than a 0.4-inch profile. But it does turn heads.

The speakers on ultraportable laptops like the X200s typically come across as a token effort to fill a feature checkbox, but the Samsung's are fairly respectable. I was pleased to see the lack of the hardware wifi kill switch that plagued me on my Thinkpad. It's also got a bigger selection of ports than you'd expect with the slim profile: USB 2, USB 3, micro-HDMI, combined mic/headphones, and Ethernet, though the latter must be used with the provided adapter.

I'm running Debian Wheezy on it because I read on a blog post that a newer kernel was needed for the wifi drivers, but apparently this was only the case with an earlier model of the Series 9 that used the Broadcom chipset; The NP900X3B uses an Intel chipset that has been well-supported for some time. Everything worked perfectly out of the box except the keys for adjusting the keyboard backlight. The camera, external monitor port, and multitouch trackpad work as you'd expect. The suspend functionality, which has traditionally lagged behind Mac OS X, resumes in a couple short seconds.

While I would love to see this same screen on a laptop with the luxurious keyboard, carbon fiber body, and replaceable battery of a Thinkpad, the Series 9 is quite slick and should keep me happy for a number of years.

-1:-- in which korean hardware pleases (Post Phil Hagelberg)--L0--C0--May 09, 2012 04:25 AM

Ben Simon: Got an iPad! Now What?

I've got a handful of clients who are targeting their sites towards iPads, so it made sense to pick one up to use for development/testing. (Technically, I had the idea to buy one, it took Shira to actually go out and make it happen!). So now I've got a brand new, super shiny, iPad in my hands.

The hardware is gorgeous. The user experience, while disorienting compared to Android or Windows, is clearly superb. My only problem is, other than using it as a test bed, what the heck should I use it for?

My first thought was travel. It sure is lightweight. But when I travel, I have pretty specific needs: (1) image backup/storage, (2) publishing photo and text to Blogger, (3) ssh capability and (4) some local development ability (emacs+subversion+sitecopy). I also do a bit of general web surfing and entertainment (watching Hulu, for example). My bought used on Craigslist netbook seems to be far more effective at all those tasks than an iPad. It doesn't seem obvious to me how I'd even get photos from DSLR into this guy.

Then I thought, I'll hand it off to Shira. She fiddled away. Her first thoughts were to use it as an eBook reader, a Skype platform and for watching TV (think Hulu and TV.com). It does in fact work as an eBook reader and for Skype, but so does her Galaxy S II - nearly just as well. And both Hulu and TV.com were a bust (at least the free versions of the sites). Oh, and she did setup the Epicurious.com recipe app, which has some promise.

Our little guy saw the iPad, and immediately said "oh, that has games on it." I then explained this one didn't, yet. But maybe using it as an education platform is the way to go.

I'm sure there's a killer use for the iPad -- I'm just not sure what it is. Want to help me out here? What do you find works better on the iPad than any other device?

-1:-- Got an iPad! Now What? (Post Ben Simon)--L0--C0--May 06, 2012 10:44 PM

Chris Ball: Letting Go

My attempt at Letting Go, by Andrew York:


Andrew York - Letting Go (youtube / download as WebM, CC-BY-SA 3.0)
-1:-- Letting Go (Post Chris Ball)--L0--C0--May 06, 2012 03:31 PM

Rodrigo Lazo: Disable Org-mode Babel evaluation confirmation per file

Today I started experimenting with Babel, part of org-mode that enables execution of code defined inside your org file. It’s great, although it’s taking me a little effort to get use to it.

Well, as with any mechanism that executes arbitrary code in your machine, it’s a security risk, so by default, org-mode ask you prompts you for confirmation before executing any block of code. Well, what if you are writing the org file and the code can be considered trustworthy, but you are asked every. single. time. whether it’s OK or not to execute that file? As I found in Irreal’s blog post, doing that is as easy as putting

# -*- org-confirm-babel-evaluate: nil -*-

at the beginning of your org-file. In retrospective, it’s the obvious solution. I always forget file vars.

-1:-- Disable Org-mode Babel evaluation confirmation per file (Post Rodrigo)--L0--C0--May 06, 2012 04:29 AM

Richard Riley: Javascript utilities

Playing with javascript lately I came across a super utility called jshint : and Dale Harvey has produced a super jshint-mode! For a javascript nOOb like me it's invaluable when combined with jslint (jshint can replace it, but you might like to turn it off at times then manually invoke jslint instead).

See : jshint and Javascript Lint

My javascript set up:-


(require 'js-beautify)

(add-to-list 'load-path "~/.emacs.d/jquery-doc")
(require 'jquery-doc)
(add-hook 'js2-mode-hook 'jquery-doc-setup)
(require 'flymake-jshint)
(add-hook 'js2-mode-hook
(lambda () (flymake-mode t)))

(add-to-list 'auto-mode-alist '("\\.js$" . js2-mode))

;; (add-to-list 'auto-mode-alist '("\\.js$" . js2-mode))
(add-hook 'js2-mode-hook '(lambda () (js-beautify)(define-key js2-mode-map (kbd "M-t") (lambda()(interactive)(js-beautify)))(define-key js2-mode-map [(shift f10)] (lambda()(interactive)(jslint-thisfile)))))

;; javascript lint
(defun jslint-thisfile ()
(interactive)
(compile (format "jsl -process %s" (buffer-file-name))))
(defun jslint-thisfile-test ()
(interactive)
(compile (format "jsl -process %s" (buffer-file-name))))

(provide 'rgr-javascript)
-1:-- Javascript utilities (Post Richard (noreply@blogger.com))--L0--C0--May 05, 2012 12:17 PM

Richard Riley: ido-mode and virtual buffers

Every now and again you come across something in Emacs and slap your forehead and say "why didn't I notice that before". ido-mode is one, but the setting ido-use-virtual-buffers really makes it. The concept is simple : ido-mode remembers all your past buffers so all you need to do is "C-x b" (ido-switch-buffer) and start to type the filename and ido will complete to it having remembered it from previous sessions. No need to use C-x C-f and possibly the ido find facility any more - just type the mainpart of the filename or part of it. Incredible cool time saver and very very useful.

From the Manual

ido-use-virtual-buffers is a variable defined in `ido.el'.
Its value is t
Original value was nil

Documentation:
If non-nil, refer to past buffers as well as existing ones.
Essentially it works as follows: Say you are visiting a file and
the buffer gets cleaned up by mignight.el. Later, you want to
switch to that buffer, but find it's no longer open. With
virtual buffers enabled, the buffer name stays in the buffer
list (using the `ido-virtual' face, and always at the end), and if
you select it, it opens the file back up again. This allows you
to think less about whether recently opened files are still open
or not. Most of the time you can quit Emacs, restart, and then
switch to a file buffer that was previously open as if it still
were.
This feature relies upon the `recentf' package, which will be
enabled if this variable is configured to a non-nil value.


my ido settings are currently:-

 '(ido-create-new-buffer (quote never))
'(ido-enable-flex-matching t)
'(ido-enable-last-directory-history nil)
'(ido-enable-regexp nil)
'(ido-max-directory-size 300000)
'(ido-max-file-prompt-width 0.1)
'(ido-use-filename-at-point (quote guess))
'(ido-use-url-at-point t)
'(ido-use-virtual-buffers t)
-1:-- ido-mode and virtual buffers (Post Richard (noreply@blogger.com))--L0--C0--May 05, 2012 12:16 PM

Irreal: Solution to the Two Challenges

The other day, I presented two Elisp coding challenges. I specified Elisp because Irreal readers tend to like Emacs related posts. The down side of that is that several of you worried about bignums and other Emacs limitations. That wasn’t my intent—I just thought the problems were interesting and might even come in handy someday in an interview.

In any event, here are my solutions. All the commenters who gave an answer for problem 1 (Given a list of integers from 1 to n but with one of the integers missing, write an efficient algorithm to find the missing integer.) gave the same O(n)/O(1) algorithm for time/space, namely just sum the integers up and subtract the sum from (n2 + n) / 2 to find the missing number.

The solution to the second problem (Given a list of integers from 1 to n but with one of the integers missing and another repeated, find an efficient algorithm to find the missing and repeated numbers.) is similar except the we sum the numbers and their squares. If we let delta1 be the difference between the sum of the integers and the sum of the integers from 1 to n, and delta2 be the difference between the sum of the squares of the integers and the sum of the squares of the integers from 1 to n, we have:

  • repeatedmissing = delta1 and
  • repeated2missing2 = delta2.

From there, some very easy algebra (the quadratic terms drop out) we get

  • repeated = (delta2 + delta12) / 2 delta1
  • missing = (delta2delta12) / 2 delta1

Now it’s easy to write an algorithm that’s linear in time and constant in space (modulo some increase in size for bignums if we’re using a language that supports them).

(require 'cl)
(defun solve (n repeat missing)
  (let ((numbers (cons repeat
                       (delete missing (loop for i from 1 to n collect i))))
        (sum-1-to-n (/ (* n (1+ n)) 2))
        (sumsq-1-to-n (/ (* n (1+ n) (+ n n 1)) 6))
        (units 0)
        (squares 0))
    (dolist (i numbers)
      (incf units i)
      (incf squares (* i i)))
    (let ((delta1 (- units sum-1-to-n))
          (delta2 (- squares sumsq-1-to-n)))
      (message "repeated number is %d, missing number is %d"
               (/ (+ delta2 (* delta1 delta1)) (* 2 delta1))
               (/ (- delta2 (* delta1 delta1)) (* 2 delta1))))))

I didn’t bother to randomize the list because I don’t make any use of the fact that it’s (almost) in numerical order. Some readers calculated the length of the list but I assumed that n was given as part of the problem. If not, it’s simple to count them up as we sum the numbers and their squares and then calculate sum-1-to-n and sumsq-1-to-n afterwards.

When we run solve, we get the expected answer

ELISP> (solve 100 35 78)
"repeated number is 35, missing number is 78"
ELISP>
-1:-- Solution to the Two Challenges (Post jcs)--L0--C0--May 04, 2012 08:44 PM

emacspeak: Emacspeak 36.0 (EPubDog) Unleashed!

Emacspeak 36.0—EPubDog—Unleashed!

1 Emacspeak-36.0 (EPubDog) Unleashed!

1.1 For Immediate Release:

San Jose, Calif., (May 4, 2011) Emacspeak: Redefining Accessibility In The Era Of Cloud Computing —Zero cost of upgrades/downgrades makes priceless software affordable!

Emacspeak Inc (NASDOG: ESPK) --http://emacspeak.sf.net-- announces the immediate world-wide availability of Emacspeak 36.0 (EPubDog) —a powerful audio desktop for leveraging today's evolving data, social and service-oriented Web cloud.

1.2 Investors Note:

With several prominent tweeters expanding coverage of #emacspeak, NASDOG: ESPK has now been consistently trading over the net at levels close to that once attained by DogCom high-fliers—and as of October 2009 is trading at levels close to that achieved by once better known stocks in the tech sector.

1.3 What Is It?

Emacspeak is a fully functional audio desktop that provides complete eyes-free access to all major 32 and 64 bit operating environments. By seamlessly blending live access to all aspects of the Internet such as Web-surfing, blogging, social computing and electronic messaging into the audio desktop, Emacspeak enables speech access to local and remote information with a consistent and well-integrated user interface. A rich suite of task-oriented tools provides efficient speech-enabled access to the evolving service-oriented social Web cloud.

1.4 Major Enhancements:

  • Full EPub support: 📑
  • Websearch enhancements and wizards: 🐚
  • Speech-enables git interaction via magit: ℣
  • Speech-enabled support for finding things fast: 🚤
  • TTS enhancements: 🙊
  • Updated url templates for task-oriented web actions: ♅
  • SSH port forwarding support for TTS servers🔉
  • Updated support for the forthcoming Emacs 24 release.

Plus many more changes too numerous to fit in this margin ∞

1.5 Establishing Liberty, Equality And Freedom:

Never a toy system, Emacspeak is voluntarily bundled with all major Linux distributions. Though designed to be modular, distributors have freely chosen to bundle the fully integrated system without any undue pressure—a documented success for the integrated innovation embodied by Emacspeak. As the system evolves, both upgrades and downgrades continue to be available at the same zero-cost to all users. The integrity of the Emacspeak codebase is ensured by the reliable and secure Linux platform used to develop and distribute the software.

Extensive studies have shown that thanks to these features, users consider Emacspeak to be absolutely priceless. Thanks to this wide-spread user demand, the present version remains priceless as ever—it is being made available at the same zero-cost as previous releases.

At the same time, Emacspeak continues to innovate in the area of eyes-free social interaction and carries forward the well-established Open Source tradition of introducing user interface features that eventually show up in luser environments.

On this theme, when once challenged by a proponent of a crash-prone but well-marketed mousetrap with the assertion "Emacs is a system from the 70's", the creator of Emacspeak evinced surprise at the unusual candor manifest in the assertion that it would take popular idiot-proven interfaces until the year 2070 to catch up to where the Emacspeak audio desktop is today. Industry experts welcomed this refreshing breath of Courage Certainty and Clarity (CCC) at a time when users are reeling from the Fear Uncertainty and Doubt (FUD) unleashed by complex software systems backed by even more convoluted press releases.

1.6 Independent Test Results:

Independent test results have proven that unlike some modern (and not so modern) software, Emacspeak can be safely uninstalled without adversely affecting the continued performance of the computer. These same tests also revealed that once uninstalled, the user stopped functioning altogether. Speaking with Aster Labrador, the creator of Emacspeak once pointed out that these results re-emphasize the user-centric design of Emacspeak; "It is the user —and not the computer— that stops functioning when Emacspeak is uninstalled!".

1.6.1 Note from Aster,Bubbles and Tilden:

UnDoctored Videos Inc. is looking for volunteers to star in a video demonstrating such complete user failure.

1.7 Obtaining Emacspeak:

Emacspeak can be downloaded from Google Code Hosting —see http://code.google.com/p/emacspeak/ You can visit Emacspeak on the WWW at http://emacspeak.sf.net. You can subscribe to the emacspeak mailing list emacspeak@cs.vassar.edu by sending mail to the list request address emacspeak-request@cs.vassar.edu. The EPubDog release is at http://emacspeak.googlecode.com/files/emacspeak-35.0.tar.bz2. The latest development snapshot of Emacspeak is available via Subversion from Google Code Hosting at http://emacspeak.googlecode.com/svn/trunk/

1.8 History:

Emacspeak 36.0 enhances the audio desktop with many new tools including full EPub support — hence the name EPubDog. Emacspeak 35.0 is all about teaching a new dog old tricks — and is aptly code-named HeadDog in honor of our new Press/Analyst contact. emacspeak-34.0 (AKA Bubbles) established a new beach-head with respect to rapid task completion in an eyes-free environment. Emacspeak-33.0 AKA StarDog brings unparalleled cloud access to the audio desktop. Emacspeak 32.0 AKA LuckyDog continues to innovate via open technologies for better access. Emacspeak 31.0 AKA TweetDog — adds tweeting to the Emacspeak desktop. Emacspeak 30.0 AKA SocialDog brings the Social Web to the audio desktop—you cant but be social if you speak! Emacspeak 29.0—AKAAbleDog—is a testament to the resilliance and innovation embodied by Open Source software—it would not exist without the thriving Emacs community that continues to ensure that Emacs remains one of the premier user environments despite perhaps also being one of the oldest. Emacspeak 28.0—AKA PuppyDog—exemplifies the rapid pace of development evinced by Open Source software. Emacspeak 27.0—AKA FastDog—is the latest in a sequence of upgrades that make previous releases obsolete and downgrades unnecessary. Emacspeak 26—AKA LeadDog—continues the tradition of introducing innovative access solutions that are unfettered by the constraints inherent in traditional adaptive technologies. Emacspeak 25 —AKA ActiveDog —re-activates open, unfettered access to online information. Emacspeak-Alive —AKA LiveDog —enlivens open, unfettered information access with a series of live updates that once again demonstrate the power and agility of open source software development. Emacspeak 23.0 -- AKA Retriever—went the extra mile in fetching full access. Emacspeak 22.0 —AKA GuideDog —helps users navigate the Web more effectively than ever before. Emacspeak 21.0 —AKA PlayDog —continued the Emacspeak tradition of relying on enhanced productivity to liberate users. Emacspeak-20.0 —AKA LeapDog —continues the long established GNU/Emacs tradition of integrated innovation to create a pleasurable computing environment for eyes-free interaction. emacspeak-19.0 —AKA WorkDog— is designed to enhance user productivity at work and leisure. Emacspeak-18.0 —code named GoodDog— continued the Emacspeak tradition of enhancing user productivity and thereby reducing total cost of ownership. Emacspeak-17.0 —code named HappyDog— enhances user productivity by exploiting today's evolving WWW standards. Emacspeak-16.0 —code named CleverDog— the follow-up to SmartDog— continued the tradition of working better, faster, smarter. Emacspeak-15.0 —code named SmartDog—followed up on TopDog as the next in a continuing a series of award-winning audio desktop releases from Emacspeak Inc. Emacspeak-14.0 —code named TopDog—was the first release of this millennium. Emacspeak-13.0 —codenamed YellowLab— was the closing release of the 20th. century. Emacspeak-12.0 —code named GoldenDog— began leveraging the evolving semantic WWW to provide task-oriented speech access to Webformation. Emacspeak-11.0 —code named Aster— went the final step in making Linux a zero-cost Internet access solution for blind and visually impaired users. Emacspeak-10.0 —(AKA Emacspeak-2000) code named WonderDog— continued the tradition of award-winning software releases designed to make eyes-free computing a productive and pleasurable experience. Emacspeak-9.0 —(AKA Emacspeak 99) code named BlackLab— continued to innovate in the areas of speech interaction and interactive accessibility. Emacspeak-8.0 —(AKA Emacspeak-98++) code named BlackDog— was a major upgrade to the speech output extension to Emacs.

Emacspeak-95 (code named Illinois) was released as OpenSource on the Internet in May 1995 as the first complete speech interface to UNIX workstations. The subsequent release, Emacspeak-96 (code named Egypt) made available in May 1996 provided significant enhancements to the interface. Emacspeak-97 (Tennessee) went further in providing a true audio desktop. Emacspeak-98 integrated Internetworking into all aspects of the audio desktop to provide the first fully interactive speech-enabled WebTop.

About Emacspeak:


Originally based at Cornell (NY) http://www.cs.cornell.edu/home/raman —home to Auditory User Interfaces (AUI) on the WWW— Emacspeak is now maintained on GoogleCode --http://code.google.com/p/emacspeak —and Sourceforge —http://emacspeak.sf.net. The system is mirrored world-wide by an international network of software archives and bundled voluntarily with all major Linux distributions. On Monday, April 12, 1999, Emacspeak became part of the Smithsonian's Permanent Research Collection on Information Technology at the Smithsonian's National Museum of American History.

The Emacspeak mailing list is archived at Vassar —the home of the Emacspeak mailing list— thanks to Greg Priest-Dorman, and provides a valuable knowledge base for new users.

2 Press/Analyst Contact: Tilden Labrador

Going forward, Tilden acknowledges his exclusive monopoly on setting the direction of the Emacspeak Audio Desktop, and promises to exercise this freedom to innovate and her resulting power responsibly (as before) in the interest of all dogs.

**About This Release:


Windows-Free (WF) is a favorite battle-cry of The League Against Forced Fenestration (LAFF). —see http://www.usdoj.gov/atr/cases/f3800/msjudgex.htm for details on the ill-effects of Forced Fenestration.

CopyWrite )C( Aster and Hubbell Labrador. All Writes Reserved. HeadDog (DM), LiveDog (DM), GoldenDog (DM), BlackDog (DM) etc., are Registered Dogmarks of Aster, Hubbell and Tilden Labrador. All other dogs belong to their respective owners.

Date: 2012-05-02 14:21:43 PDT

Author: T.V Raman

Org version 7.8.09 with Emacs version 24

Validate XHTML 1.0
-1:-- Emacspeak 36.0 (EPubDog) Unleashed! (Post T. V. Raman (noreply@blogger.com))--L0--C0--May 04, 2012 01:26 AM

Random Sample: Javascript completions in Emacs using CEDET and Firefox/mozrepl

OK, let's just directly start with setting this up, because if I talk about CEDET too much, I'm afraid you'll just stop reading. So I will describe how you can get nice completions for Javascript coding from a running Firefox, using CEDET and mozrepl. And yes, this little tutorial will also deal with the actual CEDET installation, because contrary to what you've might read elsewhere, it really isn't that hard. You will need at least Emacs 23, though - we do not support Emacs22 anymore.

So, first step: Install the CEDET development version.

  • Get the development code from CEDET. Our primary repository can be accessed through Bazaar:
    bzr checkout bzr://cedet.bzr.sourceforge.net/bzrroot/cedet/code/trunk cedet
    

    But there's also a git mirror at

    git://git.randomsample.de/cedet.git
    

    (can also be accessed via http)

  • Enter the toplevel cedet directory and call 'make'.
  • Open your init file (.emacs) and add the following lines:
    ;; Load CEDET
    ;; This should be near the top of your init file, so that this can
    ;; really replace the CEDET that ships with Emacs proper.
    (load-file "/home/foo/cedet/cedet-devel-load.el")
    
    ;; Add further minor-modes to be enabled by semantic-mode.
    ;; See doc-string of `semantic-default-submodes' for other things
    ;; you can use here.
    (add-to-list 'semantic-default-submodes 'global-semantic-idle-summary-mode)
    (add-to-list 'semantic-default-submodes 'global-semantic-idle-completions-mode)
    
    ;; Enable Semantic
    (semantic-mode 1)
    
    
  • Restart Emacs.

Congratulations, you've just installed CEDET. If you got problems with these steps, please report them in the comments or better through the CEDET-devel mailing list. We just completely switched to a new development branch for better merging with Emacs, so it might be entirely possible we have missed something.

Everything's working so far? Then let's turn to the Javascript part.

  • Install the mozrepl extension from

    https://addons.mozilla.org/firefox/addon/mozrepl/

  • Restart Firefox and start Mozrepl from its menu entry under "Tools".
  • You should now be able to connect to port 4242 (try "telnet localhost 4242" or "nc localhost 4242", which should show you the mozrepl prompt).
  • Now let's turn to Emacs. Open a file "test.html" and copy&paste the following contents:
    <!doctype html>
    <html>
    <head>
      <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.5/jquery.min.js"></script>
      <script src="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8/jquery-ui.min.js"></script>
    </head>
    <body>
    This is just a test.
    </body>
    </html>
    

    As you can see, this is basically a HTML5 file that loads the jQuery framework from googleapis.com.

  • Save the file and do
    M-x semanticdb-mozrepl-activate RET
    

    Then, enter the URL "file:///<FULL PATH>/test.html". CEDET's Semantic will then open a connection to mozrepl and load the file in Firefox. After this is done, Emacs should say "Finished activating mozrepl database for <URL>", and Firefox should show you the above test.html file.

  • Now create a new file "foo.js". Emacs should automatically switch to javascript-mode (if you configured Emacs to use another mode, it might not work). Now simply enter
    $.a
    

    and wait for a second or two. Semantic should then start to give you completions on the jQuery object '$' through the so called "ghost text" style.

    You can cycle through the completions by pressing TAB and choose one with RET. If you've never dealt with jQuery, just type

    document.get
    

    and wait for a second, and more familiar stuff should appear.

OK, if this is working so far, you are actually finished with the basic setup! Everything what follows now is dealing with the different types of completion interfaces you can use.

What you've already seen is the so called "idle completion" from Semantic, which by default uses this special kind of "ghost text" which isn't too intrusive so that it should not interfere with typing. However, you're probably more used to the typical popup-tooltips, which are also available. But first, let me say right off the bat that all completion mechanisms in CEDET are pretty basic. CEDET was always intended to be more of a framework for application developers, and luckily there are now at least two very nice completion frameworks which support CEDET, but have to be installed separately. I'll deal with those two in a minute, but first let's look at the ones you can use right away:

  • M-x semantic-ia-complete-symbol-menu: This will display a menu with the possible completions. You can choose one by pressing up/down and RET. This is an older demonstration function for completion in Semantic.

  • M-x semantic-complete-analyze-inline: This is the newer and more modern interface for doing completions in CEDET. You can define different completion styles by doing
    M-x customize-variable RET semantic-complete-inline-analyzer-displayor-class RET
    

    In this screenshot, you see the tooltip display of the completions. In the same way, you can configure the display style for idle completion, which you've already witnessed above, through 'semantic-complete-inline-analyzer-idle-displayor-class' (Yeah, I know... TAB is your friend when calling customize on those things.).

If you are unhappy with these types of completions, you might want to take a look at the separate completion packages which are available. The two most popular are auto-complete and company-mode:

  • auto-complete is available at

    http://cx4a.org/software/auto-complete/

    It has a lot of features and is pretty snappy. It supports CEDET's Semantic and I happen to like it a lot.

    In the screenshot you could see further documentation for the 'ajax' function if mozrepl could supply any (possible for other languages like Emacs Lisp, though). However, its default configuration might be a bit "too much" for you. For example, you might want to restrict the number of backends it asks for completions. If you just set

      (setq-default ac-sources '(ac-source-semantic-raw))
    

    it will only query Semantic for completions and nothing else. Also, have a look at the variables "ac-delay" and "ac-auto-start", which define when auto-completion starts; the default behavior might be too intrusive for you, especially since getting completions from Semantic might not be fast enough for fluid typing (for large C++ projects it surely isn't).

  • company-mode is available via GNU ELPA (built into Emacs24, just do M-x list-packages), or from

    http://nschum.de/src/emacs/company-mode/

    In my opinion it is easier to configure and its defaults are less intrusive than auto-complete, but it also has less features. On Emacs24, it doesn't play well with an activated header-line (if you don't know what this is, you're not using them). Otherwise, it is a really nice framework for completions and works with CEDET out of the box. You just have to configure "company-semantic-modes", for example

    (setq company-semantic-modes '(c-mode c++-mode js-mode jde-mode java-mode emacs-lisp-mode))
    

    and make sure the buffer-local 'company-backend' includes 'company-semantic'. Please see the documentation for details. You can see an example in the first screenshot on this page.

OK, I think you have enough to get started. Please let me know how this stuff works for you. It is a pretty new thing and I'm not sure how stable this actually is; communicating with mozrepl is a bit of a challenge, but at least for me, it works pretty well so far.

Now that we've dealt with the interesting part, a few words on how it works, which should also highlight the problems this approach has.

Without the mozrepl connection, CEDET cannot help you much with Javascript. Yes, we have a grammar and it happily parses function definitions, but since Javascript is so incredibly dynamic and doesn't have proper classes, you cannot do much with static parsing. Since lately I have to deal more with Javascript to investigate all the new features like Webworkers, File API and so on, I first did what I guess most people do: install Firebug and use the console and try things in a REPL-like fashion. So I found myself hopping from Emacs to Firefox and back. I dimly remembered this little extension called mozrepl, which allows you to connect to a running Firefox session, and there's also a Emacs comint mode for it. It works OK, but it lacks the completion mechanisms from the Firebug console, so I did the next best thing and hooked it into CEDET.

CEDET has the concept of an "omniscience database". For example, Emacs itself is such a database when you're coding Emacs Lisp. As soon as you evaluate your functions, Emacs knows about them, including documentation, file location, and so on. For (client-side) Javascript, the Browser essentially is what Emacs is for Emacs Lisp: the framework in which the code runs. Through mozrepl, we can ask the Browser's Javascript engine for details on objects and the DOM.

There's one thing though: if you're switching tabs, the context in mozrepl will switch, too. There is however a built-in solution to this problem, but it is not activated by default since I'm not sure yet how well it works. The semanticdb-mozrepl backend can detect if you have switched to another tab and switch back automatically if it needs to query mozrepl. The variable that controls this behavior is "semanticdb-mozrepl-switch-tabs"; just see its docstring.

OK, I think this will suffice for now. Please let me know how this works for you, either in the comments or ask question on the CEDET-devel mailing list (which BTW is also available via Gmane as gmane.emacs.cedet).

-1:-- Javascript completions in Emacs using CEDET and Firefox/mozrepl (Post randomsample)--L0--C0--May 03, 2012 06:56 PM

Raimon Grau: elisp of the day: undefine a defun

Elisp is a pretty archaic lisp. I'm not refering to the problems everyone talk about (dynamic scope, One namespace to rule them all,...) but the aspect of being very rudimentary in its construction.

That's why there have been several attempts to substitute elisp for something else (let's rewrite emacs in erlang!). For me, it's doing pretty well, and now that we have lexical scope there, I think I can live with prefixed-and-very-long-method-names.

A positive thing of elisp being archaic is that the system is wide open to users, and most things are just obvious (once you were given the proper explanation, heh).

One place that doesn't work like that is when redefining a defun to defmethod. setfing the symbol-function to nil does not work. (It makes the old defun inaccessible, but doesn't free the symbol)

That's why fmakunbound exists. fmakunbound 'symbol,  and after that you'll be able to defmethod on that symbol.

For now, I haven't found any other real use case for that, but I can imagine unloading a package by removing all methods it previously defined.  That's not much in the culture of emacs though (never seen any plugin that does that) 
-1:-- elisp of the day: undefine a defun (Post Raimon Grau (noreply@blogger.com))--L0--C0--May 03, 2012 07:54 AM

Got Emacs?: Ma Gnus v0.5 is released

The traditional May 1st release of Ma Gnus is out.  You can either use git to clone or download the latest release as a zip file. For a list of features implemented, see this post.


-1:-- Ma Gnus v0.5 is released (Post sivaram (noreply@blogger.com))--L0--C0--May 03, 2012 04:13 AM

Irreal: Two Elisp Challenges

I ran across a couple of nice interview questions and an interesting story over at Tanya Khovanova’s Math Blog. The two questions are:

  1. Given a list of integers from 1 to n but with one of the integers missing, write an efficient algorithm for finding the missing integer.
  2. Given a list of integers from 1 to n but with one of the integers missing and another integer repeated, write an efficient algorithm for finding the missing and repeated integers.

In both problems the numbers in the list are in no particular order.

So the challenge is to write the two algorithms in Emacs Lisp. The interviewer expected a O(n log n) algorithm for time for the second problem. Can you do better?

Don’t go over to Khnovanova’s blog until you have solved the problems because there are spoilers in the comments. After you solve the problem, though, be sure to read her story about “hiring the smartest people in the world.”

-1:-- Two Elisp Challenges (Post jcs)--L0--C0--May 02, 2012 07:10 PM

Flickr tag 'emacs': Presentación Emacs

ooscarr posted a photo:

Presentación Emacs

Confundí Emacs con Latex todo el tiempo y al ver la presentación me pegué el alcachofazo.

Dijo que subiría las diapositivas, me las piden todo el tiempo.

-1:-- Presentación Emacs (Post ooscarr (nobody@flickr.com))--L0--C0--May 01, 2012 01:56 PM

Alex Bennée: Now a Windows user!

I’ve finally buckled and now have a Windows laptop. It was mainly forced on me by the need to have some sort of access to the intranet during the large number of meetings I’m now involved in at work. I have to say the experience has been enlightening, especially seeing all the hacky stuff that has to be done to get things working under Windows. For example by default I couldn’t connect to any https (SSL) pages. Luckily I can just hand the laptop back to IT to fix it so I don’t have to scratch my head too much.

I’ve had only sporadic luck getting Emacs up and running on it though. Having been running the tip-of-tree release on my Linux workstation so long going back to Emacs 23 has been a bit of a retrograde step. It doesn’t help there are multiple suggestions for installation. I’ve been trying to get EmacsW32 working but I’ve run into problems on start-up. So far I’ve been unable to fix the issue as the paths Emacs references don’t seem to show up in the system file browser. This seems to be the only avenue by which I can fix the permissions it’s complaining about.

Getting my init file onto the system and in the right place has also proved to be more complex than it should have been (there seems to be two “HOME” directories, one under a Roaming title). The Windows shell has finally gained completion but it’s still a shadow of a decent Unix shell. On the positive side I can already run eshell from within Emacs which provides a nice alternative to the command shell. I’ve yet to get tramp working though but I suspect that’s just a case of getting ssh keys sorted out.

If there are better solutions to getting a decent Emacs set-up on Windows I’m all ears.

-1:-- Now a Windows user! (Post Alex)--L0--C0--May 01, 2012 12:50 PM

Random Sample: CEDET news

Maybe you have noticed that CEDET 1.1 is released. The most important changes are support for Android and Arduino development, first steps towards proper Java and Fortran 90 support, a context menu and C/C++ completions through LLVM clang, but there is also a ton of smaller changes and bugfixes.

The goal for the coming months is to set up merging with Emacs, which is a huge problem that's been with us since Emacs 23.2 was released. During the first merge, a lot of stuff was changed by the Emacs maintainers: the directory structure is completely different, practically all files were renamed, compatibility code for older Emacsen was removed, EIEIO was changed for lexbind compatibility, and most importantly: the whole setup was changed to conform with the new minor-mode standards. These are mostly good changes, but needless to say it is not easy to merge all this stuff back without annoying the hell out of the users working with upstream.

So we first had to switch to Bazaar for development and completely revamp our file and directory structure in a separate branch, merge back all the changes, fix all the stuff that broke and get all the unit and integration tests to run. And while I was at it, I mostly rewrote the EDE build stuff for Elisp and fixed all the compile warnings, so that we now have the cleanest build since I can remember, and the buildbot will make sure it stays that way.

All this stuff has now been merged into CEDET trunk. So if you're adventurous, check out the shiny new development code (BTW, there's now also a git mirror), read the INSTALL file, get going and please report any problems to the CEDET development list. There's also a little nice addition for you Javascript coders out there: a semanticdb back end for hooking into a running Firefox via mozrepl. So you just load your current code into Firefox, fire up mozrepl and you can get completions for all your objects and for frameworks like jQuery in Emacs, which is almost impossible to do with static parsing since Javascript is, well, Javascript… ;-) Don't worry, I will write more details about the mozrepl plugin in the coming days.

-1:-- CEDET news (Post randomsample)--L0--C0--April 28, 2012 08:53 AM

Mickey Petersen: Fun with Emacs Calc

The Challenge

Jon over at Irreal’s Emacs blog posted an interesting solution to a challenge raised by Xah Lee:

How do you convert a string like this 37°26′36.42″N 06°15′14.28″W into a decimal answer like this 37.44345 -6.25396.

First off, I’m not sure Xah’s example answer is entirely correct; my understanding of latitude and longitude is limited to what I can google, and if I type the original degrees, minutes and seconds into this tool by the U.S. FCC it returns 37.44345 6.25396.

Anyway, on with the challenge. Jon’s solution is very interesting, but it got me thinking: surely Emacs has the facility in place to do this already? It’s Emacs, right? Right.

Fun with Emacs Calc

Emacs is equipped with a really, really awesome and spiffy RPN calculator (HP calculator fans, rejoice) capable of manifold things like algebraic and symbolic manipulation; matrix manipulation; conversion routines and much, much more. It’s truly wonderful but really complex, but it does come with a really nice Info manual (type C-h i g (calc) and check it out.) It’s a shame so few people know about its potential, as it’s basically a much simpler version of Mathematica, or even Wolfram Alpha (arguably you’ll have as much trouble telling Calc what you want as you would Wolfram Alpha…)

Anyway, I figured the Emacs calculator would have a facility in place for converting Deg-Min-Sec to decimal form, and sure enough, it does.

To try it out, type C-x * * and the calculator will open. Two windows will appear: the calculator mode and the trail containing a trail — a history — of commands and actions. The first thing we need to do is switch the calculator to “HMS” mode so we can try it out. To do this, type m h in the mode (the left) window and the modeline will change and say something like Calc: 12 Hms. The 12 is the floating point precision.

Next, type in the expression, replacing the unicode symbols above with @ for degrees; ' for minutes; and " for seconds. If you typed it in correctly, it will appear in the calculator window.

All we have to do now is convert it. Calc can convert between a wide range of units and systems, but we only care about decimals. Type c d and Calc will convert it to a decimal number. If you entered 37@26'36.42" you should see 37.44345 appear in its place.

OK, so we know it can do it, but how do we weaponize it? It so happens that Calc comes with a neat, little (though underdocumented) command called calc-eval.

Entering IELM, M-x ielm, we can query the calculator in real time:

ELISP> (calc-eval "1+2")
"3"

Yep. It does work. The hardest part about using it is mapping the “algebraic” notation used above with the indirect, keybinding-based input you use in the RPN calculator. Thankfully, the manual and (often) the trail will tell you the name of the function you are calling.

Let’s digress a little so I can show you how neat this calculator actually is: solving the elementary equation 2x+5=10. In the calculator, type m a to go to algebraic mode; next, type (2x + 5 = 10) — don’t forget the brackets — and it should appear as an equation. Finally, type a S to “solve for” a variable — and when prompted, answer x. The answer will appear in your calculator window. How awesome is that?

Back to the challenge. Calling the “convert to degrees” function is what we need to do, and the answer is hidden in the trail — it’s called deg.

Putting it all together, and we get:

ELISP> (calc-eval "deg(37@ 26' 36.42\")")
"37.44345"

That looks right. But the original challenge said that we had to take a string, like the one given above, and map that. So here’s my solution:

(defun hms-to-dec (hms-str)
  (let ((hms (split-string hms-str "[°′″NW ]" t)))
    (flet ((to-deg ()
                   (string-to-number
                    (calc-eval (format "deg(%s@ %s' %s\")"
                                       (pop hms) (pop hms) (pop hms))))))
      (list (to-deg) (to-deg)))))

Calling it from IELM yields the following answer:

ELISP> (hms-to-dec "37°26′36.42″N 06°15′14.28″W")
(37.44345 6.25396666667)

Looks good to me. Job done, and a fun challenge.

Share

-1:-- Fun with Emacs Calc (Post mickey)--L0--C0--April 25, 2012 12:57 PM

Julien Danjou: OpenStack Swift eventual consistency analysis & bottlenecks

Swift is the software behind the OpenStack Object Storage service.

This service provides a simple storage service for applications using RESTful interfaces, providing maximum data availability and storage capacity.

I explain here how some parts of the storage and replication in Swift works, and show some of its current limitations.

If you don't know Swift and want to read a more "shallow" overview first, you can read John Dickinson's Swift Tech Overview.

How Swift storage works

If we refer to the CAP theorem, Swift chose availability and partition tolerance and dropped consistency. That means that you'll always get your data, they will be dispersed on many places, but you could get an old version of them (or no data at all) in some odd cases (like some server overload or failure). This compromise is made to allow maximum availability and scalability of the storage platform.

But there are mechanisms built into Swift to minimize the potential data inconsistency window: they are responsible for data replication and consistency.

The official Swift documentation explains the internal storage in a certain way, but I'm going to write my own explanation here about this.

Consistent hashing

Swift uses the principle of consistent hashing. It builds what it calls a ring. A ring represents the space of all possible computed hash values divided in equivalent parts. Each part of this space is called a partition.

The following schema (stolen from the Riak project) shows the principle nicely:

Consistent hashing ring

In a simple world, if you wanted to store some objects and distribute them on 4 nodes, you would split your hash space in 4. You would have 4 partitions, and computing hash(object) modulo 4 would tell you where to store your object: on node 0, 1, 2 or 3.

But since you want to be able to extend your storage cluster to more nodes without breaking the whole hash mapping and moving everything around, you need to build a lot more partitions. Let's say we're going to build 210 partitions. Since we have 4 nodes, each node will have 210 ÷ 4 = 256 partitions. If we ever want to add a 5th node, it's easy: we just have to re-balance the partitions and move 1⁄4 of the partitions from each node to this 5th node. That means all our nodes will end up with 210 ÷ 5 ≈ 204 partitions. We can also define a weight for each node, in order for some nodes to get more partitions than others.

With 210 partitions, we can have up to 210 nodes in our cluster. Yeepee.

For reference, Gregory Holt, one of the Swift authors, also wrote an explanation post about the ring.

Concretely, when building one Swift ring, you'll have to say how much partitions you want, and this is what this value is really about.

Data duplication

Now, to assure availability and partitioning (as seen in the CAP theorem) we also want to store replicas of our objects. By default, Swift stores 3 copies of every objects, but that's configurable.

In that case, we need to store each partition defined above not only on 1 node, but on 2 others. So Swift adds another concept: zones. A zone is an isolated space that does not depends on other zone, so in case of an outage on a zone, the other zones are still available. Concretely, a zone is likely to be a disk, a server, or a whole cabinet, depending on the size of your cluster. It's up to you to chose anyway.

Consequently, each partitions has not to be mapped to 1 host only anymore, but to N hosts. Each node will therefore store this number of partitions:

number of partition stored on one node = number of replicas × total number of partitions ÷ number of node

Examples:

We split the ring in 210 = 1024 partitions. We have 3 nodes. We want 3 replicas of data.
→ Each node will store a copy of the full partition space: 3 × 210 ÷ 3 = 210 = 1024 partitions.
We split the ring in 211 = 2048 partitions. We have 5 nodes. We want 3 replicas of data.
→ Each node will store 211 × 3 ÷ 5 ≈ 1129 partitions.
We split the ring in 211 = 2048 partitions. We have 6 nodes. We want 3 replicas of data.
→ Each node will store 211 × 3 ÷ 6 = 1024 partitions.

Three rings to rule them all

In Swift, there is 3 categories of thing to store: account, container and objects.

An account is what you'd expect it to be, a user account. An account contains containers (the equivalent of Amazon S3's buckets). Each container can contains user-defined key and values (just like a hash table or a dictionary): values are what Swift call objects.

Swift wants you to build 3 different and independent rings to store its 3 kind of things (accounts, containers and objects).

Internally, the two first categories are stored as SQLite databases, whereas the last one is stored using regular files.

Note that this 3 rings can be stored and managed on 3 completely different set of servers.

Swift storage schema

Data replication

Now that we have our storage theory in place (accounts, containers and objects distributed into partitions, themselves stored into multiple zones), let's go the replication practice.

When you put something in one of the 3 rings (being an account, a container or an object) it is uploaded into all the zones responsible for the ring partition the object belongs to. This upload into the different zones is the responsibility of the swift-proxy daemon.

Swift proxy schema

But if one of the zone is failing, you can't upload all your copies in all zones at the upload time. So you need a mechanism to be sure the failing zone will catch up to a correct state at some point.

That's the role of the swift-{container,account,object}-replicator processes. This processes are running on each node part of a zone and replicates their contents to nodes of the other zones.

When they run, they walk through all the contents from all the partitions on the whole file system and for each partition, issue a special REPLICATE HTTP request to all the other zones responsible for that same partition. The other zone responds with information about the local state of the partition. That allows the replicator process to decide if the remote zone has an up-to-date version of the partition.

In case of account and containers, it doesn't check at the partition level, but check each account/container contained inside each partition.

If something is not up-to-date, it will be pushed using rsync by the replicator process. This is why you'll read that the replication updates are "push based" in Swift documentation.

# Pseudo code describing replication process for accounts
# The principle is exactly the same for containers
for account in accounts:
    # Determine the partition used to store this account
    partition = hash(account) % number_of_partitions
    # The number of zone is the number of replicas configured
    for zone in partition.get_zones_storing_this_partition():
        # Send a HTTP REPLICATE command to the remote swift-account-server process
        version_of_account = zone.send_HTTP_REPLICATE_for(account):
        if version_of_account < account.version()
            account.sync_to(zone)

This replication process is O(number of account × number of replicas). The more your number of account will increase and the more you will want replicas for your data, the more the replication time for your accounts will grow. The same rule applies for containers.

# Pseudo code describing replication process for objects
for partition in partitions_storing_objects:
    # The number of zone is the number of replicas configured
    for zone in partition.get_zones_storing_this_partition():
        # Send a HTTP REPLICATE command to the remote swift-object-server process
        verion_of_partition = zone.send_HTTP_REPLICATE_for(partition):
        if version_of_partition < partition.version()
            # Use rsync to synchronize the whole partition
            # and all its objects
            partition.rsync_to(zone)

This replication process is O(number of objects partitions × number of replicas). The more your number of objects partitions will increase, and the more you will want replicas for your data, the more the replication time for your objects will grow.

I think this is something important to know when deciding how to build your Swift architecture. Choose the right number the number of replicas, partitions and nodes.

Replication process bottlenecks

Copycat

File accesses

The problem, as you might have guessed, is that to replicate, it walks through every damn things, things being accounts, containers, or object's partition hash files. This means it need to open and read (part of) a every file your node stores to check that data need or not to be replicated!

For accounts & containers replication, this is done every 30 seconds by default, but it will likely take more than 30 seconds as soon as you hit around 12 000 containers on a node (see measurements below). Therefore you'll end up checking consistency of accounts & containers on each all node all the time, using obviously a lot of CPU time.

For reference, Alex Yang also did an analysis of that same problem.

TCP connections

Worst, the HTTP connections used to send the REPLICATE commands are not pooled: a new TCP connection is established each time something has to be checked against the same thing stored on a remote zone.

This is why you'll see in the Swift's Deployment Guide this lines listed under "general system tuning":

# disable TIME_WAIT.. wait..
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1

# double amount of allowed conntrack
net.ipv4.netfilter.ip_conntrack_max = 262144

In my humble opinion, this is more an ugly hack than a tuning. If you don't activate this and if you have a lot of containers on your node, you'll end up soon with thousands of connections in TIME_WAIT state, and you indeed risk to overload the IP conntrack module.

Container deletion

We also should talk about container deletion. When a user deletes a container from its account, the container is marked as deleted. And that's it. It's not deleted. Therefore the SQLite database file representing the container will continue to be checked for synchronization, over and over.

The only way to have a container permanently deleted is to mark an account as deleted. This way the swift-account-reaper will delete all its containers and, finally, the account.

Measurement

On a pretty big server, I measured the replications to be done at a speed of around 350 {account,container,object-partitions}/second, which can be a real problem if you chose to build a lots of partition and you have a low number_of_node ⁄ number_of_replicas ratio.

For example, the default parameters runs the container replication every 30 seconds. To check replication status of 12 000 containers stored on one node at the speed of 350 containers/seconds, you'll need around 34 seconds to do so. In the end, you'll never stop checking replication of your containers, and the more you'll have containers, the more your inconsistency window will increase.

Conclusion

Until some of the code is fixed (the HTTP connection pooling probably being the "easiest" one), I warmly recommend to chose correctly the different Swift parameters for your setup. The replication process optimization consists in having the minimum amount of partitions per node, which can be done by:

  • decreasing the number of partitions
  • decreasing the number of replicas
  • increasing the number of node

For very large setups, some code to speed up accounts and containers synchronization, and remove deleted containers will be required, but this does not exist yet, as far as I know.

The opinions expressed here represent my own and not those of my employer.

-1:-- OpenStack Swift eventual consistency analysis &amp; bottlenecks (Post Julien Danjou)--L0--C0--April 23, 2012 12:06 PM

Vincent Toups: Parenlab: S-expressions on top of Matlab/Octave

I am two people - one of them is a numerical scientist who spends most of his time writing code in Matlab to do data analysis. The other is a freelance Lisp programmer, where I do regular software engineering. I've long been a Lisp enthusiast, and I do a fair amount of Lisp programming in Emacs Lisp as part of my scientific work, since Emacs is my Matlab IDE, but I didn't start really programming in Lisp for "real things" until recently. As the number of hours dedicated to Lisp increased in my life, I began to really miss the features in Lisp that I didn't have in Matlab; in particular s-expression motion and editing and metaprogramming.

I can't abandon Matlab, however. Matlab's support for numerical programming is exceptional, true, but the real value is that its plotting system is rich, well integrated and customizable. Plus, I have 7+ years of Matlab utilities which it would take significant time to reproduce in another numerical analysis language, even if it was feature complete otherwise.

So I brought the mountain to Muhammad.

Parenlab

Parenlab is a Lisp which compiles to Matlab by way of Emacs Lisp. What this means is that the syntax is Emacs Lisp and the Semantics is Matlab. Metaprogramming is possible via macros which transform the elisp representation of Matlab code before the code is translated to Matlab. Because I use Emacs as my Matlab IDE, this is a reasonable solution - just use the Emacs Lisp interpreter as the macro language. One day, Parenlab may be self-hosting.

Examples:

The parenlab code:

(let* ((x (range 0 (* 2 pi) 100))
       (y (sin x)))
 (plot x y :color "g"))

Is converted to:

funcall(...
    @(x)funcall(@(y)plot(x, y, 'color', 'g'), sin(x)),...
     range(0, mtimes(2, pi), 100))

Or:

(defun (a b c) some-function (a b c)
   "Documentation."
   (setq a (++ a b c))
   (setq b (++ a b c))
   (setq c (++ a b c)))

Is converted to a file, called someFunction.m which contains:

function [a,b,c] = someFunction(a,b,c)
%Documentation.
'Documentation.';
a = plusplus(a, b, c);
b = plusplus(a, b, c);
c = plusplus(a, b, c);

defuns can be sprinkled throughout your script files - they will be transcoded as they are encountered and produce no output in the currently building function. One can also say:

(script some-script-name 
 <parenlab-code>)

Which indicates the code beneath should be transcoded to a script file rather than "in place."

Parenlab is still a work in progress, but it is rapidly approaching the point where any functionality you wish can be expressed in Parenlab.

Notes on Usage

Mangling

Parenlab, like Parenscript, tries to make life easy by letting you write lisp-style identifiers. In doing so, it mangles names during translation. Eg, a parenlab symbol:

some-function

Will be rendered as:

someFunction

eg, dashes followed by letters are converted to camel case. Other "special" characters are transcoded according to the following table:

("+" "plus")
("-" "minus")
("*" "mtimes")
("<" "lessThan")
(">" "greaterThan")
("$" "cash")
("=" "equal")
("!" "bang")
("?" "who")
(":" ":")
("/" "divide")
("\\" "mdivide")
("#" "hash")
("@" "at"))

These choices are made to improve transcoding, so that many operators don't need special cases. Eg (* 10 10) translates to mtimes(10,10) - conveniently using Matlab's built in mtimes function, which is equivalent to Matlab's * operator.

One wrinkle is that : is reserved to preserve simple matrix construction expressions. For instance,

x:some-variable:y

transcodes to

x:someVariable:y

This syntax is limited by how the lisp reader reads symbols. For complex generation expressions, use the (: ) macro. Eg:

(: start step stop)

Keywords, that is symbols starting with :, are transcoded to mangled strings, so that :a-keyword becomes 'aKeyword'.

Since many matlab functions use strings as keyword arguments, this lets you use keywords for them instead.

nil transcodes to []. There are no true and false values in Matlab.

Scope and Variables

Scope in Matlab is funny - only limited lambdas are allowed in scripts and functions, although nested functions with full definitions and full lexical scope behavior are allowed within functions (though not anonymously).

Sort of like in Python, lambdas are restricted to single-expression-only bodies. While they do form closures over their lexical environment, those closures are static, that is, they do not permit side effects of any kind.

That is,

(setq y 10)
(setq f 
 (lambda (x) 
   (setq y x)))

Will compile fine in Parenlab, but produce an error, because the inner setq is not an expression, but a statement, which isn't allowed. Parenlab does provide a progn form, so you might try:

(setq y 10)
(setq f 
 (lambda (x) 
   (progn (setq y 10)
          nil)))

This will generate Matlab code which uses eval to produce code which expresses this intent, but you'll still get an error, saying that the lambda tries to modify its static environment, which is not allowed. let and let* are implemented using lambda, so they create static lexical environments. It is conceivable that major cross compilation tricks could be used to simulate dynamic scopes, but I'd prefer to keep things simple.

You shouldn't be using side effects anyway.

The rule of thumb is write Matlab code with s-expressions, not Lisp code that calls Matlab functions. Parenlab tries to bridge the gap but in some ways Matlab is too limited.

If Statements

If in Matlab is "flat", in that each branch is a series of statements or expressions and the if statement itself doesn't return a value. Parenlab if statements are not flat - they return the value of whatever branch is evaluated.

This rule is sometimes inconveneint for the Matlab idiom, so it can be disabled if the branches are blocks. A statement of the form:

(if condition 
  (block 
    <some-code>)
  (block 
    <some-other-code>))

Transcodes to:

 if condition 
    some code
 else
    some other code
 end

Ordinarily, if expands to

fif(@()condition, @()true-code, @()false-code)

Here we use the trick of using lambda to delay evaluation. This if expansion cannot perform side effects, unfortunately. Use (block) legs if you want regular matlab semantics. Both legs must be blocks.

Cell Arrays

Indexing is identical function calls, syntactically, so the index expression:

x(1:10)

Is written as

(x 1:10)

In parenlab. Matlab also has cell array indexing, which looks like this:

x{1:10}

Which is expressed like this in Parenlab:

({} x 1:10)

These are macros, not functions, and so the use of end as an identifier inside them is fine, eg:

({} x 1:end)

Will convert to:

x{1:end}

Structs

Structure access can be written as symbol.name since the parenlab mangler leaves dots in symbol names. Programmatic access, where a string is used, is written with ->, eg:

(-> s :field)

Which is equivalent to s.('field'). -> supports nested access, so you can say;

(-> s :f1 :f2 :f3)

Which is like this: s.('f1').('f2').('f3') which may not actually be valid Matlab syntax. Parenlab implements this as a function call.

Macros

In elisp, parenlab macros can be defined with pl:def-pl-macro. During transcoding, the form defmacro causes a new parenlab macro to be defined and generates no output. Parenlab macros have the same semantics as regular Macros. You get syntax passed to the macro as arguments, you transform it into valid parenlab, and you return the result.

For instance, a macro which introduces a lexical variable binding called with, like this:

(with x 10 (* 2 x)) ;-> 20

Is implemented like this:

(pl:def-pl-macro with (symbol value expression)
 `(funcall (lambda (,symbol) ,expression) ,value))

Or, inside some Parenlab code:

(defmacro with (symbol value expression)
 `(funcall (lambda (,symbol) ,expression) ,value))

Other Notes

Right now you use parenlab by invoking pl:transcode on an s-expression. This will transcode in the current buffer. pl:transcode-to-string will transcode to a string instead.

Parenlab requires matlab-mode so that it can indent its outputted code correctly.

I have a highly idiosyncratic Matlab setup, so I put some integration code in auxilliary.el. If you have a Matlab process running in a buffer called *evalshell*, then this code will let you load a parenlab file and press C-c C-c to "compile" and execute the code therein. This process will write any script and defun forms to files before executing the code. C-x C-e will evaluate the last s-expression as Matlab code in the interpreter.

Parenlab depends on Shadchen-el my pattern matching library.

-1:-- Parenlab: S-expressions on top of Matlab/Octave (Post J.V. Toups (noreply@blogger.com))--L0--C0--April 18, 2012 07:29 PM

Chris Ball: Time; Cambridge, MA

-1:-- Time; Cambridge, MA (Post Chris Ball)--L0--C0--April 18, 2012 11:16 AM