Documenting Common Lisp Projects for GitHub

I want to be able to automatically create the for GitHub whenever I create a new Common Lisp project.  I was impressed by the number of systems that exist for documenting a Common Lisp package.  However, I couldn’t find anything that was simple and that created Markdown output, like the that GitHub uses.

I was in a hurry, so after a few minutes of Googling, I decided to create my own function for documenting packages for GitHub.  It worked.  Sort of.  It documents function only, not macros or other things.  Here’s the function:

(defun document-package (package output-filename)
  "Documents the Common Lisp package PACKAGE and writes that documentation to the file given by OUTPUT-FILENAME."
  (loop for function being the external-symbols of (find-package package)
     when (and (fboundp function) (documentation (symbol-function function) t))
       (list :function function
             :function-name (string-downcase function)
             :documentation (documentation (symbol-function function) t))
     into functions
       (return (loop for function in 
                    (sort functions #'string<
                          :key (lambda (x) (getf x :function-name)))
                  collect (format nil "## ~a ~a~%~a"
                                  (string-downcase (getf function :function-name))
                                   (format nil "~s"
                                            (symbol-function (getf function :function))))
                                   '(("\\s\\s+" " ")
                                     ("DC-UTILITIES::" "")))
                                  (getf function :documentation))
                  into function-docs
                    finally (spew (format nil "# ~a~%~{~a~^~%~%~}" package function-docs)

That function, when called like this: (document-package :dc-utilities “~/”) produces Markdown that looks like this:

## alist-values (ALIST &REST KEYS)
Returns the values associated with KEYS in ALIST.  ALIST is an associative list.

## bytes-to-uint (BYTE-LIST)
Converts the list of bytes BYTE-LIST into an unsigned integer.

## change-per-second (FUNCTION-OR-SYMBOL &OPTIONAL (SECONDS 1))
Given the function FUNCTION-OR-SYMBOL, who's return value changes over time, or a variable who's value changes over time, with the change being unidirectional, this function computes the rate of change by calling the function, sleeping, then calling the function again, then computing the rate of change per second.  You can optionally specify the number of seconds to wait between calls.  If FUNCTION-OR-SYMBOL is a variable, then this function retrieves the value of the variable, sleeps, then retrieves the value of the variable again.

## create-directory (DIR &KEY WITH-PARENTS)
Works just like the mkdir shell command.  DIR is the directory you want to create. Use WITH-PARENTS if you want the function to create parent directories as necessary.

## cull-named-params (NAMED-PARAMS CULL-KEYS)
Given a value for NAMED-PARAMS like this one

    '(:one 1 :two 2 :three 3)

and a list of CULL-KEYS like this one

    '(:one :two)

this function returns a list of named parameters that excludes the names (and their values) that match the names in CULL-KEYS.  In the above example, the result is

    '(:three 3)

## directory-exists (PATH)
Returns a boolean value indicating if the directory specified by PATH exists.


Which looks like this on GitHub:

Is there a package out there already that can do this better?  If so, please leave me a comment.  If not, I need to find out how to get the documentation for a macro, for a method, for a special variable, for a class, and so on.  Any help would be greatly appreciated.

Other things I want this function to do:

  • Check the .asd file to see if there’s a license bit: :license "MIT License"If there is, the code should check to see if there’s a LICENSE file in the project.  If there isn’t, the code should determine if the name of the license is known.  If the code is able to get the text for the license, it should include it at the beginning of the documentation.
  • Optionally produce a table of contents.
  • Output other formats (aside from Markdown) as well, like Text or Emacs Org-Mode.

For now, the document-package function is just a function in the dc-utilities package.  However, once I flesh the function out a little, I will probably move it to its own repo.  If you get ahead of me with any of this, please send me a note.  I could really use this functionality.

Predicting a User’s Movie Preferences

One thing that online video distributors (such as Netflix) don’t do yet (and that they will most certainly do in the future–some of the tech is already in place) is to take into account the content of the movies, such that they can learn a user’s likes in a way that is independent from (and complimentary to) the likes of other users with similar preferences.

It’s been possible for while, for example, to automatically identify “types” of documents and to tag or cluster those documents with a high degree of accuracy. Early last year, using a euclidean-distance algorithm that worked on trigram-vector representations of text, I created an example of just such a thing, document affinity, using Wikipedia articles. In the example, you could choose a Wikipedia article and the system would list the other Wikipedia articles in order of similarity to the one you chose. Despite the fact that this method is rather well known, the results of that experiment were nothing short of astonishing, and provided a quality of links among Wikipedia articles that exceeds anything that exists today, much better even than the human-made links that are in the articles. Moreover, consider that the bag-of-trigrams method I just described is archaic by comparison to some of the algorithms that research groups are exploring today, like recurrent neural networks using word vectors (my favorite).

Imagine if, in addition to resorting to the standard preference algorithms, an online video distributor (such as Eros Now or Netflix) were to analyze the script of the movie (subtitles, for example), identify similar movie scripts, and account for that when choosing other movies that you might like. The distributor’s predictions would improve in a dramatic way.

The technology to analyze the text of the movies exists already. But, if you look into the future, you’ll find that soon computers will be able to analyze the visual part of the movie, the frames themselves, the motion, and the sounds. Computers will be able to combine that type of analysis with script analysis for a full-content analysis that will increase the preference predictions even more markedly. Google and others have already started to do some work in related areas, largely based on the contributions to deep learning that Geoffrey Hinton and his students have made in the last decade, including rectified linear units, dropout, ways of combining convolutional neural-network layers with fully-connected layers for higher abstraction, and word-vectors and phrase-vector training.

We’re going to see far better preference predictions in the near future, probably by smaller companies in the beginning. In the long run, this kind of prediction, and the more advanced ones that I described above, which encroach into the realm of human cognitive abilities, will become widespread.

My Wishlist for the Future

I’m am waiting for the following developments and contributing in any way that I can to their fruition:


  • All languages to converge toward a Lisp-like language
  • Trillion-fold increase in computer processing power
  • Superhuman-level artificial general intelligence
  • Human mind uploads
  • Indefinite human lifespans


  • Dissolution of all international borders
  • Fully distributed, internet-like, free-market, and non-invasive economy, with multiple distributed-ledger based currencies
  • Consolidation of all taxes, including income tax, sales tax, gas tax, local taxes, and property taxes into a single, monthly bill, at the local level
  • Completely open government process that collects and incorporates feedback from citizens, uses revision control, and uses open-source software to automate all aspects of government
  • Replacement of all government-issued business licenses (including medical licenses, legal licenses, engineering licenses, education licenses, and driving licenses) with commercial rating systems as well as legal and enforcement systems that can encourage honesty and deter wrongdoers
  • 1 year sunset on all regulations

I fully expect all of these developments to happen in my lifetime. They will bring tremendous wealth, health, equality, meaning, and worthiness to all of humanity.

Cleaning House by Open Sourcing

I’m going to start open sourcing some of the code that I currently have in private repos. It’s not going to be much at first, because I tend to collaborate with others when writing code, or I write it for companies that would never consider open sourcing their code. However, I’m going to start now and I’m going to try to talk collaborators into letting me open source stuff that I write in the future.

Open sourcing some of this code might encourage me to make it more usable.

I’m starting with some code that I use to tinker with neural networks: dc-ann

It’s disorganized right now, but, in time, I’ll put up some examples of how to use it, convert it to a quicklisp project, and make it easier to use. Furthermore, the code, as released, leaves out the transfer-function and weight-initialization changes that make the neural network a deep-learning neural network. I’ll get those in as soon as I clean up that code in the private repo.

Here are some other bits that I’m tossing into the open source community now:
dc-affinity (Lisp code to demonstrate the effects of measuring Euclidean Distance between trigram vectors)
Utyls (Perl utilities that I use frequently)

More bits that I plan to release soon include a ton of Emacs Lisp code and more Perl utilities.

API for a Mobile App in Common Lisp

Not too long ago, I released a mobile app called Lil’ Baby Names, for iOS and Android. I had a few friends help me. Mark Edelsberg designed the UI, Liu Xinjun built the Android front end, and Pritesh Shah built the iOS front end. I built the API that the devices use. What’s interesting about the API is that it’s written in Common Lisp. Why? Most of all because I love Lisp code. To my eye, the same algorithm written in Lisp or any other language always looks prettier in Lisp. But, I’ll get to that in another post. Here are some other reasons why I chose Lisp:

  • The code compilation process is fast and transparent
  • The code compiles directly into machine code
  • I can recompile functions in a running program, so I don’t have to restart the production server if I change any code
  • The REPL allows me to test pieces of my code as I go, providing continuous feedback on the work that I’m doing
  • Lisp macros and other lisp features make the code concise and easy to follow
  • The resulting service is really fast

For example, the Lil’ Baby Names API has a single endpoint and here’s how I define that endpoint:

  (get "/api/names" api-names))

Here’s another example. This code reads a small map of genders to gender IDs from the database:

(flatten (db-cmd *db* :query
                 (:select 'gender 'id :from 'genders)))

Note how I write my SQL in Common Lisp. That’s possible thanks to Lisp macros. This is not just beautiful, but also convenient: now, my editor can properly highlight, indent, and detect syntax errors in my SQL code.

As a final example, take a look at code that formats the results into JSON or plain text, depending on the value of the format variable:

(case format
    (ds `(:map :time ,(format nil "~ss" (read-time event))
               :total-matches ,results-length
               :page ,page
               :page-size ,page-size
               :regex ,regex
               :starts-with ,starts-with
               :ends-with ,ends-with
               :contains-string ,(get-parameter "contains-string")
               :contains-letters ,(get-parameter "contains-letters")
               :min-length ,min-length
               :max-length ,max-length
               :format ,format
               :sort ,sort
               :results ,(ds-list sorted-paged-results)))))
   (format nil "~{~a~^~%~}"
           (mapcar (λ (x) (gethash :name x))

Note the use of the Common Lisp format function, and how concise it can make the code. Also, notice how easy it is to transform a Lisp object or nested list into JSON.

Sure, there are some negative aspects of the language. For one, Common Lisp is a big language with a number of fairly powerful functions and macros (think format and loop). It can take a little time to get the hang of those. Furthermore, the best Common Lisp editor is probably Emacs, and who wants to learn that? However, for those brave programmers who choose to overcome those obstacles, Common Lisp provides a real glimpse into the future of programming.

Valentines for Nina and Lola

Nina and Lola,

Whenever I’m away from you and I spend a lot of time writing computer code, I think about the pretty patterns that your names make in binary code. Your names, Nina and Lola, are wonderful in so many ways that I could never describe them all. Here’s one of those ways.

If you use the following Common Lisp expression, you can find the graphical binary representation of your names in binary code.

(defun binary-name (name &optional (zero #\Space) (one #\O))
    (map 'string (lambda (c) (case c (#\1 one) (#\0 zero) (t c)))
         (format nil "~%~{~b~%~}" (map 'list #'char-code name))))

After defining a function like that, you can call the function with your names, like this, for example:

(binary-name "Nina")

Which returns the following pattern:

    O  OOO 
    OO O  O
    OO OOO 
    OO    O

That pattern represents your name! The letters are in rows. So, ‘N’ is ‘O OOO’ and ‘i’ is ‘OO O O’. Here’s the same pattern with letters in front of each row:

    N => O  OOO 
    i => OO O  O
    n => OO OOO 
    a => OO    O

One interesting thing is that, in your name at least, the capital letters start with ‘O ‘ and the lower-case letters start with ‘OO’.

Lola, here’s the pattern for your name:

    O  OO  
    OO OO  
    OO    O

Here’s the pattern for ‘Daddy’:

    O   O  
    OO    O
    OO  O  
    OO  O  
    OOOO  O

Notice how the pattern for ‘Daddy’ is bigger than the patterns for ‘Lola’ and ‘Nina’? That’s just because ‘Daddy’ has more letters.

Here’s the pattern for ‘loves’:

    OO OO  
    OOO OO 
    OO  O O
    OOO  OO

Here’s the pattern for ‘&’:

    O  OO

So, what do you think this means:

    O   O      OO OO   
    OO    O    OO OOOO     O  OOO               O  OO  
    OO  O      OOO OO      OO O  O              OO OOOO
    OO  O      OO  O O     OO OOO               OO OO  
    OOOO  O    OOO  OO     OO    O    O  OO     OO    O

Happy Valentines to you both! I love you more than anything else in the whole world and I’ll always be there for you.


Catching the Human Mind

I was benchmarking PCs to try to determine where to put some Chattermancy code and I started to wonder once more where we are in the unintentional technological race to build a cheap computer that can outperform the human brain. I’ve done this exercise a half-dozen times over the years. Here’s what I found this time.

Calculating how long it’s going to take for PCs to become as powerful as the human brain is easy if you can make a few assumptions. Here are my assumptions:

  • Processing power of the human brain: 100 teramips
  • Processing power of an inexpensive PC today: 0.025 teramips
  • Rate of increase of computer processing power: doubling every year

With those assumptions and a tool like a simple spreadsheet, a small computer program, or a pencil and some paper, anyone can calculate more or less when an inexpensive PC will have the processing power of the human brain. I chose the computer program path. Here’s an example in Common Lisp:

(defun forecast (&key (today-teramips 0.025) (target-teramips 100))
  (let ((power (- (/ (log target-teramips) (log 2))
                  (/ (log today-teramips) (log 2)))))
    (format nil "~a~a~a~%~a~3$~a~%~a~$"
            "Human brain throughput: " target-teramips " teramips"
            "PC throughput: " today-teramips " teramips"
            "Years until inexpensive PC outperforms human brain: " power)))

If you run that program like this

(forecast :today-teramips 0.025 :target-teramips 100)

the program will spit out the following text:

Human brain throughput: 100 teramips
PC throughput: 0.025 teramips
Years until PC outperforms human brain: 11.97

If your assumption about the processing power of the human mind is off by an order of magnitude, such as 100 teramips would be if the human mind actually ran at 1,000 teramips, the result differs little (15 years instead of 12, in this example).

If you make the same assumptions that I made, then the calculation itself is easy and probably not disputable. However, the assumptions are definitely quite disputable, so I will explain how I arrived at those.

Processing power of the human brain

This number is the hard one to pin down. Picking the right value for this variable is the key to this exercise. Here are a few starting points:

Processing power of an inexpensive PC today

To figure this out, I started by downloading the BYTEmark benchmarking software from here: then extracted, compiled, and ran the software on a quad-core core-2 duo machine with 6GB of RAM, a machine that is worth about US$800.00 today. One number that I used from the results of that test was 81.153, the original BYTEmark results integer index. This number means that the computer is roughly 81 times faster than the baseline for this test, a Pentium 90. I found information on the Internet that indicated that the Pentium 90 ran at approximately 150 mips.


That information allowed me to calculate a mips rating for my computer:

81 * 150 mips = 12172 mips

But my computer has 4 processors and this test measures the throughput of a single processor. I multiplied the result by 2 rather than by 4 to obtain a conservative 25,000 mips (or 0.025 teramips) rating for my computer.

Rate of increase of computer processing power:

This was the easiest of all assumptions to make. I simply checked the processing power of the fastest computer you can buy for US$1000 today and it turned out to be about double the of the fastest computer you could buy a year ago for the same amount of money. However, I believe that this assumption of computer power doubling every year is extremely conservative because there are now desktop supercomputers (still expensive–in the ten-thousand-dollar range) that, for some tasks, are hundreds of times faster than a PC. This indicates that in the near future we’re going to see inexpensive computers with hundreds of processors that are likely going to make the conservative estimate we’re using here (computer power doubles every year) seem far too conservative.

Common Plisp: List Processing in Perl

I was recently talking on the phone with a person who lives at least 2,200 miles away and whom I’d never met or spoken to before. This is a surprisingly common occurrence in this day and age. I was explaining some of the things that I like about Perl. When I got to the part about how I love writing 4 or 5 lines of code where programmers of other languages have to write 20 or 30, my new friend hinted that he thought I was talking about completely unreadable Perl code. I was quick to point out that I wasn’t talking about the Perl golf that looks like encrypted text, but rather about using higher-order functions like map and grep. I’ve written about this sort of thing before. See, for example, the article titled Concise Programming and Super General Functions.

But, here’s another example of how Perl can be very concise. I’m going to present a program with 4 lines of code that can read a quoted CSV file (with a column-headings row) and parse it into an array of hash references that allow you to access a piece of data by line number and field name, like this:

print "keyword: $hdata[2]->{keyword}\n",
      " visits: $hdata[2]->{visits}\n";

The above code would print

        keyword: convert soap to rest
         visits: 81

Here’s the code:

use Text::ParseWords;

my @data= map {s/\s$//g; +[quotewords ',', 0, $_]} <>;
my @fields= map {lc $_} @{shift @data};
my @hdata= map {my $v= $_; +{map {($_ => (shift @{$v}))} @fields}} @data;

# Print the top 3 keywords and their visits
print "$_->{keyword} => $_->{visits}\n" for @hdata[0..2];

Here’s the sample CSV file:

"perl thread pool",210,1.00,00:00:00,23.81%,100.00%
"soap vs rest web services",152,1.00,00:00:00,100.00%,100.00%
"convert soap to rest",81,1.00,00:00:00,12.50%,100.00%
"perl threads writing to the same socket",63,1.00,00:00:00,0.00%,100.00%
"object oriented perl resumes.",52,1.00,00:00:00,20.00%,100.00%
"perl thread::queue thread pool",54,1.00,00:00:00,0.00%,100.00%,
"donnie knows marklogic install",43,1.00,00:00:00,25.00%,100.00%
"perl threaded server",45,2.00,00:01:28,75.00%,25.00%
"slava akhmechet xpath",44,1.50,00:08:46,0.00%,75.00%
"donnie knows",36,6.67,00:02:56,66.67%,0.00%

And here’s an example of how the code stores a line of that CSV file in

   keyword => "perl thread pool",
   visits => 210,
   pages_per_visit => 1.00,
   avg_time_on_site => "00:00:00",
   new_visits => "23.81%",
   bounce_rate => "100.00%"

The program doesn’t use any modules that don’t ship with Perl, so you don’t have to install anything beyond Perl itself to make this program work. Also, once you learn a few standard Perl functions for processing lists and a little about Perl data structures, the code is actually very readable.

Let’s take a detailed look at the code to see how so little of it can accomplish so much. We’ll start with the definition of @data.

my @data= map {s/\s$//g; +[quotewords ',', 0, $_]} <>;

List processing and filtering functions are often best read backwards. Let’s start with the <> operator, which in list context reads all the lines from the file name you provide to the program. If you save the program as, for example, and you run it like this:

./ file.csv

Then the <> operator will read the contents of file.csv and return it as an array of lines. The map function accepts some code and an array and returns a new array. Each element in the new array consists of the result of applying the given code to the corresponding element in the original array. Here’s an example of how to use the map function:

@n_plus_2 = map {$_ + 2} qw/1 2 3/;

@n_plus_2 ends up with the values 3, 4, and 5.

The function we pass to map in the @data assignment removes trailing spaces from the each line of text, then splits the line of text into values at the commas—excluding quoted commas, of course, and allowing escaped quotes within a value. So @data ends up looking like this (only the first two lines of the example CSV file included here):

   ["keyword", "visits", "pages_per_visit", "avg_time_on_site",
     "new_visits", "bounce_rate"],

   ["perl thread pool", 210, 1.00, "00:00:00", "23.81%", "100.00%"],

   ["soap vs rest web services", 210, 1.00, "00:00:00", "23.81%",

The @fields assignment simply pulls the first row out of the @data array, lower-cases each column title, and assigns the resulting array to @fields.

Finally, the @hdata assignment converts each array reference in @data into a hash reference. It does so by associating a key (a column title) with each value in the array reference. The resulting @hdata array contains hash references.

How many lines of code does it take to do this kind of stuff in your favorite language?

Perl Sockets Swimming in a Thread Pool

I’ve written a simple multithreaded TCP server that uses sockets and a thread pool to handle client requests. This server is packaged into a class (DcServer) that can be trivial to use. Here’s how you might use DcServer to build a server that accepts text from clients and returns the same text with the characters in reverse order:

use DcServer;
my $server= DcServer->new(processor_cb => \&reverse_text);

sub reverse_text {
    my $data= shift;
    join('', reverse(split //, $data));

Run like this: perl

The server defaults to 10 threads, so you could have clients connecting concurrently to have their text reversed. And you could probably serve a great many of these clients per second.

I’ve also created code to help me build clients. You can, for example, build a client for the above server like this:

use DcClient;
my $message= shift;
die "Usage: perl string\n" unless defined($message);
my $c= DcClient->new;
print "$message => ", $c->query($message), "\n";

Run like this: perl "hello"

The code for the server is short and bare-bones and illustrates how to pass a socket from one thread to another.

Yes, I suffer from the NIH (not-invented-here) syndrome. I have no excuse for having written this. Let’s leave it at that. Nevertheless, I hope the code is useful to someone out there, directly or as a simple example of how to set up thread pools and how to share sockets among threads.

Here’s a little reference for how to use the DcServer module, which depends on these modules only:
– threads
– threads::shared
– Thread::Queue
– IO::Socket
– Time::HiRes


DcServer is a class that encapsulates a TCP server that implements a thread pool to handle client requests. The class allows the instantiator to provide call back functions for processing client requests, performing background work, and performing tasks after the server has shut down. The instantiator can shut down the server from within two of these callback functions. For example, the instantiator can shut down the server when certain arbitrary conditions are met (regardless of whether clients are connecting or not) or when a client explicitly requests it.

DcServer Methods

This method instantiates a server. It takes the following parameters.

  • host An IP address or host name that resolves to an IP address. This is to tell the server where it will be running. Clients should specify the same host name in order to connect. This value defaults to ‘localhost’ if you don’t specify it.
  • port The port on which the server will listen for connections. This value defaults to 8191 if you don’t provide it.
  • thread_count The number of threads that you want the server to start for its thread pool. The server uses the thread pool for handling client connections. This value defaults to 10 if you don’t specify it.
  • eom_marker This is the sequence of characters that the server and the client use to tell each other that they’re done transmitting data. This value defaults to “\\n\\.\\n”, which means “new line, period, new line”.
  • main_yield The server allows the instantiator to specify a callback function that the server will call in a loop. After calling this function, the server sleeps for main_yield seconds before calling the function again.. The value of main_yield defaults to 5 seconds.
  • main_cb A reference to a function that the server calls over and over again, independently of accepting and processing client connections. The server calls the function specified via main_cb with two positional parameters, $counter and $fnstop. The first parameter, $counter, contains an integer with the number of times that the server has called the main_cb function. The second parameter, $fnstop, is a code reference that, when called as a function, causes the server to stop. You can call the function in $fnstop like this: $fnstop->()

    When you call the $fnstop function, the function returns immediately, but by then the call has already initiated a server shutdown.

  • done_cb A reference to a function that the server calls when it has completed its shutdown procedure.
  • processor_cb A reference to a function that the server calls to process the data that the client has sent. The function should accept a string (the client’s request) and it should return another string (the result of processing the client’s request). More specifically, the server calls the processor_cb function with the following positional parameters: $data, $ip, $tid, and $fnstop. There following list describes these parameters:
    • $data: The data that the client sent to the server. This amounts to the client’s request. It is up to the instantiator to interpret and process that request.
    • $ip: The IP address of the client.
    • $tid: The ID of the thread that is handling the request.
    • $fnstop: A reference to a function that you can call to stop the server. You can call this function like this: $fnstop->()

    The processor_cb function should return a string consisting of an answer to the client’s request. The processor_cb function should be able to call the $fnstop function and still return data to the client. But if you do that, the call to $fnstop should occur immediately before the function returns, not before the function processes the client’s data.

This method takes no parameters. It simply starts the server that you previously instantiated with the new method.


While writing this code, I ran into some serious trouble with passing sockets from one thread to another. I was able to resolve the issue thanks to

something about sockets and threads that BrowserUk posted back in 2006.

Thanks BrowserUk! (And thanks PerlMonks!) I am not worthy of your guidance.

Source Code

You can view or download the code for the

DcServer and DcClient modules here.

You’re welcome also to add the code to CPAN or to POD it or to attempt to convince me to do any or all of this. I know that I should, but I’m such a sloth that I’ve never contributed even a single module to CPAN. The only thing I ask is that you let me know if you manage to improve the code in any significant way.