One of my minor duties on this dual-job is to assemble slides from about twelve people into a large presentation, with cover slides for each speaker, every Friday for a progress meeting we all have. Naturally the first Friday I just did it by hand by importing each one in turn into PowerPoint. However it is a fairly tedious job, and to paraphrase a certain member of staff: why do something by hand when I have a powerful computer under the desk?

So I began to investigate automating the process.

Turns out that Python has an IMAP module in its standard library, which isn’t too surprising I suppose as the Python standard library is enormous. After some playing I managed to write a program that logged into my university email account and downloaded the appropriate PowerPoint attachments.

from imaplib import IMAP4_SSL
from email import message_from_string
from datetime import datetime, timedelta
import re
 
M = IMAP4_SSL ('imapserver')
M.login ('user', 'pass')
M.select ()
 
since = (datetime.today () - timedelta (days=5)).strftime ("%d-%b-%Y")
 
for user in users:
    typ, data = M.search (None, '(HEADER FROM "%s" SINCE %s)' % (user, since))
 
    for id in data[0].split ():
        typ, data = M.fetch (id, 'RFC822')
        msg = message_from_string (data[0][1])
        if not msg.is_multipart ():
            continue
 
        for part in msg.walk ():
            fn  = part.get_filename ()
            ext = re.match (r'.*\.(pptx?)$', fn):
            if not ext
                continue
 
            fp = file ('%s.%s.temp' % (user, ext.group (1)), 'w')
            fp.write (part.get_payload ())
            fp.close ()
 
M.close ()

This code would download any .ppt or .pptx attachment from anyone with an email address in the users list and save the file as <user>.ppt.temp or <user>.pptx.temp. It should be possible to decode the message part with msg.get_payload (decode=True) but it seemed to introduce a fair number of errors into the file. I think this is because it seems to convert line-by-line instead of as one large block. So I used the following code to fix this.

raw      = part.get_payload ()
raw      = re.sub (r'[\r\n]+', '', raw)
encoding = part['Content-Transfer-Encoding'].lower ()
if encoding == 'base64':
    raw = b64decode (raw)
elif encoding == 'quoted-printable':
    raw = quopri.decodestring (raw)
else:
    print "ERROR: Unknown coding strategy - ignoring."
    continue

Not particularly pretty, but it does the job.

So now I had all the emails downloaded, but I still needed to merge them all into a single file. At first I was thinking of making a C++ program to exploit the PowerPoint COM interface, but then I found Python for Windows Extensions which fully supports COM!

pythoncom.CoInitializeEx (pythoncom.COINIT_APARTMENTTHREADED)
gencache.EnsureModule ('{2DF8D04C-5BFA-101B-BDE5-00AA0044DE52}', 0, 2, 4)
gencache.EnsureDispatch ("PowerPoint.Application.12")
 
# Create an instance of PowerPoint and presentation
pp = Dispatch ("PowerPoint.Application.12")
pp.Activate ()
pres = pp.Presentations.Add ()
 
# Insert all the downloaded presentations
count = 1
for filename in os.listdir ('presentations'):
    pres.Slides.InsertFromFile (os.path.realpath (filename), count)
    count = pres.Slides.Count
 
# Save and exit
pres.SaveAs ('compiled.pptx')
pres.Close ()
pp.Quit ()
pythoncom.CoUninitialize ()

I was quite shocked at how straightforward it was to automate PowerPoint from Python, but this solution wasn’t quite good enough yet. Using InsertFromFile means that the imported presentation acquires the formatting of pres which is not what I wanted. Also, there appears to be a bug in PowerPoint 2007 which causes image references to be broken when importing from a .pptx into a .pptx with the COM interface.

Searching for a solution to the import with formatting issue lead me to this awesome VBA function which has been referenced many, many times. I ported this to Python and it worked perfectly! There was still the weird image problem, but I used a really crude fix for that:

for filename in [f for f in os.listdir ('presentations') if f[-4:] == 'pptx']:
    pres = pp.Presentations.Open (path.realpath (filename))
    pres.SaveAs (path.realpath (filename[:-1]))
    pres.Close ()

Yes. I converted all the .pptx’s into .ppt’s. Nothing intelligent here. Finally, I wrote a function to replace all the fonts added by the import with Arial. Job done.

fonts = {}
for i in range (1, pres.Fonts.Count + 1):
    fonts[pres.Fonts.Item (i).Name] = 1
fonts = fonts.keys ()
fonts.remove (u'Arial')
for font in fonts:
    pres.Fonts.Replace (font, u'Arial')

So after all that I have a Python program that logs into my email, downloads a load of PowerPoint files, converts them all to .ppt format, inserts them into a blank presentation, and then normalises the font to Arial. Not bad for a few hundred lines of code!

I did change the program a bit so that it would copy a template with title slides in and add the presentations to that instead of a blank file, and then set the date appropriately on the main title slide. But the key point here is that I’ve replaced my tedious Friday-morning activity with a single command: makepres.

Victory.

Yesterday I managed to download the front page of my website using libcurl. As good as that was as a learning experience, it wasn’t interesting or useful in the slightest. Today however, I decided to see if I could fetch my status updates from Twitter and display them in a program. So I had a look at the API documentation and it looks quite easy to use, with the exception of OAuth which I’m yet to get my head around. Thankfully, for now, basic authentication is still supported.

The Twitter API uses the REST (REpresentational State Transfer) paradigm which means there’s no concept of a state on the server; i.e. each transaction is considered separately. It also means that it uses HTTP, which is pretty simple to understand. Basically in a REST protocol the URI’s are objects in the system, and the HTTP verbs are how you interact with them. So a GET on a http://server/article?name=REST object would download an article named REST. Simple eh? Check this article if you’re interested.

Anyway, onto the meat ‘n’ taters. Data in a REST transaction is typically stored as XML or JSON. I considered downloading LibYAML and taking the JSON route but a) I already had Xerces, b) I understand XML more than JSON, and c) I couldn’t be bothered to learn yet another new thing.

Xerces is incredibly well written. If you look at the class listings of Xerces or Xalan you’ll appreciate they’re both enormous and support basically everything. In fact, right out of the box Xerces supports fetching XML documents over the internet using HTTP GET. I chose not to use this purely because I wanted to use libcurl. Thankfully libcurl is surprisingly easy to use:

static size_t _CurlWriteCB (void* ptr, size_t nLen, size_t cbElem,
                            CMemFile* pFile)
{
    size_t cbSizeAtStart;
    size_t cbSizeAtEnd;
 
    // Write data to file, but measure buffer size before and after.
    cbSizeAtStart = (size_t)pFile->GetLength ();
    pFile->Write (ptr, (UINT)(nLen * cbElem));
    cbSizeAtEnd   = (size_t)pFile->GetLength ();
 
    // Return the difference in buffer size, i.e. number of bytes written.
    return (cbSizeAtEnd - cbSizeAtStart);
}
 
BYTE* GetStatusesFromTwitter (char* szUserName, UINT& uiSize)
{
    // Attempt to initialise curl
    CURL* curl = curl_easy_init ();
    if (curl != NULL) {
        // Set up the http target
        CString strFmt;
        strFmt.Format (IDS_TWITTER_STATUS, szUserName);
        curl_easy_setopt (curl, CURLOPT_URL, strFmt.GetString ());
 
        // Save the result into memory for now.
        CMemFile buffer;
        curl_easy_setopt (curl, CURLOPT_WRITEFUNCTION, _CurlWriteCB);
        curl_easy_setopt (curl, CURLOPT_WRITEDATA,     &buffer);
 
        // Attempt to grab the data from Twitter.
        CURLcode res = curl_easy_perform (curl);
        curl_easy_cleanup (curl);
 
        // Return the data.
        if (res == 0) {
            uiSize = (UINT)buffer.GetLength ();
            return buffer.Detach ();
        }
    }
 
    return NULL;
}

The above listing will download a users tweets and store them in a growable buffer (see CMemFile). But now we have to present this to Xerces in a way that it will understand. Thankfully we can supply an arbitrary InputSource to a DOMParser, including one that will wrap a piece of memory.

bool DoGetStatuses (char* szUserName)
{
    // Query Twitter
    UINT  uiSize;
    BYTE* pbData = GetStatusesFromTwitter (szUserName, uiSize)
    if (pbData == NULL)
        return false;
 
    // Move the memory into an object Xerces understands.
    MemBufInputSource* pDataSrc = new MemBufInputSource
        (pbData, uiSize, L"TwitterXML", true);
 
    // Parse the data
    XercesDOMParser parser;
    parser.setValidationScheme (XercesDOMParser::Val_Never);
    parser.setDoNamespaces (false);
    parser.setDoSchema (false);
    parser.setDoValidation (false);
    parser.parse (*(InputSource*)pDataSrc);
 
    // Get the root node
    DOMDocument* pDoc = parser.getDocument ();
 
    //...
 
    // Free memory.
    delete pDataSrc;
    return true;
}

I was especially lazy in that last listing actually, because I told Xerces to adopt my buffer which means it’ll free it for me when it’s finished with it. Curiously though, even a memory object needs a system id which is the purpose L”TwitterXML” serves. With a DOMDocument in memory it was trivial to add the statuses to a list box.

I was quite surprised at how complex a task I’d achieved given the effort I’d put in; hats off to both Xerces and libcurl. Now that I’d managed to list my tweets, naturally the next step is to try and submit one! So I made a new dialog for the occasion:

Clicking OK causes some magic to happen:

int CPostStatusDlg::DoStatusUpdate ()
{
    static const char* cszUrl =
        "http://api.twitter.com/1/statuses/update.xml";
 
    // Attempt to initialise cURL.
    CURL* curl = curl_easy_init ();
    if (curl == NULL)
        return 0;
 
    // Configure the authentication
    CString strFmt;
    strFmt.Format (_T("%s:%s"), m_strUserName, m_strPassword);
    curl_easy_setopt (curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
    curl_easy_setopt (curl, CURLOPT_USERPWD,  (char*)strFmt.GetString ());
 
    // Format the string entire in C form.
    char* szStatus = curl_easy_escape (curl, m_strStatus.GetString (),
        m_strStatus.GetLength ());
    char szPostBody[BUFSIZ];
    sprintf (szPostBody, "status=%s", szStatus);
    curl_free (szStatus);
 
    // Set up the HTTP connection and use the POST method
    curl_easy_setopt (curl, CURLOPT_POST,           1L);
    curl_easy_setopt (curl, CURLOPT_POSTFIELDSIZE,  strlen (szPostBody));
    curl_easy_setopt (curl, CURLOPT_POSTFIELDS,     szPostBody);
 
    // Finally, set the callback function and the URL.
    CMemFile buffer;
    curl_easy_setopt (curl, CURLOPT_WRITEFUNCTION, _CurlWriteCB);
    curl_easy_setopt (curl, CURLOPT_WRITEDATA,     &buffer);
    curl_easy_setopt (curl, CURLOPT_URL,           cszUrl);
 
    // Now we can execute at last!
    int nResponse = 0;
    CURLcode res = curl_easy_perform (curl);
    curl_easy_getinfo (curl, CURLINFO_RESPONSE_CODE, &nResponse);
    curl_easy_cleanup (curl);
 
    // Check for success
    return nResponse;
}

Boom! Tweet submitted!

In the above listing, the growable buffer and the callback are largely to just eat the output from libcurl because we don’t really care about it. CMemFile will free the memory it allocated when the function returns too, which saves hassle. I originally wrote all the code listings with Unicode in mind which is why they might appear to be a bit odd. libcurl is an ANSI C library so you may need to convert your strings for it to work. Thankfully Xerces includes some basic support because it uses Unicode internally.

Little victory indeed.

I drafted a post about my adventures with trying to compile some open source software, but it turned out far too long and it didn’t actually concentrate on the point I was trying to make. So instead I’ll jump right to it.

Yesterday I decided, randomly, to have a look at libcurl. It’s a surprisingly powerful library for C that supports a huge amount of internet protocols with only a few function calls. The code examples on the website made it look easy enough to use, so I thought I’d download it and see what I can do with it. Unfortunately compiling it myself turned out to be annoyingly difficult, so I just downloaded the binaries. Naturally these were all in MinGW form and I needed them them to be MSVC compatible.

After some googling I found that MinGW comes with a tool called pexports which dumps all the symbols of a DLL. So using that I wrote a Bash script that does this for all the DLLs in a directory:

for i in $( ls | grep ".*\.dll" | sed "s/.dll//" ); do
    pexports "$i.dll" > "$i.def"
    echo "lib /machine:i386 /def:$i.def" >> makelib.bat
done

This creates a .def file for each DLL which lists all the symbols that must be exported, and adds a line to makelib.bat which will create the MSVC import library. The two pronged approach is unfortunately necessary here because the Microsoft LIB tool won’t run through the Cygwin prompt.

The first line might seem complicated, but it’s not too bad. ls lists all the files in the current directory, grep then singles out all the .dll files, and finally sed removes the trailing .dll. In the loop itself $i will therefore always be of the form “<mydll>” instead of “<mydll>.dll“.

I’m no expert with Bash scripting so there’s probably an easier way to do it.

I can’t remember what I was doing now, but for some reason I needed to copy the full path of some file into some program to do some… thing. I was finding it increasingly annoying that I had to copy the path from the Explorer window, then either hand-transcribe (complete with mistakes) or rename, select all, copy the file name.

“Why can’t I just right click and select Copy Filename or something?” I said to myself, “I wonder…”

So I searched CodeProject for some information on how to write my own damn Shell Extension (with Blackjack and Hookers) and stumbled across this fantastic article by Michael Dunn. In the space of an hour I had managed to learn enough to make half of my extension: I added a context menu item to Explorer!

Now over to MSDN to learn about the Clipboard and history was made. After around an hour and a half I’d gone from knowing nothing about writing Shell Extensions or using the Clipboard to having a working Shell Extension that used the Clipboard. I love the internet.

This is one of the smallest pieces of code I’ve ever written and, ironically, one of the few projects I consider myself to have finished. For more information, including how to download it, go to the CopyExt page.