Posted by Anil Madhavapeddy
Sat, 01 Nov 2008 16:17:00 GMT
My favourite session of PDC 2008 was by Anders Hejlsberg as he described the Future of C#. Anders is an excellent speaker, and it was educational watching how he made fairly complex type systems come across to an audience more used to simpler programming languages. I say this after attending ICFP recently, where that wasn't true of all the presentations!
The high-level message was exactly what I wanted to hear: in order to exploit multi-core processors more effectively, Microsoft's languages are focussing on improvements in three areas:
Declarative constructs: making the programmer define overall goals and constraints enables a tool-chain to more effectively scale computation across multiple processors. An example of this in C# 3.0 is the Language Integrated Query (LINQ) domain specific language for defining queries. Dynamic interoperability: Although Anders is a big fan of static type systems (robust, high-performance, enable intelligent tools), he pointed out that a lot of the systems that C# programmers have to deal with are fundamentally dynamic. Examples are web interfaces such as SOAP/REST APIs, COM objects for talking to other Microsoft applications, or scripting language integration (e.g. Python or Ruby). Concurrent programming: dealing with multiple cores requires splitting up computation into parallel threads, and the resulting parallel mass of code is often hard to debug (e.g. difficulties of reproducing obscure concurrency issues) and statically reason about.
An interesting aside was his assertion about "co-evolution". Microsoft have a number of languages which have recently been unified under the CLR, such as Visual Basic, C# and most recently F#. Rather than have one language race ahead of the others, they like to "borrow" features as appropriate into other languages. This is obviously made easier by having a common run-time foundation, and an example of co-evolution is the introduction of lambdas into C# 3.0 as obviously seen in F#.
Concurrency
Anders observes that all the attempts to automatically parallelize software has not yielded good results in the past, and so support needs to be built into the language to do "top-down" parallelism instead of the compiler inferring it from the bottom-up. To this end, they are introducing parallel extensions into the .NET framework. This makes use of functional features to parallelize code, including LINQ queries or CPU-intensive compute.
For an example of how this code looks, check out Jurgen Van Gael's post on using F# with the Task Parallel Library. I wanted to learn more about this, and wandered over to the Hands On Labs where F# and Visual Studio 10 were all pre-installed. I then ran into an extremely cool feature of F#... asynchronous workflows.
F# introduces an extension to the usual let ML operator in the form of let!. As Don Syme explains on his blog ), this can be interpreted as "run the asynchronous computation on the right and wait for its result. If necessary suspend the rest of the workflow as a callback awaiting some system event". So this construct lets you write straight-line code which can potentially block, without the hassle of spawning threads or encoding continuation passing style constructs in the code. Already a nice improvement over OCaml! (although to be fair I've not had a chance to check out some of the parallel OCaml extensions).
Dynamic Programming
Dynamic programming is the main focus of C# 4.0. In order to support dynamic objects as first-class citizens in a statically typed world, they introduce a dynamic static type. This forces all method resolution on that object to happen at run-time and disables static checks by the compiler (aside from looking from mixups between dynamic and static objects). The dynamic type now makes it easy to wrap support for Python, Ruby and Javascript, since the relevant dispatch functions can all be hidden away in the method resolver at run-time, leaving the programmer with a single syntax for invoking methods across these different languages.
The actual definition of method resolvers is pretty straight-forward; he demonstrated custom getters and setters (similar to Python for example) by using the IDynamicObject interface to define actions to take when properties are accessed. His example did the usual dictionary wrapper which mapped setting arbitrary properties onto an internal dictionary variable.
Another improvement in this space is the addition of optional arguments and labelled arguments. Both of these have well-defined semantics (optional arguments have to come after non-optional ones, and evaluation of arguments is left-to-right) and are purely syntactic improvements with no run-time cost. One of the best examples he showed of using these around COM interoperability. In current versions of C#, due to the lack of named arguments a common function such as "Save As" might require 12 or more stub arguments to be specified as ref missing. Now, those long, repetitive lines can be folded down to only the arguments which are required.
Sort of related to the earlier co-evolution was his demonstration of how similar Javascript and C# syntax is now with this new support. He took a Silverlight 2 code snippet written in Javascript and ported it over to C# 4.0 with some very mechanical changes. The languages are definitely converging fast!
If you're interested in checking out more on this topic, look at IronPython, the Dynamic Language Runtime (DLR), and the TL10 session on dynamic languages on .NET. The DLR is currently maintained as part of IronPython, but is being moved out to sit on top of the CLR for more languages such as Visual Basic, Javascript or COM. The DLR covers common optimizations which are useful for scripting languages, such as call-site caching, dynamic dispatch and expression trees (e.g. for LINQ). The DLR has a bunch of binders which bridge between different backends such as .NET, Silverlight, the native Python or Ruby backends, or COM applications such as Microsoft office.
Meta-programming
Later on at the Future of Programming Languages panel session, Anders talked about meta-programming as being one of the future improvement he's looking at. Currently, there is a lot of ad-hoc code generation in place when creating Windows applications, and unifying this into the language would give safety and maintainability improvements.
In order to do this, for C# 5.0 they are rewriting the compiler to be self-hosting in C#, since it has historically been a C++ application. This permits them to switch the compiler from being a traditional "black box" compiler to a hosted .NET service which can be called directly by .NET programs in order to do dynamic run-time compilation of code. Other portions of the compiler chain are also exposed to permit incremental program construction by third-party code.
He demonstrated this with a pretty nifty C# top-level, into which he directly typed Winforms code to construct a window with a few simple buttons using the C# compiler server. Not to be outdone by this, Miguel de Icaza promptly upstaged Anders at his (fantastic) Mono 2.2 session. He demonstrated the new C# shell which is present in Mono trunk builds and can essentially be used like an OCaml or Python top-level to mess around and manipulate C# code. He also talked about embedded Mono and SIMD support which pushes their compiler ahead of Microsoft's in the 3D performance game.
Summary
I'm firmly convinced about the potential of F# now. I had the opportunity at the Open Spaces area to quiz Scott Guthrie about whether or not F# was a toy language. He replied using the same arguments as Anders that the higher-level language approach (declarative, functional) was very important strategically to Microsoft to let their developer platform continue to survive in a multi-core world.
This boils down to the individual languages not being that important any more (as seen by the sharing of features between C# and F#), and the underlying execution layer (the CLR/DLR) adding efficient support. Now any old language can adopt higher-level features without having to re-do all the optimization grunt work again and again. Much like Xen offers a new golden age for innovative new OS research by freeing programmers from writing a million hardware device drivers, it looks like .NET is ushering in a new age of programming language innovation!
Inspired by the PDC talks, I've got MonoDevelop and F# up and running on my Macbook Air, and am just playing with GTK# and CocoaSharp#. If this works as well as OCaml, then it might finally be time to abandon the old stalwart and move to a new language for my day-to-day stuff!
Posted in research, hacking | Tags pdc2008 | no comments
Posted by Anil Madhavapeddy
Thu, 17 Apr 2008 10:58:00 GMT
I've been looking around at refreshing my current photo gallery to make it easier to update, since I'm around 1000 photos behind over the last year. Part of making it easier to use is to adopt an Aperture-based system of marking the metadata directly in the image itself, and figuring out how to render so many pictures more effectively in my gallery.
The best library I could find to do the job is Exiv2 and its associated Python bindings pyexiv2. They use the rather complex Boost.Python bindings, which are a total pain to compile on MacOS X Leopard.
In order to get it to work:
- Use the latest Fink to install:
python25boost1.34.python25libexiv2scons
- Download the latest pyexiv2 distribution, extract it, and drop in my exiv2 Leopard Makefile into the extracted directory.
- Type in:
- Test it out by running python2.5 and experimenting with importing pyexiv2 (see the pyexiv2 developer guide for more information.
I'll tidy this up into a fink package sometime, but for now I want to press on with finishing my Django experiments.
Posted in hacking | Tags macosx | no comments
Posted by Anil Madhavapeddy
Sat, 30 Dec 2006 01:11:00 GMT
Inspired by finishing my PhD corrections (!) today, I decided to hook up the DNS server from our Melange project up to the Internet. The authoritative server is called deens (since the co-author is one Tim Deegan, geddit?), and is written in pure OCaml.
This is all rather experimental, to put it mildly, but I stuck in the zone file below, hooked it up as a delegate to our main name-servers, checked it against the DNS Report, and it all seems to be working!
$ORIGIN deens.recoil.org. ;
$TTL 240
deens.recoil.org. 604800 IN SOA (
deens.recoil.org. anil.recoil.org.
2006122401 3600 1800 3024000 1800
)
IN NS ns1.deens.recoil.org.
IN NS deensns.recoil.org.
ns1 IN A 194.70.3.132
dynamic IN CNAME dynamic.recoil.org.
static IN CNAME static.recoil.org.
anil IN CNAME dynamic
stats IN CNAME dynamic
I also modified stats.recoil.org to be an alias to stats.deens.recoil.org, so all the requests for that domain will go via the deens setup. You actually need a user/pass to access the site, but that doesn't matter; if it gets that far, the DNS bit has worked.
There's still an awful lot of tedious work to get the server into a production-ready state, such as proper logging, more error handling and recovery, etc., but I really hope to find the time in 2007 to polish this up somewhat. Performance is excellent already; faster than BIND by quite a lot, and it can optionally use more memory to cache responses to shoot up to crazy levels.
Incidentally, the dig replacement utility also seems to be working fairly well, and David Scott has been messing around with a Bonjour implementation that will get finished sometime in 2007 as well (honest!).
Posted in research, hacking, net | no comments
Posted by Anil Madhavapeddy
Wed, 27 Dec 2006 20:59:00 GMT
Our lighttpd setup has been very unstable in recent months, probably brought on by the load of the large Mercurial repositories hosted on Recoil since the Google Summer of Code mentoring.
The source of the instability was really hard to track down, but it seems to be the automatic spawning of FastCGI processes by the web-server, and lighttpd failing to handle a SIGCHLD somewhere when a child process crashes. To sort this out, I just converted all the Ruby on Rails setups (this blog and Nick's) to use an external spawn.
This only leaves our Mercurial vhost hg.recoil.org to switch to using FastCGI, and I couldn't find a module for this anywhere and so lashed up some Python glue to do the job.
You can download the small distribution for Mercurial 0.9 (hg-fcgi-0.9.tar.gz). It has a FastCGI library written by someone else, the Python files to glue the Mercurial and FastCGI libraries together, and a simple rc script to launch the external web process.
Read more...
Posted in hacking, net | no comments
Posted by Anil Madhavapeddy
Mon, 28 Aug 2006 22:12:00 GMT
I noticed on the OCaml mailing list that Practical OCaml is due to be released in the US quite soon. Could this be the first real competition that Jon Harrop will face for his excellent but pricey Objective Caml for Scientists ?!
Things are heating up on the OCaml book front at last, and Jon has been getting some rave reviews of his book! We have a long-running joke about how much I hate the colour syntax highlighting he used in his book, and it amused me greatly to note that someone else in the comments for the rave review of his book also shared my heretical opinion, ha ha!
Incidentally, I just pushed the beginnings of an OCaml config file parsing library to the Melange source tree. The code in there is almost beginning to look useable...
Posted in hacking, books | 5 comments | no trackbacks
Posted by Anil Madhavapeddy
Sat, 15 Jul 2006 02:45:00 GMT
Ever since I got my first Powerbook back in 2000, I've been meaning to learn Cocoa and Carbon to hack on MacOS X GUIs. On the flight over to San Francisco, I finally found the uninterrupted time to knock up a simple GUI interface to my gallery meta-data files. Overall, the experience was pretty positive. It's definitely more satisfying than my experiences with Windows and UNIX GUI programming, mainly because the result is so pretty!
Objective-C has a really nice dynamic message dispatch mechanism, used to good effect by Cocoa. The autorelease mechanism plays well with C memory management to provide a reference-counted interface which is fairly well abstracted. My application still leaks memory like a sieve though; I miss OCaml! Cocoa provides a huge number of ways to do the same thing. Bindings are the fancy new thing, MVC is the older "paradigm", and of course Carbon predates all of this. The documentation is a bit of a mess for a beginner to the framework, as crucial facts on how to get simple stuff done are scattered across a myriad of documents. Thanks Google. Tiger has some funky new Cocoa controls. Check out the use of NSTokenField on the screenshot which lets the tag entry field auto-complete just like Mail does! The APIs can be unfortunately verbose at times, as the following code snippet which does a little bit of simple string manipulation shows.
NSArray *a = [tag componentsSeparatedByString:@" "];
iter2 = [a objectEnumerator];
while (s = [iter2 nextObject]) {
NSString *e = [s stringByTrimmingCharactersInSet:
[NSCharacterSet whitespaceCharacterSet]];
if (![e isEqual:@""])
[tags addObject:[e retain]];
}
I'm quite looking forward to doing more work on this simple application now that the basics are mastered. Hmm... preferences panes next, then a bit of Cocoa bindings, and perhaps I feel OCamlCocoa coming on...
Posted in hacking | 2 comments | no trackbacks
Posted by avsm
Fri, 23 Sep 2005 14:38:10 GMT
Well I've been back in Cambridge for a week now, catching up on stuff that's been languishing in the last month of travelling. So here are some blog quickies!
- Shouts out to the newest blogger, the least German-like German person I know! Whooooop yeah yeah...
- Released a new version of OpenFX, with some really cool new features from Dr. Stuart Ferguson!
- Sorry about the backlog of OpenBSD-related things; I think I've caught up on all my mails, and imported and updated lots of bits and bobs and ports. Please re-mail me if I've ignored anything you have sent me recently.
- Released a test version of review2atom, the result of my ocamlduce hackery (and many thanks to Alan Frisch for answering my incessant questions!).
- Submitted DarwinPorts for calendar (dports/devel/caml-calendar) and ocamlduce (dports/lang/ocamlduce).
- I'm going to buy a juicer; turnip smoothie is en route!
Posted in cambridge, hacking | no comments
Posted by avsm
Fri, 24 Jun 2005 13:14:37 GMT
I bought a Nokia 7610 some time ago, and have taken hundreds of pictures using it. When creating the gallery section of my site, I ran into an annoying limitation: the pictures contained on the phone did not embed the date the picture was taken as part of the image. Manually copying the date and time for 350 images was a bit much as well...
The problem with sending images via Bluetooth to my laptop is that the OBEX transfer stamps the date of the transfer instead of the date the picture was taken. There turned out to be a conveniently easy answer to this problem though, as Nokia released a version of Python for Symbian Series 60. This suddenly elevates the phone from being a complete pain to program (due to the Symbian C++ libraries being amazingly badly documented), to something that's actually quite usable.
It took about an hour to come up with the first cut of imgcollect.py, which assembles a list of mtimes for each image on my phone, performs Bluetooth Device Discovery, and transmits the mtimes and all the pictures to my laptop. After that, a little shell script on the laptop is enough to stamp the pictures with the correct mtime:
#!/bin/sh
while read FNAME MTIME; do
DATE=`date -r $MTIME "+%G%m%d%H%M.%S"`
touch -t $DATE $FNAME
done
I'm actually quite impressed with Nokia at last! Next step is to work out a two-way synchronization so that I don't keep re-transmitting images which I already have on the phone (unfortunately, iSync is pretty useless with the Nokia 7610, even in Tiger).
Posted in hacking | no comments
| 
