Lost BoyA journal of no fixed aims or direction, by Leigh Dodds

CIDR 2009 looks like it was an interesting conference, there were a lot of very interesting papers covering a whole range of data management and retrieval issues. The full list of papers can be browsed online, or downloaded as a zip file. There’s plenty of good stuff in there ranging from the energy costs of data management, forms of query analysis and computation on “big data”, and discussions on managing inconsistency in distributed systems.
Below I’ve pulled out a few of the papers that particularly caught my eye. You can find some other picks and summary on the Data Beta blog: part 1, and part 2.
Requirements for Science Databases and SciDB from Michael Stonebraker et al, presents the results of a requirement analysis covering the data management needs of scientific researchers in a number of different fields. Interestingly it seems that for none of the fields covered, which includes astronomy, oceanography, biologic, genomics and chemistry, is a relational structure a good fit for the underlying data models used in the data capture or analysis. In most cases an array based system is most suitable, while for biology, chemistry and genomics in particular a graph database would be best; semantic web folk take note. The paper goes on to discuss the design of SciDB which will be an open source array-based database suitable for use in a range of disciplines.
The Case for RodentStore, an Adaptive, Declarative Storage System, Cudre-Mauroux et al, introduces RodentStore an adaptive storage system that can be used at the heart of a number of different data management solutions. The system provides a declarative storage algebra that allows a logical schema to be mapped to a specific physical disk layout. This is interesting as it allows greater experimentation within the storage engine, allowing exploration of how different layouts may be used to optimise performance for specific applications and datasets. The system supports a range of different structures, including multi-dimensional data, and the authors note that the system can be used to manage RDF data.
Principles for Inconsistency, proposes some approaches for cleanly managing inconsistency in distributed applications, providing some useful additional context and implementation experience for those wrapping their heads around the notion of eventual consistency. I’m not sure that’d I’d follow all of these principles, mainly due to the implementation and/or storage overheads, but there’s a lot of good common sense here.
Harnessing the Deep Web: Present and Future, Madhavan et al, describes some recent work at Google to explore how to begin surfacing “Deep Web” information and data into search indexes. The Deep Web is defined by them as pages that are currently hidden behind search forms and that are not currently accessible to crawlers through other means. The work essentially involved discovering web forms, analysing existing pages from the same site in order to find candidate values to fill in fields in those forms, then automatically submitting the forms and indexing the results. The authors describe how this approach can be used to help answer factual queries, and is already in production on Google. This probably explains the factual answers that are appearing on search results pages. The approach is clearly in-line with Google’s mission to do as much as possible with statistical analysis of document corpora as possible, there’s very little synergy with other efforts going on elsewhere, e.g. linked data. There is reference to how understanding the semantics of forms, in particular the valid range of values for a field (e.g. a zip code) and co-dependencies between fields, could improve the results, but the authors also note that they’ve achieved a high level of accuracy in automated approaches to identifying common fields such as zip code, etc. A proposed further avenue for research is exploration of whether the contents of an underlying relational database can be reconsistuted through automated form submission and scraping of structured data from the resulting pages. Personally I think there are easier ways to achieve greater data publishing on the web! The authors reference some work on a search engine specifically for data surfaced in this way, called Web Tables which I’ve not looked at yet.
DBMSs Should Talk Back Too, Yannis Ioannidis and Alkis Simitsis, describes some work to explore how database query results and queries themselves can be turned into human-readable text (i.e. the reverse of a typical natural-language query system), arguing that this provides a good foundation for building more accessible data access mechanisms, as well as allowing easier summarisation of what a query is going to do, in order to validate it against the users expectations. The conversion of queries to text was less interesting to me than the exploration of how to walk a logical datamodel to generate text. I’ve very briefly explored summarising data in FOAF files, in order to generate an audible report using a text-to-speech engine, and so it was interesting to me to see that the authors were using a graph based representation of the data model to drive their engine. Class and relation labelling, with textual templates, are a key part of the system, and it seems much of this would work well against RDF datasets.
SocialScope: Enabling Information Discovery on Social Content Sites, Amer-Yahia et al, is a broad paper that introduces SocialScope a logical architecture for managing, analysing and presentation information derived from social content graphs. The paper introduces a logical algebra for describing operations on the social graph, e.g. producing recommendations based on analysis of a users social network; introduces a categorisation for types of content present in the social graph and means for managing it; and also discusses some ways to present results of searches against the content graph (e.g. for travel recommendations) using different facets and explanations of how recommendations are derived.

In an attempt to put my various projects into the Public Domain, it seems that I’ve caused some confusion.
All I want to do is the following: label my code as being in the Public Domain, but require that people at least acknowledge the fact that they’re using something I wrote. I’d prefer it if people didn’t take anything I wrote and make a quick buck out of it, but I’m not adverse to my code being bundled in a payware application. But that’s a nice to have, I basically just want to give stuff away.
This lead me to start adding Creative Commons licences to my work. The Attribution-ShareALike licence seemed to exactly cover my requirements. Previously I’d either not included a licence, or labelled it as “Public Domain”. But I’d seen some code I wrote used verbatim with someone else’s name on it and that naturally upset me. I won’t go into details about who or where, but how hard is it to add an @author tag to Java source (or better yet, leave the one that’s already in there)?
So anyway, the CC licence seemed to fit. However when the Jaikoz developers contacted me a few months ago about reusing my MusicBrainz API they weren’t sure whether they could, as their application is payware. I said they could.
This week Henning Koch emailed me under similar confusion: would his application have to be similarly licensed. I didn’t think so, and that certainly wasn’t my intention. Koch pointed me at the CC FAQ entry that I’d stupidly overlooked:
CC licenses are not written for software. They should not be used for software…
But which one of the many licences should I use? Why does it have to be so difficult to give stuff away? I know creating new open source licences is discouraged but to be frank, its not that easy to pick and choose, and I’m not sure I want to wade through endless legal documents: I want to give stuff away, but be acknowledged. That’s it.
Why does open sourcing software have to be so difficult? It seems to me that the Creative Commons folk could help clean up this mess. They’re wading into scientific research, so why not software?
A nice example of how broken software licencing is, is this summary of the creative commons licences from a Debian perspective. Conclusion: they’re not free. Debian has a reputation for being particularly prescriptive, but this seems a little barking.
I guess the answer is either RTFL (Read the F’ing Licence), or just switch back to a plain “This work is in the Public Domain” statement.

Another great article from Sean McGrath: The mysteries of flexible software. Bang on the money.
I don’t know how many times I’ve encounted software (and yes, some of my own devising) that has all sorts of wonderful flexibility but in all the wrong places. Time spent factoring applications into layer cakes, introducing endless layers of abstraction may have some benefits, but exactly how often do you go through an application and rip out the entire persistence layer? And when you do, what’s the biggest hurdle: changing the code, or the data migration and testing involved to guarantee that you’ve not broken any of the data integrity? Exactly how often do you swap in and out XML parsers and XSLT engines?
I’ve been a keen advocate of design patterns for some time, but it’s easy to get carried away: achieving a particular design pattern (or set of patterns) becomes a requirement in itself and that in all likelihood isn’t going to affect the success of a product. The “Just Do It” aspect to XP is one obvious reaction to that experience. Renewed interest in more flexible, easy to author languages like Python is perhaps another.
Abstraction ought to be a defense mechanism. If a particular area of code or functionality is continually under change, then introduce some abstraction to help manage that change. Trust your ability to refactor. Don’t over architect too early.

For my sins I’ve been writing a bit of PL/SQL recently. It’s been nearly 4 years since I had to do that in anger and predictably I’ve forgotten way more than I remember. At the time I was responsible for redesigning the database for a Laboratory Information Management System used by researchers at Pfizer looking for new drugs. After redesigning the data model I had to write code to port from one to the other. That was a lot of code, and required a lot of testing. Fun project though, and an interesting application.
Of course now I know all about test driven development and the first thing that occured to me was: “how do I test this stuff?”.

Continue reading →