There has been controversy around p-values in recent years, often linked to issues with reproducibility in psychology.  p-values are also often reported in empirical software engineering papers. We haven’t yet seen widespread public controversy about software engineering studies, but that’s not because there aren’t problems!
The American Statistical Association has just released a clarifying statement about p-values. (pdf)
p-values are not inherently broken. The problems are about mis-interpreting them, about poor study design and practice, and about poor reporting. The ASA statement seems like a useful contribution to the debate.

In January, the the journal Synthese accepted and published the first of two papers of mine on the philosophy of engineering.  The second installment is now also accepted and published: “Critical rationalism and engineering: methodology” (author’s preprint here). Woot!  In the new paper I use the three worlds schema from the first paper to look at possible sources and responses to falsification of engineering theories.  I also discuss the growth of knowledge in engineering.  Finally, I talk about assurance in engineering.  There are perhaps more open questions than answers, but the questions are important and interesting.
Assurance is key for engineering.  Engineers design and create artefacts that other people use.  But engineers don’t just throw artefacts “over the wall” (or into the market) – they also warrant that those artefacts can be used to meet people’s needs. Those assurances don’t just get made up.  They are backed by explicit justifications – arguments using empirically-validated engineering theories.  For safety-critical systems, if those arguments are invalid or those theories are false, people will die or get hurt.  That’s why it’s worth understanding engineering epistemology.
 

What is engineering? Sometimes people think engineering is just the same as science, but in a new paper on the philosophy of engineering (preprint here), I argue why that’s not the case. Engineering is similar, but different to Science, and its epistemological issues are also similar but different.
I got into this question because of problems in assurance for software engineering and formal methods that are essentially philosophical problems. But having work available on the philosophy of engineering available should also help with perennial questions like “Is Software Engineering a field of engineering?” and “Is Computer Science a science?”.

Robin Milner gave a presentation “Is Informatics a Science?” at a conference at ENS, 10 December 2007, where he discussed the challenge of better understanding relationships between models in computer science – how they “explain” (specify, refine, implement, abstract, realise) each other. I don’t believe he captured these thoughts in a journal or conference paper, but the ENS presentation follows an earlier similar 2006 presentation (for which there is a transcript) on “Scientific Foundation for Global Computing” .
An audio recording of the ENS presentation exists.  I’ve created a PDF transcript of that recording.  However, I don’t have the slides that Robin presented – I’d be interested to have a copy if anyone could send me one.

What is software architecture? There have been many definitions. Here’s mine.
First let’s consider some of the earlier definitions. SEI has a huge collection of definitions on its website, including “classic” definitions, bibliographic definitions (stops in 1996?), “modern” definitions, and definitions submitted from the community.  Perry and Wolf (1992) have perhaps the most classic definition, though it’s a little sketchy:

Architecture = {elements, form, rationale}

where elements are Processing, Data, or Connecting elements. Taylor et al. (2010) note that when people talk about software architecture in terms of Components and Connectors, that’s an over-simplification of Perry and Wolf’s definition – over-simplified because it doesn’t always work. For example in REST, Data elements are pre-eminent.
The ANSI/IEEE 1471-2000 definition expands on Perry and Wolf’s definition, and also slips environment into the scope of form (relationships).

Architecture is the fundamental organisation of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.

Elements (components) and their form (relationships) are clearly key to understanding what software architecture is, but many people think that’s all it is! Instead, software architecture researchers have understood for a long time that rationale (design/evolution principles) is also a key part of what software architecture is.
But contrari-wise, software architecture is not just rationale. So although Taylor et al. (2010) is a great architecture textbook, their definition of software architecture isn’t so great:

A software system’s architecture is the set of principal design decisions made about the system.

I would instead say that design decisions are the means by which elements, form, and rationale are created. The design decisions are not the architecture per se.
Software architecture is commonly misunderstood to be an exclusively structural model. Perhaps that’s because UML class diagrams and deployment diagrams are often presented as iconic for software architecture. The definition from the Bass et al. (2008) classic textbook also encourages this view:

The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them.

But there has been a shift in the software architecture research community to think about architecture more abstractly: in terms of rules or constraints or styles. So instead of “structures”, I prefer using the word “abstractions”, to more easily accommodate a rule-based architectural perspective. The abstractions here are not just of software, but also non-software elements in the system, including the environment. Fielding (2000) also talks about abstractions this way (though for some reason he limits software architecture to run-time, which I don’t agree with):

A software architecture is an abstraction of the run-time elements of a software system during some phase of its operation. A system may be composed of many levels of abstraction and many phases of operation, each with its own software architecture.

Apparently Clements et al. (2010) has another new definition:

The software architecture of a system is the set of structures needed to reason about the system, which comprise software elements, relations among them, and properties of both.

Here we have (ignoring “structures” for the moment) Perry and Wolf’s elements and form (relations) again, but now also with properties for each. Perry and Wolf’s rationale has not disappeared, but here appears as a qualifier (“needed to reason about”). I like this. I think the whole point of architecture is abstraction and analysis of a system for particular purposes. Here the “reasoning” objective implicitly encompasses those purposes and analyses.
I’d slightly prefer to say “communicate and reason” instead of “reason”, though perhaps you could say that understanding a communication is-or-requires reasoning. I’d also prefer to talk about “a” (not “the”) software architecture of a system (and similarly “a set” not “the set”), to more readily accommodate multiple views/perspectives of a single system.
So, in summary, my definition would be something like:

A software architecture of a system is a set of abstractions needed to communicate and reason about software in the system. A software architecture models elements of the system and its environment, relations among those elements, and properties of those elements and relations.

Just heard in a QESP webinar on Software Innovation in Australia from Julian Day of the Australia Consensus Awards:

In business, invention is the conversion of cash into ideas, but innovation is the conversion of ideas into cash.

Nice.  I see this is also on wikipedia.  I wonder what’s the original source for this quote?

Researchers “stand on the shoulders of giants“, which in practice means reading a lot of academic papers and reports.  Lots.  You not only want to read them, but also cite them in papers you write, search them, and organise them by whatever topics you’re investigating.  How do you do that?
When I was a PhD student, I kept hard copies of the papers I read, and a collection of bibtex files containing reference information.  The bibtex files were separate, each on a different topic.  Since getting back into research in 2004, I’ve gone digital and have tried a few solutions: Reference Manager, EndNote, bibtex again, and zotero.  After each one, I kept reverting to my clumsy manual approach: storing PDF documents in directories, with each directory representing a topic.  Often, papers relate to more than one topic – sometimes then I put a copy or soft link in each topic directory.  I said it was clumsy!  But at least I can work when I’m travelling and off-line.
As a result, I now have dozens of directories stuffed with thousands of PDFs.  (I have less than three thousand, but a colleague has more than ten thousand.)  These calcified directories represent a fixed collection of topics that, as my research focus evolves over the years, is increasingly inappropriate.  I suspect a lot of people work like this.
So I’ve been delighted to discover Mendeley Desktop.  It’s still in beta, but I like its approach.  It lets me keep my PDFs as they are, and works with me to index them and import their bibliographic information into a database, using text recognition and bibliographic web services.  The quality of both of those mechanisms are a bit patchy right now, but it has “needs review” status tracking so I can manually check and correct that bibliographic data over time.  What’s also cool is that I can tell it to “watch” my directories – if I dump more PDFs in there, it’ll incrementally import those too. Mendeley has all the normal features: reference-importing bookmarklets, exporting in bibtex/whatever, Word and Open Office plugins for creating reference lists, etc.  And of course it also has tagging: so now I can create tag topics for my references – organised into as many overlapping topic areas as I need.
I feel like my reference collection has opened up to me, and is becoming a much more useful resource.  That’s fun and exciting but I have to make sure I don’t spend so much time organising my references that I end up not actually doing research!

I had to laugh at Liming’s latest micro-blog posting, Why are papers in top conferences very boring (these days)? It’s funny, but I’m not sure I entirely agree – I think top conferences do have interesting papers.  Liming is saying interesting ideas won’t necessarily have had time to be well validated, and by the time you have validated and published your idea in a top conference, it’s no longer new (and interesting).  However, I don’t want to see completely unvalidated ideas.  Ideas are cheap.  I want to see ideas that are realisable, and whose value has been described and justified somehow.
To the extent that Liming’s wry diagram is true, I think it’s more true of journals than conferences. In most academic disciplines, journals are regarded as the “proper” place to publish significant results.  Computer science is different – top conferences in computer science (and software engineering) can be more important than journals.  Citeseer statistics show most of the highest impact compsci venues are conferences, and even some workshops have more impact that some top journals!  But (or perhaps because!) in computer science, journals have longer review and publication lead times than conferences, so the results there can be more out-of-date and so less interesting.  (That is a bit odd when you think about it – journals are published several times a year, whereas each conference happens at most once a year – surely journals should be able to be more responsive than conferences in publishing new results?!)
Anyway, it makes me wonder how Ricky’s citemine system would work in the conference milieu.  I guess for maximum market efficiency in citemine, the evaluation for new papers should take place in public. So, no workshop or conference would ever have “new” results – everything that made it through “review” (weighted average market price over the period since the last conference greater than some threshold for papers within some discipline boundary?) would have been published for the best part of a year.  Conferences would be more like the Oscars – glorifying new exciting productions – rather than a way of learning about recent results.  Maybe that’s OK – I think the greatest value of conferences is networking and nuance, and you would still get that at a Computer Science discipline’s “Academy Awards”.  But these would be very different events, and norms of academic precedence would need to be re-conceptualised.

WICSA was fun.  I usually find the most I can hope for in a conference is 1 or 2 papers that are really interesting, but I think WICSA cleared 5, so it was well worthwhile.  What I particularly enjoy about conferences is hearing how people verbally describe the ideas and challenges in the field.  You can get so much more nuance and emphasis from hearing people talk about their research, compared to just reading papers.
A great example was the final keynote for the conference, by Alexander Wolf.  He covered reflections on his personal history working in software architecture, but as one of the “fathers” of the field, his talk was also a history of early software architecture research.  It was fun to play spot the co-authors in the audience and also among other acquaintances.
He talked about the importance of simulation and experimentation for architecture, and called for more work to be done in the area.  At NICTA, Jenny Liu and Paul Brebner have been leading work in these areas, particularly for performance analysis of enterprise architectures.  They’ve been getting huge interest from industry.  It’s a very promising approach and I can support the observation that simulation and experimentation are critically important to the discipline of software architecture.
Alexander Wolf was also previously involved with Software Configuration Management research, which is an interest of mine.  He didn’t really elaborate on that line of work, but he did mention a paper of his discussing the relatedness of software architecture and configuration management.  I think there’s still a lot more that can be said in this area, particularly concerning architecture evolution.

How can you tell what the next big thing is going to be? Google’s pagerank algorithm will tell you what web pages have been important enough in the past for other people to have linked to.   Google trends will tell you what search terms people have been using recently, again in the past. What about the future?
Some predictions about the future are doomed to failure.   For example, Popper’s Poverty of Historicism is largely about the futility of predicting future society. However, some aspects of the future are largely predictable – science and technology work because they accurately predict the behavior of the physical world.  There’s a large middle ground of futures that aren’t easy to predict.  Prediction markets have been proposed as a way of getting better-than-chance predictions of these events.
Ricky Robinson at NICTA has recently launched citemine – a prediction market for academic papers.  The predictions being made are about how much each paper will be cited by other papers.  Ironically for citemine, one of the poverties of historicism that Popper identifies is a poverty of imagination about the possibilities of the impact of future science and technology! (Still, I imagine that Popper’s criticism only applies to long-term predictions of the impacts of science on society, not the shorter-term predictions of the importance of recently published scientific papers.)
The benefit of citemine is that it can be a leading indicator of the quality of publications, whereas existing citation metrics are very lagging indicators of the quality of publications and researchers.   Ricky’s hope is that academics will care enough to trade in citemine to acquire its “Reals” which may become a widely recognised measure of academic reputation.  Your personal worth in Reals is a measure of two things: your ability to have written highly cited papers, and how much better you have been than others at spotting papers that will be highly cited.  You can tell how much of your worth is due to each different source.  (Interestingly, I think both of these are lagging indicators, despite that the market price of a publication is a leading indicator.)
Even if such a market could work well if universally adopted and in a steady state, it’s a challenge to launch it.   It’s a chicken and egg problem – activity is required to make the market function, but a functioning market is required to generate interest in being active in the market.  The market has to bootstrap Reals into having value in the real world somehow.
citemine is “very beta”, and there are certainly a few issues at the moment:

• Some matches aren’t being made in the market – there are buyers and sellers at the same price who aren’t doing a deal.   (Looks like a bug?)
• There’s currently very low market depth, especially among sellers.
• There’s no sophisticated market overview mechanism – just a list of papers at their current prices.
• There’s no market metrics for papers – e.g. historical returns, price volatility, etc.

Ricky’s paper explains citemine.  I have two queries, and two observations…
Is citemine a zero-sum game?  In citemine, Reals are given to shareholders as dividends based on citations, but those Reals come from the previously-paid cost of submitting the citing papers.  So it looks like a zero-sum game. In my limited understanding of the economics of real stock markets, value gets created through primary production and through productivity improvements in other sectors.   I don’t see how that happens in citemine. Which leads me to my next query…
Is citemine a pyramid/ponzi scheme?  In citemine, the only source of new Reals is from the registration of new users, whose initial allocation of Reals is used to submit papers to pay dividends for existing users.  This question is more stark because there’s no leverage in citemine (debt, shorts).  Maybe I’m just confusing value with liquidity.
My intuition is manuscripts in citemine will behave more like mining stocks than industrial stocks in real stock markets. (Is that why it’s called citemine? )  Mines have a limited finite quantity of ore, and the value of the stocks for that mine decrease as the ore is removed from the mine.   The value of a manuscript in citemine derives from future citations, but for almost all scientific papers, there is a finite time horizon for possible citation.  At some point people lose interest in moderately influential papers and cite later derived works.  Even very influential papers become part of assumed/background knowledge and get cited less.  I think that in citemine, most manuscripts will trend to a near-zero market price.
Finally, there’s a “meta-gaming” anomaly currently at play in the citemine market. If it turns out to be a successful market, then Reals get real value, and Ricky’s citemine paper (and closely related papers by other authors) will also inevitably be highly cited.  If the market turns out to fade into obscurity, then the free Reals you get on joining stay as play money, so it doesn’t matter how you will have spent them.   Ricky’s paper (and related papers) are a safe bet – you can’t lose!  I would have bought some, but no one was selling – and I have no idea about how to pick a good price to offer!