Expect The Unexpected

In science unexpected results occur more often than we might think. These anomalies, the black swans of legend, range from the trivial to the momentous. They can be nails in the coffin of a cherished theory; more grist for the mathematical mill, or just simply errors.

Some are ignored by busy scientists as they pursue more fruitful lines of inquiry, while others are swept under the carpet by the lazy or unethical. In contrast, some are delivered prematurely into the full glare of publicity by those seeking fame and, perhaps, fortune.

So, how seriously should we take the ‘support’ that an unexpected result may appear to give an implausible therapy; like apparantly anomalous properties of water invoked in support of homeopathy, for example?

Tensions of this kind feature in the philosophy of Thomas Kuhn. For him anomalies can be both the fuel of revolution and the bread-and-butter of normal science.

How can we tell the difference? How do we assess an anomaly’s place on the “…continuum from the shocking to the anticipated result”? [p.62 – all page numbers in square brackets refer to “The Structure of Scientific Revolutions“, 3rd Ed., 1996]

Kuhn provides some clues that can enable us to at least weigh the likelihood that we are in the presence of greatness or hubris. This ‘Kuhnian Checklist‘ will emerge as we look at the following questions:

• Are there any features “…characteristic of all discoveries from which new sorts of phenomena emerge…”? [p.62]
• Should new sorts of phenomena (or theories) emerge easily from anomalies?
• Is the resistance of scientists to change a positive or negative quality?

So, should new sorts of phenomena emerge easily from anomalies? Well, whether or not they should, Kuhn’s historical analysis of science shows that, on the whole, they do not.

“…previous awareness of anomaly, the gradual and simultaneous emergence of both observational and conceptual recognition, and the consequent change of paradigm [disciplinary matrix] categories and procedures often accompanied by resistance…” [p.62]

In the early part of his treatise Kuhn lays particular emphasis on the importance of previous exposure to the anomaly. He seems to have been strongly influenced by psychological experiments of the period where observers resisted seeing anomalous variations of the familiar, like a red six of spades from a deck of playing cards [pp 62-63]. In experiments of this kind recognition of the anomaly emerged slowly with increased exposure. For some, it never occurred at all.

Later, he backed away from this simple analogy. Of arguments which asserted that the treatment of anomalies by scientists is a ‘gestalt process’ Kuhn concluded that they “…do not demonstrate that the careful and controlled observation exercised by the research scientist at all partakes of those characteristics…” [p.113]

However, his view did continue to be that “…novelty emerges only with difficulty, manifested by resistance, against a background of expectation…” [p.64]. As we explore his ideas we’ll see that this resistance is important.

So, how do anomalies arise in the first place?

Here we come to a provocative conundrum: if science between revolutions (Kuhn’s normal science) tends to resist change (in the form of new sorts of phenomena or new theories) how is it that anomalies arise as often as they do? How is it that science progresses as rapidly as it does?

Kuhn reflected on why: “…normal science, a pursuit not directed to novelties and tending at first to suppress them, should nevertheless be so effective in causing them to rise…” [p.64]. Then he provided an insightful explanation; one that helps us begin to separate the ‘wheat from the chaff’:

“…normal science leads to a detail of information and to a precision of observation-theory match that could be achieved in no other way…” [p.65]

It is the detail and precision of normal science that is the engine of anomaly. Without these traits “…the results that ultimately lead to novelty could not occur…” [p.65] His explanation goes beyond this to the values and training of the scientist:

“…novelty ordinarily emerges only for the man who, knowing with precision what he should expect, is able to recognise that something has gone wrong…” [p.65]

Only discipline and rigour in both the professional training and conduct of scientists sets the scene for the emergence of worthwhile anomalies. So, Kuhn is not talking about the results of sloppy experiments (like when, as a graduate trainee, I built a circuit that according to my measurements violated Kirschhoff’s laws!) but results that are unexpected to a scientist well versed in appropriate theories who conducts careful experiments.

Why does even this sort of anomaly face resistance? Well, it’s not because scientists as a group are doctrinaire, closed-minded or cannot ‘think outside the box’ as some faux-Kuhnians protest. Kuhn saw a powerful utility in such resistance:

“…by ensuring that the paradigm [disciplinary matrix] will not be easily surrendered, resistance guarantees that scientists will not be likely distracted and that the anomalies that lead to paradigm [disciplinary matrix] change will penetrate existing knowledge to the core…” [p.65]

Here Kuhn hints at his evolutionary view of scientific progress: a natural selection of the fittest ideas. Against a resistive, even hostile, environment only the most important anomalies will prevail.

This resistance to anomaly also has another vital benefit: “…The scientist who pauses to examine every anomaly he notes will seldom get any significant work done…” [p.82]. Every scientist chasing down every anomaly would bring scientific progress to a grinding halt. Not all real anomalies are particularly important; this is particularly true if there are fruitful lines of investigation still open within the disciplinary matrix. (I did some work recently on near-wake asymmetry in bluff body flows which, unfortunately, seems to fit into this category.)

If this seems a little cavalier, then it’s good to remember that science is a community activity. Even if this pragmatic individual conduct misses some opportunities for progress, history demonstrates that others in the community will make different judgements on the importance of the anomaly and pursue it.

This effort usually results in the accommodation of the unexpected result within the existing disciplinary matrix: “…there are always some discrepancies, even the most stubborn usually respond at last to normal practise…” [p.81]

So here is an important lesson from Kuhn: don’t get too carried away with the implications of an unexpected result: usually they are resolved without recourse to new theories; usually they are not a genuinely new sort of phenomenon.

What sort of anomaly starts a revolution?

“…if an anomaly is to evoke crisis, it must usually be more than just an anomaly. There are always difficulties somewhere in the paradigm [disciplinary matrix] -nature fit …an anomaly comes to seem more than just another puzzle of normal science…” [p.82]

This is where Kuhn, understandably, leaves the question open. If there was a criterion for deciding which anomalies were worth chasing ahead of time, the practise of science would be relatively simple. New discoveries would not require that rare combination of genius and rigour seen in Newton, Maxwell, Einstein, etc.

Still, we can see that a revolutionary spark needs to be a puzzle that the normal science of the day cannot solve; and likely will not.

So, what should we think of scientists who pursue anomalies? Are they wasting their time? Are they necessarily mavericks and cranks? Kuhn makes this very sensible observation: “…if … no one reacted … there would be few or no revolutions…” [p.186]

Under Kuhn’s community view of science the risk of pursuing the apparently novel, but most likely fruitless, is shared by the community. It’s rather like balancing the risk of investment by biasing your selections towards low to medium risk opportunities, whilst still investing smaller sums in riskier opportunities.

Similarly, most of a scientific community will likely pursue more clearly fruitful lines of research rather than exploring riskier anomalies. However, some will take the less travelled road. Most will find nothing worthwhile; a few will make a substantial contribution to human knowledge: the community shares the risk.

To enable this to happen, in Kuhn’s view, scientific investigation relies on shared values, rather than shared rules. If this is the case it argues forcefully that shared values need to be maintained within scientific communities.

What sort of values did Kuhn see as unifying scientific communities? He identified that “Probably the most deeply held values concern predictions: they must be accurate; quantitative predictions are preferable to qualitative ones; whatever the margin of permissible error, it should be consistently satisfied in a given field…” [p.185]

So the requirement for detail and precision that characterises normal science is a key value that carries over into investigating the unexpected.

When it comes to theory choice, Kuhn asserts that new theories must permit “puzzle-formulation and solution” [p.185]. If a theory did not, it wouldn’t be of much use. Where possible, he asserts, new theories should be simple, self-consistent, plausible and compatible with other theories currently in use.

These values are not likely to be applied in the same way by different individuals. It is clear that some may weight these values differently; valuing plausibility above simplicity, for example. The personality, culture and experiences of individual scientists may well lead them to weight or apply these values differently.

In fact, dealing with the unexpected may require the additional flexibility that the application of values rather then rules affords.

However commitment to these values is, according to Kuhn, “…deep and constitutive of science…” [p.185]. Conversely, it is perhaps reasonable for us to conclude that abandoning (rather than just weighting differently) the shared values of science in pursuit of novelty is a characteristic of pseudo-scientists; though Kuhn probably wouldn’t go that far.

These values are an important counter-balance to the beliefs of scientists. For example, Confirmation bias can cause an investigator to latch onto anomalies that seem to support a pre-existing belief. This could lead them to rush past mundane, but likely, explanations to arrive at wild conclusions having little or no plausibility.

However, if he is deeply committed to values such as theories should be simple, self-consistent, plausible and compatible with other theories this may hold in check the inclination to see a confirmation of a belief. If it does not, these values will lead the community to judge his ideas appropriately.

Therefore, real scientists apply core scientific values as they explore anomalies; but may vary the weight they give to individual values. The conclusions of the scientific community are the aggregate of many such judgements; thus the effects of individual biases are weakened.

On the other hand, pseudo-scientists pursue anomalies with all the ethics of an ambulance-chasing lawyer: scientific values are sacrificed on the altar of expected rewards.

So, we are now in a position to draw out some of Kuhn’s lessons into a checklist that should provide some help in assessing unexpected results.

A Kuhnian Checklist

1. Remember that resistance is to be expected, irrespective of the merits of the anomaly: it is not evidence for either the defence or prosecution.
2. Resistance is necessary: if too many scientists became anomaly chasers progress would be limited; if all scientists became anomaly chasers science would judder to a halt. It also ensures that only the most meaningful anomalies are taken into the body of scientific knowledge.
3. Science always works in the presence of some anomalies which are usually resolved; the presence of anomalies is not a predictor of impending revolution.
4. Progress is not inhibited by the values of science: normal science, for all its conservatism, is an excellent discoverer of anomalies. Kuhn remarked at “…the completeness with which that traditional pursuit prepares the way for its own change…” [p.65]
5. Don’t get too excited too soon, normal science is also an excellent tool for resolving anomalies: “…most anomalies are resolved by normal means; most proposals for new theories do prove to be wrong…” [p.186]
6. It’s not pursuing an anomaly that turns a scientist into a pseudo-scientist or a crank: it’s the abandonment for scientific values.
7. Most importantly, worthwhile anomalies are born from the pursuit of detail and precision in the observation-theory match: not sloppiness.

I thought that it would be an interesting exercise to examine some widely reported anomalies that have been cited as support from homeopathy and see how that stand against these ideas.

Anomalous Results and the Memory of Water

One of the things that got me thinking about unexpected results was the discussion surrounding the infamous ‘Memory of Water’ issue of the journal Homeopathy. Many of the papers reported anomalous measurements from which the editor of the journal inferred, if not support for the clinical practise of homeopathy, evidence for its basic plausibility.

So how do some of these anomalies fare against my proposed Kuhnian checklist? (in fairness I must note that Kuhn would not likely use these statements as a demarcation criterion: the ideas discussed here are Kuhn’s, but this part of the critique is mine.)

Let’s start with some general observations. The papers that presented possible mechanisms for water memory are necessarily ignoring the simple and well substantiated alternative: water has no persistant structures and therefore no memory.

It follows from this that any proposed long-term structures and potential memory mechanisms will lack plausibility. They are also incompatible with current theory.

These general observations indicate that the investigators featured in this issue are, perhaps, in danger of abandoning core scientific values. The most generous implication is that they attach very little weight to the values mentioned above.

Some of the authors complain about the resistance of the scientific mainstream to their ideas (e.g. Thomas, Vybíral and Voráček, Milgrom). As we have seen, from Kuhn’s perspective this is entirely expected and actually useful; it is not evidence either for or against.

How do the individual contributions stack up against my ‘Kuhnian Checklist‘?

In the issue Yolène Thomas describes some badly designed (overly complex and indirect) experiments purporting to show that the properties of a solution can be recorded and broadcast into another solution. This is just sloppy work; no precision in the match between theory and observation here. The values of simplicity, self-consistency, plausibility and compatibility are absent; rather than just given little wieght: I’d say that here we see a pseudo-scientist at work.

Vybíral and Voráček claim to have discovered a new sort of phenomenon: autothixotropy in water. This anomaly does seem to have been the result of reasonable experiments by people who knew what to expect. The problem is that they have not done the most basic scientific investigation of this proposed phenomenon; instead they have pleaded both financial and intellectual poverty.

In this case the detailed information that is required to bring a worthwhile anomaly to light is missing. Perhaps this is real, but there is not enough here to get excited about.

I don’t think that these authors are cranks or pseudo-scientists: on this evidence, they’re just not very good.

The results of Rao et al have the hallmark of sloppy experimentation. Kerr et al have recently drawn “…attention to serious anomalies and incongruities in the UV absorption data…” in this paper. The author’s refusal to engage with the serious criticisms made increases the chances that this anomaly is just the result of poor science. Again the qualities of detail and precision in the observation-theory match are absent.

Rey, like other authors in this issue, sets out to find evidence for what he already believes. His experiments are just sloppy science (e.g. lack of controls): no precision or detail. His anomalies are likely to be no more than errors of measurement and interpretation.

The paper by Elia et al admit that their experiments suffer from a lack of reproducibility:

“…It is important to emphasise that, from the studies so far conducted, we cannot derive reproducible information concerning the influence of the different degrees of homeopathic dilution or the nature of the active principle (solute) on the measured physicochemical parameters…”

They also admit a possible role for contaminants in some of their results. This is not work that demonstrates detailed investigation and precision in the observation-theory match. Like other papers in this issue the values of simplicity, self-consistency, plausibility and compatibility are ignored. In all likelihood their unexpected results are just the result of contanimants; a measure of sloppiness in their experiments. In my view: more pseudo-science.

Voeikov‘s theoretical speculation on a possible role for active oxygen in water memory suffers from a lack of self-consistency, as has been pointed out, he conflates speculative information from two sources which deal with very different experiments. It’s clear the outcome he wishes to arrive at, and his scientific values are not stong enough to pull him back on to a more rational course.

Anick and Ives’ paper advances the so-called silica hypothesis for water memory. This relies on the presence of free SiO₂ in the glassware used to hold homeopathic solutions during their preparation; that silica can form a template of the initial ingredient, freeing itself from the templated molecule and that such free templates can then self-replicate once the original solute has been diluted out of the solution. None of this is simple, remotely plausible or compatible with current theory and known observations. There is also a lack of self-consistency on display.

Anick‘s second paper suffers from trying to model the implausible. It may be self-consistent, but it’s not at all simple, plausible or compatible with current theory or experimental observations.

When reporting the results of experiments Weingärtner makes a virtue of inconsistency, implausibility, complexity and incompatibility. He treats analysis and theory in the same way. A typical tendency to assume the desired result and work from there is also clearly on display. My conclusion: a crank.

Milgrom‘s kind of quantum mechanical entanglement between patient, practitioner and remedy is an alternative speculation not related to water memory. I’d fail it on, among other things, the abject lack of scientific values, namely: simplicity, self-consistency, plausibility and compatibility. They are all conspicuous by their absence. I also think that it does not provide any scope for the formulation and solution of new scientific puzzles; it’s just a piece of post-hoc justification.

The abandonment of scientific values is, in this case, so clear as to make its advocate a crank.

The lack of precision in the use of his theoretical structure also indicates that this new theory will “prove to be wrong”.

(There is much more that could, and has, been said about these papers.  I have put together an index of useful critiques that are available on-line.)

Taking this body of work as a whole it is possible to see trends. Scientific values are mostly abandoned in pursuit of anomalies. Both experiment and theory are handled with sloppiness, rather than the precision and detail required for confidence in the reported outcomes.

Not all the authors are, by any means, pseudo-scientists and cranks: but some are. Of the genuine scientists here Vybíral and Voráček turn out to be apologists for homeopathy; this is perhaps an explanation for their lack of rigour.

I think the following quote from Kuhn provides an excellent indication of how the unexpected results advocated as demonstrating water memory are best viewed:

“…most anomalies are resolved by normal means; most proposals for new theories do prove to be wrong. If all members of a community responded to each anomaly as a source of crisis or embraced each new theory advanced by a colleague science would cease…” [p.186]

To sum up: always expect the unexpected, but don’t get over-excited.