In algorithms, as in life, persistence usually pays off.

― Steven S. Skiena

Over the past year I’ve opined that algorithm (method) research is under-supported and under-appreciated as a source of progress in computing. I’m not going to backtrack one single inch on this. We are not putting enough effort into using computers better, and we are too much effort to building bigger, less useful and very hard to use computers. Without the smarts to use these machines wisely this effort will end up being a massive misapplication of resources.

The problem is that the issues with algorithm research are even worse than this. The algorithm research we are supporting is mostly similarly misdirected. It turns out we are focused on algorithm research that does even more to damage our prospects for success. In other words, even within the spectrum of algorithm research there isn’t an equality of impact.

The fundamental law of computer science: As machines become more powerful, the efficiency of algorithms grows more important, not less.

— Nick Trefethen

There are two fundamental flavors of algorithm research with intrinsically different value to the capability of computing. One flavor involves the development of new algorithms with improved properties compared to existing algorithms. The most impactful algorithmic research focuses on solving the unsolved problem. This research is groundbreaking and almost limitless in impact. Whole new fields of work can erupt from these discoveries. Not surprisingly, this sort of research is the most poorly supported. Despite its ability to have enormous and far-reaching impact, this research is quite risky and prone to failure.

If failure is not an option, then neither is success.

― Seth Godin

It is the epitome of the risk-reward dichotomy. If you want a big reward, you need to take a big risk, or really lots of big risks. We as a society completely suck at taking risks. Algorithm research is just one of enumerable examples. Today we don’t do risk and we don’t do long-term. Today we do low-risk and short-term payoff.

Redesigning your application to run multithreaded on a multicore machine is a little like learning to swim by jumping into the deep end.

—Herb Sutter

A second and kindred version of this research is the development of improved solutions. These improvements can provide lower cost of solution through better scaling of operation count, or better accuracy. These innovations can provide new vistas for computing and enable the solution of new problems by virtue of efficiency. This sort of research can be groundbreaking when it enables something to be done that couldn’t be reached due to inefficiency.

This is the form of algorithm research forms a greater boon to efficiency of computing than Moore’s law has provided. A sterling example comes from numerical linear algebra where costly methods have been replaced by methods that made solving billions of equations simultaneously well within reach of existing computers. Another really good example were the breakthroughs in the 1970’s by Jay Boris and Bram Van Leer whose discretization methods allowed an important class of problems to be solved effectively. This powered a massive explosion in the capacity of computational fluid dynamics (CFD) to produce meaningful results. Without their algorithmic advances CFD might still be ineffective for most engineering and science problems.

The third kind of algorithm research is focused on the computational implementation of existing algorithms. Typically these days this involves making an algorithm work on parallel computers. More and more it focuses on GPU implementations. This research certainly adds value and improves efficiency, but its impact pales in comparison to the other kind of research. Not that it isn’t important or useful, it simply doesn’t carry the same “bang for the buck” as the other two.

In the long run, our large-scale computations must inevitably be carried out in parallel.

—Nick Trefethen

Care to guess where we’ve been focusing for the past 25 years?

The last kind of research gets the lion’s share of the attention. One key reason for this focus is the relative low risk nature of implementation research. It needs to be done and generally it succeeds. Progress is almost guaranteed because of the non-conceptual nature of the work. This doesn’t imply that it isn’t hard, or requires less expertise. It just can’t compete with the level of impact as the more fundamental work. The change in computing due to the demise of Moore’s law has brought parallelism, and we need to make stuff work on these computers.

Both are necessary and valuable to conduct, but the proper balance between the two is a necessity. The lack of tolerance for risk is one of the key factors contributing to this entire problem. Low-risk attitudes contribute to the dominance of focus on computing hardware and the appetite for the continued reign of Moore’s law. It also compounds and contributes to the dearth of focus on more fundamental and impactful algorithm research. We are buying massively parallel computers, and our codes need to run on them. Therefore the algorithms that comprise our codes need to work on these computers. QED.

The problem with this point of view is it’s absolute disconnect with the true value of computing. Computing’s true value comes from the ability to solve models of reality. We solve those models with algorithms (or methods). These algorithms are then represented in code for the computer to understand. Then we run them on a computer. The computer is the most distant thing from the value of computing (ironic, but true). The models are the most important thing, followed by how we solve the model using methods and algorithms.

Our current view and the national “exascale” initiative represents a horribly distorted and simplistic view of how scientific value is derived from computing, and as such makes for a poor investment strategy for the future. The computer, the thing the greatest distance from value, is the focus of the program. In fact the emphasis in the national program is focused at the opposite end of the spectrum from the value.

I only hope we get some better leadership before this simple-minded mentality savages our future.

Extraordinary benefits also accrue to the tiny majority with the guts to quit early and refocus their efforts on something new.

― Seth Godin

Conformity is the jailer of freedom and the enemy of growth.

― John F. Kennedy

In watching the ongoing discussions regarding the National Exascale initiative many observations can be made. I happen to think the program is woefully out of balance, and focused on the wrong side of the value proposition for computing. In a nutshell it is stuck in the past.

All the heroes of tomorrow are the heretics of today.

― E.Y. Harburg

The program is obsessively focused on hardware and the software most closely relatedto hardware. As the software gets closer to the application, the focus starts to drift. As the application gets closer and modeling is approached, the focus is non-existent. It is simply assumed that the modeling just needs a really huge computer and the waters will magically part and the path the promised land of predictive simulation will just appear. Science doesn’t work this way, or more correctly well functioning science doesn’t work like this. Science works with a push-pull relationship between theory, experiment and tools. Sometimes theory is pushing experiments to catch up. Sometimes tools are finding new things for theory to answer. Computing is such a tool, but it isn’t be allowed to push theory, or more properly theory should be changing to accommodate what the tools show us.

The opposite of courage in our society is not cowardice, it’s conformity.

― Rollo May

I’ve written a lot about all of these problems.

One of the other observations I haven’t written about is how antiquated this entire point of view is. The supercomputers are run in a manner consistent with the old fashioned “mainframes” that IBM used to produce. Mainframes have faded from prominence, but still exist. They are no longer the central part of computing, and this change has been good for everyone. The overly corporate and centralized computing model associated with mainframes is still in place. It is orthogonal to the nature of computing in most places. The decentralized computing associated with phones, and laptops, and tablets and the cloud all democratized computing. That democratization led the way for everyone using computing, and often not realizing they were. It was one of the keys to value and the explosion of information, data and computing. It is completely opposite of supercomputing.

 The conventional view serves to protect us from the painful job of thinking.

― John Kenneth Galbraith

The question is whether there is some way to learn from everyone else. How can this centralized supercomputing be broken down in a way to help the productivity of the scientist. One of the things that happened when mainframes went away was an explosion of productivity. The centralized computing is quite unproductive and constrained. Computing today is the opposite, unconstrained and completely productive. It is completely integrated into the very fabric of our lives. Work and play are integrated too. Everything happens all the time at the same time. Instead of maintaining the old-fashioned model we should be looking into harvesting the best of modern computing to overthrow the old model.

Mainframes represent the old way and conformity; freedom from them represents the new way and freedom. To succeed at supercomputing freedom is the path to success

Great people have one thing in common: they do not conform.

― P.K. Shaw

No amount of genius can overcome a preoccupation with detail

—Levy’s Eighth Law

I’ve been inundated with thinking about exascale computing this week. Programming models, code, computer languages, libraries, and massively parallel implementations of algorithms. At the end of all the talk about advanced computing, I’m left thinking that something really key is being ignored moving forward. We are already indrowning in data whether we are talking about the Internet in general, the coming “Internet of things” or the scientific use of computing. The future is going to be much worse and we are already overwhelmed. If we try to deal with every single detail, we are destined to fail.

How can we move forward and keep our sanity?

Of course reality is actually much simpler, or at least the part we care about. In almost every decision of any importance, the details fade away and we are left with only an important core of significance. This is a key concept moving forward in computing, sparsity. Not everything matters and the important thing is discovering how to unmask this kernel of essential information. If we can’t the data deluge will drown us.

Fortunately some concepts have emerged recently that hold promise. The whole area of compressed sensing is structured around the capacity to unveil the important signalin all the noise and represent this importance compactly and optimally. This class of ideas will be important in managing the Tsunami of data that awaits us.

The future will give us more data than we can ever wade through, and we need principled ways to manage our view of it. In many cases we won’t even be able to get the data off the computer at all, only a part of it. If our code or calculation crashes we won’t be able to restart from exactly the same state. We are going to have to let go of the details. This should be easier because the reality is that they don’t matter, or more properly the vast majority of the details don’t. The trick is holding on to the details that do matter.

Tradition becomes our security, and when the mind is secure it is in decay.

― Jiddu Krishnamurti

Over the past couple of posts I’ve opined that the essence of value in computing should be best found in the real world. This is true for scientific computing as it is for the broader world. The ability of computers to impact reality more completely has powered an incredible rise in the value of computing and transformed the World. Despite this seemingly obvious proposition, in recent years and with current plans, the scientific community has focused its efforts on the part of computing most distant from reality, the computing hardware. The bridge from the real world to the artificial reality of the simulation are our models of reality.

Tradition is a fragile thing in a culture built entirely on the memories of the elders.

― Alice Albinia

In science these models are often cast in the esoteric form of differential equations tobe solved by exotic methods and algorithms. Ultimately, these methods and algorithms must be expressed as computer code before the computers can be turned loose on their approximate solution. These models are relics. The whole enterprise of describing the real world through these models arose from the efforts of intellectual giants starting with Newton and continuing with Leibnitz, Euler, and a host of brilliant 17^th, 18^th and 19^th Century scientists. Eventually, if not almost immediately, models became virtually impossible to solve via available (analytical) methods except for a handful of special cases.

There is no creation without tradition; the ‘new’ is an inflection on a preceding form; novelty is always a variation on the past.

― Carlos Fuentes

When computing came into use in the middle of the 20^th Century some of these limitations could be lifted. As computing matured fewer and fewer limitations remained, and the models of the past 300 years became accessible to solution albeit through approximate means. The success has been stunning as the combination of intellectual labor on methods and algorithms along with computer code, and massive gains in hardware capability have transformed our view of these models. Along the way new phenomena have been recognized including dynamical systems or chaos opening doors to understanding the World. Despite the progress I believe we have much more to achieve.

What might be holding us back? The models are not evolving and advancing in reaction to the access to solution via computing.

 The difficulty lies not so much in developing new ideas as in escaping from old ones.

― John Maynard Keynes

Today we are largely holding to the models of reality developed prior to the advent of computing as a means of solution. The availability of solution has not yielded the balanced examination of the models themselves. These models are
artifacts of an age where the nature of solution was radically different. One might wonder what sorts of modifications of the existing paradigm would be in order should the means of solution be factored in. For example the notion of deterministic unique solutions to the governing equations is pervasive, yet reality clearly shows this to be wrong. Solutions to reality are always a little bit, to very different even given nearly identical initial conditions.

The assumption of an absolute determinism is the essential foundation of every scientific enquiry.

― Max Planck

Originally the models focused on the average or mean tendency of reality. This is reasonable for much of science and engineering, but as the point-of-view becomes refined other issues begin to crowd this out. These variations in outcome can dominate the utility of these models. For many cases the consequence of reality is driven by the uncommon or unusual outcomes (i.e., the tails of the distributiuon). Most of our current modeling approach and philosophy is utterly incapable of studying this problemeffectively. This gets to the core of studying uncertainty in physical systems. We need to overhaul our approach of reality to really come to grips with this. Computers, code and algorithms are probably at or beyond the point where this can be tackled.

It is impossible to trap modern physics into predicting anything with perfect determinism because it deals with probabilities from the outset.

― Arthur Stanley Eddington

Here is the problem. Despite the need for this sort of modeling, the efforts in computing are focused at the opposite end of the spectrum. Current funding and focus is aimed at the computing hardware, and code with little effort being applied to algorithms, methods and models. The entire enterprise needs a serious injection of intellectual energy in the proper side of the value proposition.

Cynics are – beneath it all – only idealists with awkwardly high standards.

― Alain de Botton

…Next time you’re faced with a choice, do the right thing. It hurts everyone less in the long run.

As the best practices in scientific computing continue to improve, verification is more frequently being seen in papers and reports. The progress over the past decade has been fantastic to see. Despite this progress there are some underlying problems that are pervasive in the community’s practice, and whose impact will ultimately reduce progress. These poorly executed practices are inhibiting the characterization of methods and their impact (positive or negative) on solutions.

Firstly, code verification is almost always applied to problems that bear little resemblance to the problems that are intended to be solved in the application of a method. Code verification usually only reports order of accuracy for the purposes of matching the theoretical expectations. This is meeting the minimal requirements of code verification as a practice. Often ignored is the capability to report the precise numerical error for the problem being computed. Both the rate of convergence and the error contain important and useful information for the developers and users of a numerical method. Both should be systematically reported rather than just the minimum requirement.

For solution verification the problem is much worse. Even when solution verification is done we are missing important details. The biggest problem is the lack of solution verification for the application of scientific computing to problems. Usually the problem is simply computed and graphs are overlaid, and success is declared. The comparison looks good enough. No sense of whether the solution is accurate is given at least quantitatively. An error estimate for the solution shown, or better yet a convergence study would provide much enhanced faith in the results. In addition to the numerical error, the rate of convergence would also provide information on the tangible expectations for the solution for practical problems. Today such expectations are largely left to be guessed by the reader.

In any moment of decision, the best thing you can do is the right thing. The worst thing you can do is nothing.

I believe on of the deeper issues is the belief that the rate of convergence and numerical error only matter for problems with analytical results. This matters for code verification purposes, but also matters greatly for practical problems. In fact it is probably more important for practical problems, yet it is rarely reported. To get things working better we need to move to a practice where both convergence and error are reported as a matter of course. It would be a great service to the community.

If you have built castles in the air, your work need not be lost; that is where they should be. Now put the foundations under them.

― Henry David Thoreau

In all endeavors we desire success, and the best success endures. The endurance of success is predicated on the foundations upon which that success is grounded. If foundations are systematically deprived of the basis, they will crumble and induce a crisis. Another way of saying this is success is dependent on balance. If the short-term success is continually rewarded, the long-term success will be undermined. These principles apply broadly including to the conduct of computational science and scientific computing.

To apply this principle it is important to understand the nature of the foundation, and how the inter-linking areas of focus come together to provide a broad base for success. I see “computing” as a general stream of activities running from an impact in the reality of people’s lives to the method of achieving this on a computer (models with methods and algorithms). These methods and algorithms need to be expressed to the computer in useful form through computer code, and ultimately have a computing platform adequate to the purpose. Every single step in the chain is important, but the relative value and priority of each is different. A lot depends on what the pacing requirements for progress are, but the focus of the value proposition should be an imperitive.

Insanity is doing the same thing, over and over again, but expecting different results

― Narcotics Anonymous

Let’s explore.

I made the argument that the thing that has set apart computers in recent times is the ability to make things matter to our daily lives, in and out of work. Computers can now have a huge impact on every aspect of living. When this happened the value of the entire computing enterprise exploded to a level unimaginable before. Every other aspect the model, algorithm, code and computer needed to be competently executed and adequate, but the connection to reality was the enabler for unprecedented growth.

Observing and understanding are two different things.

― Mary E. Pearson

The secondary fuel for this revolution is the model of interaction and the algorithms to efficiently deliver the value. The actual code and compute needs of this delivery needs to be competently executed, but beyond that offer nothing distinguishing to it. This is a massive lesson right in front of the scientific community, which seems to be not understood these observations as measured by its actions. Today’s computing for science emphasis has completely inverted the value stream revolutionizing computing in the rest of the World.

The computing hardware has taken center stage in scientific computing followed by computer code. The methods and algorithms have greatly diminished importance in charting the path forward. More troublingly the methods and algorithm work is typically focused upon the effective implementation on new exotic computing hardware, not establishing fundamentally new capabilities. It is important to get the most out of expensive computers, but we fail to harness the power of algorithms; the greatest power of algorithms is to transform what is possible to do with a model of reality. They can change what is even conceivable to solve, and open new vistas of fidelity to solution. A prime example is Google’s search, the value is putting the right information is people’s hand, the model is the connectivity of the Internet, and the PageRank algorithm makes it happen well enough. The code and computers putting it together are necessary, but not innovative.

But better to get hurt by the truth than comforted with a lie.
― Khaled Hosseini

The models of reality are important as the interface between reality and algorithms for solution. Without the model all the algorithms work is for naught. Without an algorithm all the beautiful code and powerful computers are useless. Without the model you don’t have a connection to reality. Thus the lack of focus on modeling in scientific computing is perhaps even worse.

Current work almost assumes that modeling available is adequate for the purposes. It is most assuredly not presently adequate, and it will almost as assuredly never be completely adequate. Modeling must always be improving. If we are doing our computing correctly the models we use should continually be coming up short. Instead, the models seem to be completely frozen in time. They aren’t advancing. For example, I believe we should be undoing the chains of determinism in simulation, but even today deterministic simulations are virtually all of the workload.

Instead of seeing a need for improvement of the underlying models, and the way these models are solved, we have a program that tries to solve the same models, with the same algorithms on massive computers only changing the fidelity of the discretization. This assumes that everything in this chain is already at its ultimate state. This implicit assumption should be rejected out of principle.

To acquire knowledge, one must study;
but to acquire wisdom, one must observe.

― Marilyn Vos Savant

These concepts should be almost self-evident, but in practice we continually trade long-term success for short-term gains. We have adopted practices that lower the short-term risk by raising the long-term risk. Ultimately the entire enterprise is lurching toward a crisis in sustainability. The key to this crisis is starving the foundation of value in scientific computing that is found foremost in models and their solution via algorithms and methods. The other aspect that has been systematically shortchanged is the value of the people who provide the ideas that form model, methods and algorithms. Ultimately, the innovation in scientific computing is the intellectual labor of talented individuals.

The scientific man does not aim at an immediate result. He does not expect that his advanced ideas will be readily taken up. His work is like that of the planter—for the future. His duty is to lay the foundation for those who are to come, and point the way.

― Nikola Tesla

A computer lets you make more mistakes faster than any other invention with the possible exceptions of handguns and Tequila.

― Mitch Ratcliffe

Computers have never been that important, but really its software that is important, but actually its algorithms that matter, which really isn’t that true either. It is the sum total of these things that matters to our daily lives. This is true for iPhone apps and modeling & simulation in the sciences. The real value in each of these things is how they connect to real things like grocery shopping or airflow over an airplane’s wing.

Therein lies the real truth, the value of computers is the code they run; the value in code are the algorithms they implement; the value in algorithms are the problem solving of the people devising them; the value of algorithms is found in the real World. All of this revolves around a single unassailable truth, this is a fundamentally human activity at every level expressed using tools that make rote calculations trivial, and power connections between people.

This truth is valid whether the human activity is the search on your phone or laptop, purchasing through Amazon, predicting tomorrow’s weather, solving the airflow over an aircraft wing, or the flow neutrinos in a supernova using the Boltzmann transport equation. The real revolution in computing is the ability of computing to matter to how we live our daily lives whatever the activity. Given that the value in all of this is the added capacity to achieve our goals, it might be worth considering whether our priorities actually reflect this. Where these values are present in computing the importance and value of computing has swelled. Given my personal focus on the scientific use of computing my assessment would be that we have lost our way. The values in computing programs are horribly distorted and out of balance. A key to this is the loss of perspective on what really matters.

Its useful to discuss what a code actually is. I’ll focus on a scientific code in particular. I will argue that the part of it that is considered the “code” is actually the least important aspect of the entire activity. The classically viewed code is a set of instructions that the computer can understand to take a sequence of steps. Usually this code is defined to solve a problem or better yet a class or type of problem. The code is a collection of ideas that the computer assists the solution of. A code is no better than the ideas it expresses and the skill of the programmer in making these instructions work. Despite this obvious aspect of computing the quality of the ideas in code has shrunk from importance. They are merely assumed to be important. Clever, crafty and innovative aspects of the problem solving define algorithms, methods and heuristics that make the computed solution better or faster or bot

Code itself is an algorithm. The machine actually understands a horribly opaque and obscure language that defines basic operations, and moves data around. The code is a way of expressing these basic ideas using human comprehensible language and abstractions that collect basic operations together into units. Fortran was the first of these languages, and it is considered one of the most important algorithms of the 20^th Century. It allowed the expression of more complex ideas to computers and greatly aided the advance of computing. Other languages have come into exisitence, but always with the same intent as Fortran originally had. The code is the key to unleasing the power of the computer.

 Let us remember that the automatic machine is the precise economic equivalent of slave labor. Any labor which competes with slave labor must accept the economic consequences of slave labor.

― Norbert Wiener

In scientific computing the key connection to reality are models. The most basic models are the governing equations such as the Euler, or Navier-Stokes or Boltzmann equations. These models are augmented by other models of subprocesses (often called subgrid models), and constitutive data that are typically experimentally measured and define the mean behavior of materials (accumulating the effects that would otherwise be statistical). These descriptions are the essential element in the value of computing to human activity. Their value transcends any of the other aspects: the algorithm, the code, and the computer itself. If the basic models are inadequate or faulty everything else is basically for naught. If the model is good, the rest of the components need to get it right, the algorithm or method needs to correctly or accurately solve the model, the implementation in code needs to be correct, and the computer needs to be capable of solving the problem. It is an exercise in balance and perspective. Our key issue
today is a lack of balance caused by a faulty perspective.

That’s the thing about people who think they hate computers. What they really hate is lousy programmers.

― Larry Niven

Another key element is the impact and connection of the code to human activity. The use of calculated results for science or engineering is one outcome. Another is the intersection of computing with the development and continuity of talent. In all cases the human intellect and talent is the core of the value stream, this aspect cannot be overlooked or the core of the value is lost.

The computer focuses ruthlessly on things that can be represented in numbers. In so doing, it seduces people into thinking that other aspects of knowledge are either unreal or unimportant. The computer treats reason as an instrument for achieving things, not for contemplating things. It narrows dramatically what we know and intended by reason.

― George Friedman

One of the things that seem intriguing is the appearance of the algorithm in the broader cultural milieu. Despite its inherently esoteric and abstract character, the algorithm is becoming a bit of a celebrity these days. Popular press articles have started to examine the impact of the algorithm on our daily lives and explore the power and dangers of relying upon them.

 When we change the way we communicate, we change society

― Clay Shirky

Why? What is happening?

Moore’s law is fading and approaching the end of its wonderful reign (for most of the computing world its already effectively pushing up daisies). Gone are the halcyon days when we could be assured of waiting a couple of years and purchasing a new computer that offered double or more of the performance of the old one. Given this extra power, the software on the older computer rapidly becomes equally or more obsolete. This is still happening today, but for different reasons; the software now has new ideas in it, new algorithms with new capabilities. From 1975 to 2005 Moore’s law produced a growth in computer power that fueled a rise in computing from a scientific backwater or corporate niche to the centerpiece in the World’s economy. Halfway through this great lurch forward, the Internet became the tie that bound all that power into a whole that was greater than any of its parts. Computing power around the world was connected together along with all the people whose numbers recently swelled to all of humanity through cell phones becoming magical handheld computers.

 As the massive gains from computer power wound down, and simultaneously the Internet transitioned into a huge web of human connectivity, the value proposition for computing changed. Suddenly the greatest value in all of this power switched to connection, access and sorting information. There were some fitful starts at attacking this key problem, but one solution rose above the rest, Google. Based on the work of a couple of Stanford graduate students and some really cool mathematics, Google took the world by storm. In a decade it had transformed itself into the World’s most powerful company. An algorithm that solved the data and connectivity access problem better than anything before it fundamentally powered Google.

Communications tools don’t get socially interesting until they get technologically boring.

― Clay Shirky

Google replaced a computer software company as the World’s most powerful company, Microsoft. In both cases computer programming was the engineering vehicle for these companies. Programming is a technique where intellectual labor is committed to a form where a computer can automatically execute a method, or algorithm to solve a problem. Usually the computer program is actually a large collection of methods,
algorithms, and heuristics that are uniquely composed together to solve problems. As these problems are more difficult and elaborate, the software gains more value.

The bottom line is that all of a sudden the algorithm and its software manifestation had eclipsed the computer hardware as a source of value. This transformation began when Microsoft rushed past IBM. IBM failed to see that software’s importance was about to eclipse hardware, and paid for it. Google put the algorithm together with the ability to give people access to information and connectivity to eclipse Microsoft. The algorithm had moved from being a topic of nerdish academic interest to one of the most powerful things in the World. The world’s economy spun on an axis determined by a handful of algorithms.

Change almost never fails because it’s too early. It almost always fails because it’s too late.

― Seth Godin

Meanwhile scientific computing has lost its mind and decided that the path that led IBM down the path towards disaster is its chosen path. The end of Moore’s law has resulted in a collective insanity of spending vast sums of money supporting the hardware path in the face of looming disaster. At the same time they have turned their backs on algorithms. Effort and focus flows into obtaining and building massive computers that are increasingly useless for real science while ignoring the value that algorithms bring. The infatuation with the biggest and fastest computer measured by an increasingly meaningless benchmark only grows with time. This continues while the key to progress stares them in the eye every time they do an Internet search, the power of the algorithm.

The easiest way to solve a problem is to deny it exists.

― Isaac Asimov

What the hell is going on?

Part of the problem is the ability to artificially breathe life into the corpse of Moore’s law through increasingly massively parallel computers. This has been done through moving the goalposts significantly. The LINPAC benchmark never had much to do with the core of scientific computing and this distance has only grown over the past three decades. This benchmark papers over the myriad of vexing issues with the new computers. What once was a gulf or irritating width has widened into a chasm of dangerous proportions. Disaster looms in the not too distant future as a result.

A secondary goalpost moving is the adoption of “weak scaling”. Scaling is the metric of how well an algorithm or code uses the parallel computing to solve problems faster. True (strong) scaling would ask how much faster could I solve a certain problem with more processors. Perfect scaling means that with “N” processors I would solve the problem “N” times faster. Weak scaling changes this reasonable measure by making the problem “N” times bigger at the same time as the number of processors grows. If the performance of a code is poor on a single processor, weak scaling will successfully hide this fact (most scientific codes in fact suck on single processors, and suck more on many processors). In fact, our codes are performing worse and worse on single processors, and little or nothing has been done about it, weak scaling carries some of the blame by hiding the problem.

Scientific computing is fundamentally about problem solving with computers, not computers unto themselves. The field is being perverted into a fetish where the focus is computers, and problem solving is secondary. This is where we come back to a necessary focus on algorithms. Algorithms are fundamentally about solving problems, and algorithm research is about better, faster, more efficient problem solving. Everything we do in scientific computing runs through an algorithm instantiated in software. Without the algorithms and software the computers are worthless. Without the model being solved and its connection to physics and engineering the value to society is questionable. The combination of algorithm and model is an expression of human intellect and problem solving. It needs a capable computer to allow the solution, but the essence is all-human. We have lost the context of the place of the computer as a tool; it should never be an end unto itself. Yet that is what it’s become.

Any sufficiently advanced technology is indistinguishable from magic.

― Arthur C. Clarke

At the end of the 20^th Century a list of the top algorithms was published (Dongarra, Jack, and Francis Sullivan. “Guest editors’ introduction: The top 10 algorithms.” Computing in Science & Engineering 2.1 (2000): 22-23.):

1. 1946: The Metropolis Algorithm for Monte Carlo.

2. 1947: Simplex Method for Linear Programming.
3. 1950: Krylov Subspace Iteration Method.
4. 1951: The Decompositional Approach to Matrix Computations.
5. 1957: The Fortran Optimizing Compiler.
6. 1959: QR Algorithm for Computing Eigenvalues.
7. 1962: Quicksort Algorithms for Sorting.
8. 1965: Fast Fourier Transform.
9. 1977: Integer Relation Detection.
10. 1987: Fast Multipole Method.

One can argue for a few differences (finite elements, shock capturing, multigrid, cyptography,…) in the list, but the bottom line is that scientific computing dominates the list. What about since the turn of the 21^st Century? The algorithmic heavy hitters are Google, Facebook, Netflix, encryption, iPhone apps, … If that top ten list were redone now, Google’s PageRank would almost certainly take one of the places. Scientific computing has shrunk from the algorithmic limelight, and commercial interests have leapt to the fore. The intellectual core of scientific computing has committed to utilizing these massive computers instead of solving problems better, or smarter. It is a truly tragic loss of leadership and immensely short sighted.

…invention is a somewhat erratic thing.

— J. Robert Oppenheimer

The key to progress is balance coupled with faith in the human intellect and its power to create. These creations are wonders, this includes computers, but they are machines that are merely tools. As tools they are only as good as what controls them, the algorithms and the software. I am convinced that breakthroughs are still possible. All that is needed is the focus and resources so that great minds will prevail. The modern world of computing offers vast opportunity for science that remains unexplored. Current leadership only seems to see the same path as we have taken in the past. It seems like a low risk path, but in fact represents the highest risk possible, the loss of potential. The lessons from commercial computing are there to be seen plain as day, algorithms rule. All we need to do is pay attention to what is sitting right in front of us.

We all die. The goal isn’t to live forever, the goal is to create something that will.

― Chuck Palahniuk

When we achieve a modicum of success professionally it usually stems from a large degree of expertise or achievement in a fairly narrow realm. At the same time this expertise or achievement has a price; it was gained through a great degree of focus, luck and specialization. Over time this causes a lack of perspective for the importance of your profession in the broader world. It is often difficult to understand why others can’t see the intrinsic value in what you’re doing. There is a good reason for this, you have probably lost the reason why what you do is valuable.

Ultimately, the value of an activity is measured in terms of its impact in the broader world. Often times these days economic activity is used to imply value fairly directly. This isn’t perfect by any means, but useful nonetheless. For some areas of necessary achievement this can be a jarring realization, but a vital one. Many monumental achievements actually have distinctly little value in reality, or the value comes far after the discovery. In many cases the discoverer lacks the perspective or skill to translate the work into practical value. Some of these are necessary to achieve things of greater value. Achieving the necessary balance in these cases is quite difficult, and rarely, if ever achieved.

It’s always important to keep the most important things in mind, and along with quality, the value of the work is always a top priority. In thinking about computing, the place where the computers change how reality is engaged is where value resides. Computer’s original uses were confined to business, science and engineering. Historically, computers were mostly the purview of the business operations such as accounting, payroll and personnel management. They were important, but not very important. People could easily go through life without ever encountering a computer and their impact was indirect.

As computing was democratized via the personal computer, the decentralization of access to computer power allowed it to grow to an unprecedented scale, but an even greater transformation laid ahead. Even this change made an enormous impact because people almost invariably had direct contact with computers. The functions that were once centralized were at the fingertips of the masses. At the same time the scope of computer’s impact on people’s lives began to grow. More and more of people’s daily activities were being modified by what computing did. This coincided with the reign of Moore’s law and its massive growth in the power and/or the decrease in the cost of computing capability. Now computing has become the most dominant force in the World’s economy.

Why? It wasn’t Moore’s law although it helped. The reason was simply that computing began to matter to everyone in a deep, visceral way.

Nothing is more damaging to a new truth than an old error.

— Johann Wolfgang von Goethe

The combination of the Internet with telecommunications and super-portable personal computers allowed computing to obtain massive value in people’s lives. The combination of ubiquity and applicability to the day-to-day life made computing’s valuable. The value came from defining a set of applications that impact people’s lives directly and always within arm’s reach. Once these computers became the principle vehicle of communication and the way to get directions, find a place to eat, catch up with old friends, and answer almost any question at will, the money started flow. The key to the explosion of value wasn’t the way the applications were written, or coded or run on computers, it was their impact on our lives. The way the applications work, their implementation in computer code, or the computers themselves just needed to be adequate. Their characteristics had very little to do with the success.

It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with experiment, it’s wrong.

― Richard P. Feynman

Scientific computing is no different; the true value lies in its impact on reality. How can it impact our lives, the products we have or the decisions we make. The impact of climate modeling is found in its influence on policy, politics and various economic factors. Computational fluid dynamics can impact a wide range of products through better engineering. Other computer simulation and modeling disciplines can impact the military choices, or provide decision makers with ideas about consequences for actions. In every case the ability of these things to influence reality is predicated on a model of reality. If the model is flawed, the advice is flawed. If the model is good, the advice is good. No amount of algorithmic efficiency, software professionalism or raw computer power can save a bad model from itself. When a model is good the solution algorithms and methods found in computer code, and running on computers enable its outcomes. Each of these activities needs to be competently and professionally executed. Each of these activities adds value, but without the path to reality and utility its value is at risk. 

Despite this bulletproof assertion about the core of value in scientific computing, the amount of effort focusing on improving modeling is scant. Our current scientific computing program is predicated on the proposition that the modeling is good enough already. It is not. If the scientific process were working, our models would be improving from feedback. Instead they are stagnant and the entire enterprise is focused almost exclusively on computer hardware. The false proposition is that the computers simply need to get faster and the reality will yield to modeling and simulation.

So we have a national program that is focused on the least valuable thing in the process, and ignores the most valuable piece. What is the likely outcome? Failure, or worse than that abject failure. The most stunning thing about the entire program is the focus is absolutely orthogonal to the value of the activities. Software is the next largest focus after hardware. Methods and algorithms are the next highest focus. If one breaks out this area of work into its two pieces, the new-breakthroughs or the computational implementation work, the trend continues. The less valuable implementation work has the lion’s share of the focus, while the groundbreaking type of algorithmic work is virtually absent. Finally, modeling is nearly a complete absentee. No wonder the application case for exascale computing is so pathetically lacking!

 It is sometimes an appropriate response to reality to go insane.

― Philip K. Dick

Alas, we are going down this road whether it is a good idea or not. Ultimately this is a complete failure of the scientific leadership of our nation. No one has taken the time or effort to think this shit through. As a result the program will not be worth a shit. You’ve been warned.

The difference between genius and stupidity is; genius has its limits.

― Alexandre Dumas-fils

One of the things that Winter holiday means to me is movies, and good ones at that. It is something my wife and I love to do, enjoy and argue about. My son noted that we
are different than other families in that we see good movies and his friends see crap all the time. To sort of normalize things I’ll say that I really enjoy movies with an edge generally favoring movies with a bit of noir in their soul. A recent example of such would be “Drive” from 2011, going a bit further back some of my favorite films are “Pulp Fiction” “Fight Club” “American History X” and ”Full Metal Jacket”. I also love some of the more majestic movies of days gone by such as “Lawrence of Arabia” and “2001: A Space Odyssey” in particular.

Here are my holiday movie observations for the current season. I’ll assign each a letter grade with an Academy award winning film usually getting an “A”. I’d give all of the above-mentioned movies this grade and a few “A+”.

I’ll note that some of the quotes below contain some very bad language, which you would hear in the theatre. Be forewarned.

Whiplash, A

An absolute stunner of a movie with one of the best acting performances I can remember seeing in a long time by JK Simmons as Terence Fletcher. It is a student-teacher story set in a conservatory. The kid is a young talented jazz drummer (played with skill by Miles Teller) looking to catch the eye of the famous teacher. He does and then the fireworks start. The filming and acting produces the sort of tension that usually come from action flicks. This is literally edge-of-your-seat stuff,
the tension arising from the interplay between the teacher and student is incredible.

Terence Fletcher: There are no two words in the English language more harmful than good job.

Selma, B+

I really wanted to like this better. It was a finely acted and crafted historical drama based on a key moment in the civil rights movement. It is stunning to see the kind of things that used to happen in the United States. We’ve made progress as a country, but shockingly little as the events of the last year show. There is action in the deeply racist Alabama of 1964 and 1965, and tension between MLK and LBJ. Other figures like J. Edgar Hoover and George Wallace come across like the villainous humans they were. Overall an important movie that was competently executed, but not the brilliant movie I had hoped it would be.

“People out there actually say they’re gonna kill our children, they’re trying to get into your head.” – Coretta Scott King

Imitation Game, A-

This is a good movie, the worst of the ones I gave an “A-“ to. It is a very Hollywood version of Alan Turing’s life. Benedict Cumberbatch takes the material and produces a wonderful performance. The storytelling is unique running three timelines in parallel from Turing’s life with great lessons relevant to today’s problems. The upshot is that Turing’s life was immensely tragic, and his service to England and the World was never paid what it was due.

Joan Clarke: Sometimes it is the people who no one imagines anything of who do the things that no one can imagine.

Hobbit, C

I would sum this movie up as being thoroughly disappointing. I am guessing that the problem is that no one can tell Peter Jackson “no” any more and he is reverting to his roots. Some of the film making decisions are simply ludicrious and remind me strong of Jackson’s earlier films like “Dead Alive”. The choices almost always comical and some one should have told him, “this is a bad idea”.

Bilbo Baggins: One day I’ll remember. Remember everything that happened: the good, the bad, those who survived… and those that did not.

Wild, A-

I didn’t actually see it, my wife and daughter did while my son and I saw the Hobbit. She said it was great (she got the good end the deal to be sure).

Cheryl: What if I forgave myself? I thought. What if I forgave myself even though I’d done something I shouldn’t have?

Boyhood, A

Along with Whiplash this is my choice for the best picture of the year. The movie has a massive gimmick being filmed a bit over time a week or two a year for 12 years. It chronicles the childhood of a boy whose parent divorce and how he develops from a small boy entering school to an adult entering college. The gimmick the film uses is remarkable using the same actors to show the passage of time. The film is wonderful beyond the gimmick and delivers a wonderful tale of personal growth for all the characters. It is both simple and immensely rich.

Nicole: You know how everyone’s always saying seize the moment? I don’t know, I’m kind of thinking it’s the other way around, you know, like the moment seizes us.

Snowpiercer, A-

This movie is a wonderful bit of pay-for-view surprise and quite enjoyable on the whole. Some aspects of the movie are odd, but it is filled with great performances including surprising depth from Chris Evans. He is a much better actor than people realize. The movie has action, tension and deep commentary on our modern world and its problems. The film requires a degree of suspecnsion of disbelief regarding the basic premise, but if you can manage that it is a real gem.

Curtis: You know what I hate about myself? I know what people taste like. I know babies taste the best.

Nightcrawler, A-

This was a marvelously dark movie and portrait of a true sociopath. Jack Gylennhaal is wonderfully creapy in the roll and manages to make himself genuinely unlikable. He is driven and relentless in achieving fame and success without a hint of morality. At the same time the film succeeds in providing a tremendously insightful commentary on our modern society and our appetite for news that titillates much more than informs.

Lou Bloom: That’s my job, that’s what I do, I’d like to think if you’re seeing me you’re having the worst day of your life.

Intersteller, B

This is a film that divides opinions for good reasons. It is a wonderfully majestic movie that is horribly flawed. Good, but not great performances can be found working on a script that was uninspired. The concept and arc of time with an innovative narrative concept make the story watchable. In the end it produces a watchable film that won’t be remembered 10 years from now.

I never ask for the science parts of movies to be too realistic simply not too cartoonish. It succeeds in being realistic enough not to offend. I tend to believe that we don’t understand the universe well enough to know what isn’t possible. It offers a Hollywood friendly version of relatively forward looking science. Having Habitable planets in the vicinity of a black hole is not the most realistic thing. I would have been much happier with a wormhole alone leaving the black hole out.

Cooper: We used to look up at the sky and wonder at our place in the stars, now we just look down and worry about our place in the dirt.

Cooper: We’ve always defined ourselves by the ability to overcome the impossible. And we count these moments. These moments when we dare to aim higher, to break barriers, to reach for the stars, to make the unknown known.

The Interview, C-

This film is the controversy of the season with the hacking of Sony and the capitalization to terrorism initially declining to release the movie then coming to their senses. We saw it on pay for view. The hackers did a better job sponsoring the movie than it deserved. This was easily the worst movie we saw all season. It was amusing and thoughtlessly entertaining, but a cinematic turd. It was a couple hours of my life I can’t get back.

Dave Skylark: [admires a war tank] Holy fuckamole. Is that real?

Kim Jong-un: It was a gift to my grandfather from Stalin

Dave Skylark: In my country it’s pronounced Stallone.

Kim Jong-un: You’re so funny, Dave.

Gone Girl, A-

This was an absolute gem of a movie. We loved it. It was thrilling, dark and relentless. It was a wonderfully grim look into a romance and marriage with an edge. So much is happening out of view and hidden from the viewer adding to the tension in theatre. Rosamund Pike is simply wonderful and creates a character of great depth who ultimately generates a deep emotional response. It is never clear who is the biggest villain. It is one movie where no one is a hero.

Nick Dunne: You fucking cunt!

Amy Dunne: I’m the cunt you married. The only time you liked yourself was when you were trying to be someone this cunt might like. I’m not a quitter, I’m that cunt. I killed for you; who else can say that? You think you’d be happy with a nice Midwestern girl? No way, baby! I’m it.

Nick Dunne: Fuck. You’re delusional. I mean, you’re insane, why would you even want this? Yes, I loved you and then all we did was resent each other, try to control each other. We caused each other pain.

Amy Dunne: That’s marriage.

Here are the movies that I haven’t seen yet, but want to see (A Most Violent Year, Birdman, Theory of everything, American Sniper, CitizenFour, Foxcatcher).