Today, now, and what of the future?

From Aldo Leopold in his A Sand County Almanac:

One of the penalties of an ecological education is that one lives alone in a world of wounds. Much of the damage inflicted on land is quite invisible to laymen. An ecologist must either harden his shell and make believe that the consequences of science are none of his business, or he must be the doctor who sees the marks of death in a community that believes itself well and does not want to be told otherwise.

(Emphasis added.)

Among many other notable efforts, Aldo Leopold attempted to reconcile Ecology with economic imperatives. I still, however, cannot stomach the encouragement of hunting which Leopold’s post mortem Foundation pursues.

Posted in adaptation, agroecology, Aldo Leopold, American Association for the Advancement of Science, argoecology, being carbon dioxide, biology, Boston Ethical Society, Botany, Buckminster Fuller, Charles Darwin, climate, climate change, David Suzuki, Earle Wilson, Ecological Society of America, Ecology Action, ethics, George Sughihara, Glen Peters, global warming, Grant Foster, Humans have a lot to answer for, Hyper Anthropocene, population biology, quantitative biology, quantitative ecology, Spaceship Earth, sustainability, The Demon Haunted World, the right to be and act stupid, the right to know, the tragedy of our present civilization, tragedy of the horizon, unreason, UU Humanists | 2 Comments

Fast means, fast moments (originally devised 1984)

(Updated 4th December 2018.)

There are many devices available for making numerical calculations fast. Modern datasets and computational problems apply stylized architectures, and use approaches to problems including special algorithms for just calculating dominant eigenvectors or using non-classical statistical mechanisms like shrinkage to estimate correlation matrices well. But sometimes it’s simpler than that.

In particular, some calculations can be preconditioned. What’s done varies with the problem, but often there is a one-time investment in creating a numerical data structure which is then used repeatedly to obtain fast performance of some important kind.

This post is a re-presentation of a technique I devised in 1984 for finding sums of contiguous submatrices of a given matrix in constant time. The same technique can be used to calculate moments of such submatrices, in order to support estimating statistical moments or moments of inertia, but I won’t address those in this post. Studies of these problems, given this technique, can imagine how that might be done. If there’s an interest and a need, with comments or email, I’ll some day post algorithm and code for doing moments.

The Algorithm

Assume there is a square matrix, \mathbf{A}, having flonums. Accordingly it has both m rows and m columns. Produce from it a transformed matrix \mathbf{A_{1,2}} in two steps, first producing a transformed matrix, \mathbf{A_{1}}, and then the transformed matrix, \mathbf{A_{1,2}}. Do this by first forming \mathbf{A_{1}} as \mathbf{A} but with its j^{\text{th}} column replaced by the cumulative sum of the j^{\text{th}} column of \mathbf{A}. That is,

\mathbf{A_{1}}_{i,j} = \sum_{k=1}^{i} \mathbf{A}_{k,j}

To form \mathbf{A_{1,2}} do something similar, except with rows:

\mathbf{A_{1,2}}_{i,j} = \sum_{k=1}^{j} \mathbf{A_{1}}_{i,k}

\mathbf{A_{1,2}} is then the preconditioned version of {A}. Specifically, if the sum of any contiguous rectangular submatrix of \mathbf{A} is sought, say, with its upper left coordinate within \mathbf{A} being (i,j) and its lower right being (k,l), then lookup the value of \mathbf{A_{1,2}}_{k,l} and subtract from it the value of \mathbf{A_{1,2}}_{i-1,l} and the value of \mathbf{A_{1,2}}_{k,j-1}, and then add back in, because of the principle of inclusion and exclusion, the value of \mathbf{A_{1,2}}_{i-1,j-1}. If any of these are off the premises of \mathbf{A_{1,2}} because i-1 or j-1 are too small, treat the corresponding subtrahends as zero.

Because there are only two sums involved in the calculation of the sum of any submatrix, the algorithm does this in constant time, irrespective of the sizes of the submatrices. There are dynamic range tricks that can be applied should the sums of values in the preconditioned matrix get very large in magnitude.

The code is available in the Google Drive folder for such things.

# This section of code up to the next group of #### marks is owned and copyrighted by #
# Jan Galkowski, who has placed it in the public domain, available for any purpose by anyone. #

fastMeansFastMomentsPrecondition<- function(X)
# # (From Jan Galkowski, ``Fast means, fast moments'', 1984,
# # IBM Federal Systems Division, Owego, NY. Released into the public domain, 1994.)
stopifnot( is.matrix(X) )
M<- nrow(X)
stopifnot( M == ncol(X) )
AX1<- apply(X=X, MARGIN=2, FUN=cumsum)
# AX2<- t(apply(X=X, MARGIN=1, FUN=cumsum))
AX12<- t(apply(X=AX1, MARGIN=1, FUN=cumsum))

fastMeansFastMomentsBlock<- function(P, iUL, jUL, iLR, jLR)
# # (From Jan Galkowski, ``Fast means, fast moments'', 1984,
# # IBM Federal Systems Division, Owego, NY. Released into the public domain, 1994.)
# P is the preconditioned AX12 from above.
stopifnot( is.matrix(P) )
M<- nrow(P)
stopifnot( M == ncol(P) )
stopifnot( (1 <= iUL) && (iUL <= M) )
stopifnot( (1 <= jUL) && (jUL <= M) )
stopifnot( (1 <= iLR) && (iLR <= M) )
stopifnot( (1 <= jLR) && (jLR <= M) )
iUL1<- iUL-1
jUL1<- jUL-1
iLR1<- iLR-1
jLR1<- jLR-1
if (0 == iUL1)
W.AL<- 0
W.A<- 0
if (0 == jUL1)
W.L<- 0
} else
W.L<- P[iLR,jUL1]
} else if (0 == jUL1)
W.AL<- 0
W.L<- 0
if (0 == iUL1)
W.A<- 0
} else
W.A<- P[iUL1,jLR]
} else
W.AL<- P[iUL1,jUL1]
W.A<- P[iUL1,jLR]
W.L<- P[iLR,jUL1]
W<- P[iLR,jLR] + W.AL - W.A - W.L

# Self-test FMFM ...
cat("Fast means, fast moments self-test ...\n")
Z<- matrix(round(runif(100, min=1, max=100)), 10, 10)
Z.P<- fastMeansFastMomentsPrecondition(Z)
stopifnot( sum(Z[1:4,1:5]) == fastMeansFastMomentsBlock(Z.P, 1, 1, 4, 5) )
stopifnot( sum(Z[8:10, 8:9]) == fastMeansFastMomentsBlock(Z.P, 8, 8, 10, 9) )
stopifnot( sum(Z[4:7, 3:5]) == fastMeansFastMomentsBlock(Z.P, 4, 3, 7, 5) )
rm(list=c("Z", "Z.P"))
cat("... Self-test completed.\n")

# End of public domain code. #

randomizeSeed<- function()
# Futz with the random seed
E<- proc.time()["elapsed"]
names(E)<- NULL
rf<- E - trunc(E)
# rm(list=c("E", "rf"))
return(,, size=1), replace=TRUE)[1] )

wonkyRandom<- randomizeSeed()

So, there’s a little story of how this came to be public domain.

I used to work for IBM Federal Systems Division, in Owego, NY. I worked as both a software engineer and later, in a much more fun role, as a test engineer specializing in hardware-software which did quantitative calculations. IBM, as is common, had a strong intellectual property protection policy and framework.

Well, in 1994, Loral bought Federal Systems from IBM. As former employees, we were encouraged to join the new organization, but, naturally, were asked to sign a bunch of paperwork. To my personal surprise, there was nothing in the paperwork which had to do with IBM’s rights to intellectual property we might have originated or held. All they wanted was for us to sign onto the new division, part of Loral.

Before I signed, therefore, I approached corporate counsel and pointed out there was no restriction on intellectually property or constraints upon its disposition. They, interested in making the transition happen, said “Yes”. I went off and prepared a document of all the material and algorithms and such which I thought I had developed while on IBM time in which they hadn’t expressed any interest, including offers to put it up for patenting or whatever. It was a reasonably thick document, maybe 100 pages, and I still have a copy. I asked the attorneys to sign over the intellectual property rights of these to me, and they did. It was witnessed. I had made my coming to Loral contingent upon doing this, and they seemed happy to do it. I signed once I had this in hand. I still have a copy.

I did not know at the time, but Loral’s interest was in purchasing Federal Systems, and they had every intention of “flipping it”, as one might a house, to another buyer in a short time, and they apparently didn’t care about this.

But, as a result, this algorithm, for fast means and fast moments, which I had developed in 1984 while doing some image processing work, became mine. And I have always treated it as public domain, available to anyone for any purpose. And, here, with this post, I put it out there for your public use, for any purpose whatsoever, without constraints. Very open source. Commercial or otherwise.


It would be nice to credit me, but you don’t have to do that.

Posted in image processing, mathematics, numerical algorithms, numerical software, numerics | 2 Comments

Buckminster Fuller: Spaceship Earth. This is the Future. And it will be, here.

Posted in an ignorant American public, an uncaring American public, Buckminster Fuller, Global Carbon Project, green tech, science, Spaceship Earth, Star Trek - The Next Generation, Techno Utopias, technology, the energy of the people, the green century, the value of financial assets, tragedy of the horizon, UU Humanists | 1 Comment

Heat has no hair (from Eli Rabett)

See Eli’s post.


We can summarize the data in the figure above adding that ~40 W/m2 go directly from the surface to space as IR radiation of the 398 W/m2 leaving the surface. In and out in the table … [AT THE POST] … means into and out the surface the atmosphere and space respectively. In is taken as a positive addition to the heat content and negative a decrease …

… The important point is to realize that surface IR radiation absorbed in the atmosphere is rapidly (10 μs) thermalized and converted into random motion of the molecules in the atmosphere, just as is latent heat from condensation of water vapor and from sensible heat. Very little, less than a part per million, is directly radiated back to the surface and we can neglect that.

The 342 W/m2 of back radiation is OBSERVED, so this ain’t a model or a theory, where does it come from? It comes from ALL of the sources pushing heat into the atmosphere, from the convective and radiative heat transfer from the surface.

(Emphasis in the original.)

See the original for more detail.

Perhaps it’s just my physics training, but I never understood the primacy some (even scientists) put on the primacy of convection in terms of climate. I mean, sure, a Bunch of energy can come into the convective-dominated part of the climate system, and it might reside there for a duration, perhaps even long, but, really, that doesn’t matter. Eli’s point is that if a different bunch of the same amount doesn’t leave the climate system, it’ll warm. And it doesn’t matter how long the first bunch is in the climate system, or what path it takes through it, or anything of the kind.

So, to me, this idea that the various oscillations, like NAO or PDO or ENSO somehow have something substantial to do with the overall climate and problem is specious. Yeah, there are big energy flows from one part of the system to the other, just as there are big flows of Carbon to and from oceans to atmosphere, but that’s just slosh, and the point is the net balance. And human emissions of (about) 10 GtC per annum are affecting that a lot.

Posted in American Association for the Advancement of Science, American Meteorological Association, an ignorant American public, an uncaring American public, atmosphere, Blackbody radiation, carbon dioxide, chemistry, climate change, climate education, Eli Rabett, global warming, physics, science | Leave a comment

Blackbody radiation and the greenhouse effect, via plates (from Eli Rabett)

See Eli’s post.


Eli can keep on adding plates, Ms. Rabett has gone out to buy some extras. Here is the red plate special. If somebunny works it through they will find that b’=3/4 a, go another plate and, as Christian pointed out, now b’ has increased to 4/5 a and so on.

Eli has not said anything about how the heat is being transferred, radiation, convection or conduction but since heat transfer, no matter the mechanism, is always proportional to temperature, the temperature of the blue plate must increase as more plates are added.

See the original for more detail.

Posted in Blackbody radiation, carbon dioxide, climate change, Eli Rabett, energy, energy flux, global warming, lapse rate, physics, science | Leave a comment

The Democrats have no plan to address Climate Change (either)

Recall an article from the 15th November 2017 issue of The Atlantic:

… [T]he Democratic Party does not have a plan to address climate change. This is true at almost every level of the policy-making process: It does not have a consensus bill on the issue waiting in the wings; it does not have a shared vision for what that bill could look like; and it does not have a guiding slogan—like “Medicare for all”—to express how it wants to stop global warming.

Many people in the party know that they want to do something about climate change, but there’s no agreement about what that something may be.

This is not for lack of trying. Democrats have struggled to formulate a post-Obama climate policy because substantive political obstacles stand in their way. They have not yet identified a mechanism that will make a dent in Earth’s costly, irreversible warming while uniting the many factions of their coalition. These problems could keep the party scrambling to face the climate crisis for years to come.

This remains true. The only Democrats in the national view who keep mitigation of climate change in focus are Senator Bernie Sanders and Senator Sheldon Whitehouse. In fact, Senator Sanders and Senator Whitehouse are the only ones with plans, this being Senator Sanders’, and this being Senator Whitehouse, quite contrary to the impression The Atlantic article gives. Also, the claim that “Unlike Clinton’s policies, Sanders would surely have required a Democratic Congress to enshrine his policies”, is completely disingenuous. Only the most limited policies can be enacted without Congress, but that never should be a reason for failing to champion them or make excuses for why they can’t be done, like President Obama’s insistence that we cannot sacrifice economic growth in the pursuit of climate mitigation.

Others have observed the same.

So, I would suggest that what The Atlantic and I mean here is that the standard, vanilla-flavored Democratic Party has no idea about what to do, and it doesn’t really care. What it cares about is winning, and it will compromise on policy in order to make that happen.

This is predominantly why Claire and I are so supportive of Bob Massie as Democratic candidate for governor of Massachusetts. See his position on climate change.

It’s more tiring to say it again than it is to listen to it, but we are running out of time and the economic costs to do something real in time to stop awesome, amazing, and recurring harm from climate change increase by the month.

We determine the point of no return (PNR) for climate change, which is the latest year to take action to reduce greenhouse gases to stay, with a certain probability, within thresholds set by the Paris Agreement. For a 67% probability and a 2K (Kelvin) threshold, the PNR is the year 2035 when the share of renewable energy rises by 2% per year. We show the impact on the PNR of the speed by which emissions are cut, the risk tolerance, climate uncertainties and the potential for negative emissions.

In short, both political parties — and especially the Democrats, since they claim to know better — are failing the United States Constitution and the people of the United States:

Preamble. We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.

(Emphasis added.)

Amendment XIV (Ratified July 9, 1868)

Section 1.
All persons born or naturalized in the United States, and subject to the jurisdiction thereof, are citizens of the United States and of the State wherein they reside. No State shall make or enforce any law which shall abridge the privileges or immunities of citizens of the United States; nor shall any State deprive any person of life, liberty, or property, without due process of law; nor deny to any person within its jurisdiction the equal protection of the laws.

(Emphasis added.)

Indeed, given this situation, as I’ve mentioned before, I really wonder if the Constitution of the United States is up to this challenge, because it lacks the mechanism to achieve this. Of course, given that Congresses and Presidents disregard the Constitution, notably

Article. VI.

… This Constitution, and the Laws of the United States which shall be made in Pursuance thereof; and all Treaties made, or which shall be made, under the Authority of the United States, shall be the supreme Law of the Land; and the Judges in every State shall be bound thereby, any Thing in the Constitution or Laws of any State to the Contrary notwithstanding.

(Emphasis added.)

with respect to, say, United Nations Framework Convention on Climate Change”, which remains one of the `treaties in force’ considered so by the U.S. State Department (page 515).

This is why Juliana v United States is so essential. See the compact details.

Posted in American Meteorological Association, an ignorant American public, an uncaring American public, Anthropocene, anti-intellectualism, anti-science, being carbon dioxide, bridge to nowhere, carbon dioxide, Carbon Tax, children as political casualties, climate change, climate disruption, climate economics, consumption, corporate responsibility, Cult of Carbon, destructive economic development, environment, environmental law, games of chance, global blinding, global warming, greenhouse gases, Hyper Anthropocene, Juliana v United States | Tagged | 6 Comments

“Why we need Jean-Luc Picard in 2018”

Admiral Picard is returning.

See the story, by Daniel W Drezner. On CBS All Access.

Yes, “Make it so.”

Posted in American Association for the Advancement of Science, Bloomberg New Energy Finance, Buckminster Fuller, humanism, Jean-Luc Picard, Mathematics and Climate Research Network, open source scientific software, Our Children's Trust, Patrick Stewart, Principles of Planetary Climate, reason, reasonableness, science, Spaceship Earth, Star Trek, Star Trek - The Next Generation, STNG, The Demon Haunted World, the Final Frontier, tragedy of the horizon | Leave a comment

What will happen to fossil fuel-fired electric bills everywhere, eventually, including those fired by natural gas

See Cost of Coal: Electric Bills Skyrocket in Appalachia as Region’s Economy Collapses, by James Bruggers at Inside Climate News. Excerpt:

The common denominator is American Electric Power, one of the nation’s largest utilities. It owns Kentucky Power, along with two subsidiaries in neighboring West Virginia, Wheeling Power and Appalachian Power.

In May, Wheeling Power and Appalachian Power requested permission from the Public Service Commission of West Virginia to boost their monthly residential bill 11 percent because of declining sales. That was on top of a 29 percent increase between 2014 and 2018.

Customers in both states are furious that the regulators are going along.

“Our jobs available in this state are not a living wage, and many are working two, three jobs just to make it,” wrote Elizabeth Bland of Beckley, West Virginia, in her protest letter to the commission. “Please turn down this request from Appalachian Power for the sake of all West Virginians.”

Rising rates are just part of the problem.

Kentucky Power’s monthly bill also includes surcharges, and a line for each customer’s share of the utility’s fixed costs. These add up in precious dollars.

`They’re doubling down on coal at a time when coal is not competitive,’ said James M. Van Nostrand, a professor at the West Virginia University College of Law with decades of experience in the energy field. `It’s really tragic.’

The average bill per customer at Kentucky Power has been among the highest in the nation for an investor-owned utility, according to 2016 numbers from the U.S. Energy Information Agency, the most recent comparisons available.

`We’re hit hard,’ Alice Craft, a Whitesburg-area resident, told InsideClimate News. `The power companies, they are just greedy, greedy, greedy.’

This will inevitably happen to all regions depending primarily upon fossil-fuel fired electricity, including Massachusetts, with consequences for the public, for utility shareholders, for local real estate property values, and for local business expansion. Accordingly, the actions of the Massachusetts House on recent energy legislation is incredibly myopic to say the least, and does not support the stated goals of House leadership, especially those of Democratic House Speaker Robert DeLeo to `look out for the little guy’. His actions say he’s looking out for utility and energy companies, and the interests of AIM, whatever he says his motivations are.

But, it’s only a matter of time.

Posted in adaptation, American Solar Energy Society, Amory Lovins, an uncaring American public, Bloomberg New Energy Finance, BNEF, bridge to nowhere, bridge to somewhere, Carbon Worshipers, clean disruption, corporate responsibility, Cult of Carbon, decentralized electric power generation, decentralized energy, destructive economic development, electricity, electricity markets, energy utilities, engineering, exponential growth, fossil fuels, grid defection, Hermann Scheer, ILSR, investment in wind and solar energy, ISO-NE, John Farrell, Joseph Schumpeter, local generation, local self reliance, marginal energy sources, Massachusetts Clean Energy Center, natural gas, pipelines, public utility commissions, PUCs, rate of return regulation, rationality, reason, reasonableness, rights of the inhabitants of the Commonwealth, solar democracy, solar domination, solar energy, solar power, Sonnen community, Spaceship Earth, stranded assets, sustainability, the energy of the people, the green century, the right to be and act stupid, the right to know, the tragedy of our present civilization, the value of financial assets, tragedy of the horizon, unreason, utility company death spiral, wind energy, wind power, zero carbon | Leave a comment

Censorship of Science by the administration of President Donald Trump

See work by the Columbia Sabin Center for Climate Change Law.

… President Trump has directed EPA and DOI to reconsider regulations adopted to control greenhouse gas emissions, despite the wealth of data showing that those emissions are the key cause of climate change. Faced with this contradiction, both agencies have sought to downplay the science, including by restricting the availability of information (for example, by removing climate data from websites and deleting references to humans’ role in climate change from reports). Similar action has also been taken by a raft of other entities, with the SST indicating that at least 20 different federal bodies, including both Congress and the White House, have attempted to restrict access to scientific information or otherwise silence science …

See also reporting here on work anticipating this kind of action.

Posted in American Association for the Advancement of Science, an ignorant American public, an uncaring American public, anti-intellectualism, anti-science, Azimuth Backup Project, citizen data, Climate Science Legal Defense Fund, Donald Trump, dump Trump, Ecological Society of America, environmental law, epidemiology, global blinding, Neill deGrasse Tyson, open data, rationality, reason, reasonableness, science, secularism, The Demon Haunted World, the right to be and act stupid, the right to know, the tragedy of our present civilization, tragedy of the horizon, unreason | Leave a comment

“All of Monsanto’s problems just landed on Bayer” (by Chris Hughes at Bloomberg)

See Chris Hughes’ article.

Monsanto has touted Roundup (also known as Glyphosate but more properly as \textbf{\texttt{N-(phosphonomethyl)glycine}}) as a safe remedy for weed control, often in the taming of so-called “invasive species”. It’s used on playfields where children are exposed to it, including, apparently, in my home town of Westwood, Massachusetts.

There are more than 450 court cases in progress alleging harm from the product, and a jury in one, DEWAYNE JOHNSON VS. MONSANTO COMPANY ET AL (Case Number: CGC16550128), has found Monsanto, et al guilty with a US$289 million award. It’s long been known to affect fish and amphibians, and recently physicians have gotten concerned, particularly in its connection with cancer in humans.

Image by Benjah-bmm27Own work, Public Domain, Link

This has repercussions for Bayer, as Hughes explains.

But it is perhaps most foolish to think wishful environmental management justifies releasing such toxins where kids, adults, pets, and wildlife are exposed.

For more, check out Beyond the War on Invasive Species: A Permaculture Approach to Ecosystem Restoration by Orion and Holmgren, 2015.

Posted in agroecology, an uncaring American public, business, corporate responsibility, ecology, Ecology Action, environment, environmental law, epidemiology, evidence, invasive species, open data, Peter del Tredici, quantitative biology, quantitative ecology, rights of the inhabitants of the Commonwealth, risk, statistics, sustainability, sustainable landscaping, the right to know, Uncategorized, unreason, Westwood | Leave a comment

Local Energy Rules!

As John Farrell says, Keep your energy local. If you want to take back control of your democracy, a priority is taking back control of your energy supply. Centralized energy centralizes political power and influence.

Listen to more from a recent podcast:

There are now 52 podcasts about the primacy of local energy at ILSR.

Posted in adaptation, bridge to somewhere, Buckminster Fuller, clean disruption, climate economics, decentralized electric power generation, decentralized energy, demand-side solutions, efficiency, electricity markets, energy efficiency, energy utilities, feed-in tariff, force multiplier, fossil fuel divestment, grid defection, ILSR, investment in wind and solar energy, local generation, local self reliance, public utility commissions, solar democracy, solar domination, solar energy, solar power, Spaceship Earth, stranded assets, sustainability, the energy of the people, the green century, utility company death spiral, wind energy, wind power, zero carbon | Leave a comment

Erin Gallagher’s “#QAnon network visualizations”

See her most excellent blog post, a delve into true Data Science.

(Click on figure to see a full-size image. It is large. Use your browser Back Button to return to this blog afterwards.)

Hat tip to Bob Calder and J Berg.

Posted in data science, jibber jabber, networks | Leave a comment

On lamenting the state of the Internet or Web

From time to time, people complain about the state of the Internet or of the World Wide Web. They are sometimes parts of governments charged with mitigating crime, sometimes privacy advocates, sometimes local governments or retails lamenting loss of tax revenues, sometimes social crusaders charging it with personal isolation, bullying, vice, and other communal maladies.

Certain people have made the pointing out of Web ills their principal theme. Jaron Lanier has long done so, and has written many books on the matter. Cathy O’Neill is a more recent critic, not only of the Web but of businesses which employ it and other data collecting mechanisms to mine imperfectly and abuse their intrinsically imperfect pictures for profit.

Others have underscored the effects of what is predominantly sampling bias. The thing about that is this should be no surprise. What is a surprise is the companies involved don’t see and respond to this as the statistical problem it is. How representative a sample actually is of a population of interest is perhaps the key question in any statistical study. That these companies settle for samples of convenience rather than validated one shows they are practicing very weak data methods, no matter how many people with doctorates are associated with these projects.

There is also the criticism from Professor Lawrence Lessig who understood early the social and legal ramifications of how the Internet and Web are built, particularly in such incisive books as Code and Other Laws of Cyberspace,
Code: Version 2.0
, and Remix. In Code: Version 2.0 Lessig continued and reemphasized the warnings issued in Code and Other Laws of Cyberspace that given the way the Internet and Web were technically structured and funded, the idea of a free market of ideas was slipping away and it was becoming more regulable, more subject to influence by a few big groups, and the user-as-product problem with which Powazek, among others, has taken issue.

Perhaps Lessig has come closest to it, but what the audience of these critics should understand is that the shortcomings they articulate are inherently implied by the technical design of the Internet and Web as they are. The Internet and Web at the experiential level are constructed using the Big Ball Of Mud design anti-pattern. Accordingly, as when any ball of wet mud is subjected to sufficiently large outside forces, it deforms, and becomes arbitrary shapes. Given their present size, however, such deformation is having big social implications, whether China’s aggressive censorship, or foreign influences of United States elections, probably from Russia, or sales of private user data, whether wittingly or not, by large Internet presences.

The thing of it is, there were people who thought carefully and long about how such all-connecting networks should operate and devised specific and careful design principles for them. While there were several (see history), one of the least known, particularly today, is Theodor Holm Nelson. “Ted” Nelson conceived of a non-anonymous network of producers and consumers of content whereby, through a technical device he termed transclusion (coined in Literary Machines), readers would make micropayments to reader or otherwise access content produced by others, with the bulk of these going as compensation to producers and some used for managing the network apparatus. This was termed Xanadu and Nelson and colleagues made several attempts to realize it and several technically related and useful ideas.

This is a difficult problem. Such a structure, if it is not to be defeated or subjugated, needs mechanisms for this built into its technical structure, along with a strong authentication (public key cryptography?) built into it to both prevent theft and identify the party both sending and accepting payments and content. The Internet and Web grew up and grow in a combination of deliberate and careful crafting with haphazard, business-driven choices. Just study how companies operating their innards are paid, and how it got that way. Imposing a rigorous design would make growth expensive, slow, and difficult, demanding a large number of readers and consumers before there was anything to read. Accordingly, Xanadu not only didn’t happen, it couldn’t happen.

However, look where the Internet and Web are now? Spam, malicious attacks, election interference, theft of credit card information, identity theft, viruses, cryptocurrency-driven consumption of excess electrical energy, tracking of individuals by their phones, targeted advertising are some of the places we’ve gone.

What’s intriguing to me is the possibility that Ted Nelson was right all along, and the kind of careful design he had in mind for Xanadu may one day become necessary if the Internet and Web are to survive, and not just splinter into a hundred subsidiary networks each controlled by the biggest Local Thug, whether that is a government or telecommunications giant. Nelson himself believes we can still learn many things from Xanadu.

So, in many senses, the Internet and Web did not have to be the way they are. There were other, better ideas. In fact, considering that, and considering what we’re doing to Earth’s climate through our unmitigated worship of Carbon and growth, if humanity ever needs an epitaph, I think it ought to be:

We did a lot of things, but unfortunately we didn’t think them through.

Posted in American Association for the Advancement of Science, an ignorant American public, an uncaring American public, Anthropocene, being carbon dioxide, bollocks, Boston Ethical Society, bridge to nowhere, Buckminster Fuller, capricious gods, Carbon Worshipers, card games, civilization, climate change, consumption, corporate responsibility, Cult of Carbon, Daniel Kahneman, data centers, David Suzuki, denial, design science, ethical ideals, Faster Forward, Hyper Anthropocene, hypertext, ignorance, Internet, Joseph Schumpeter, making money, Mathbabe, networks, organizational failures, superstition, Ted Nelson, the right to know, the tragedy of our present civilization, transclusion, Xanadu, ZigZag | Leave a comment

“Space, climate change, and the real meaning of theory”

(From The New Yorker, 17th August 2016, by the late former astronaut Dr Piers Sellers)

Excerpt from “Space, climate change, and the real meaning of theory”:

The facts of climate change are straightforward: there’s been a warming surge over the past hundred years, with a dramatic uptick in this new century. We are seeing the effects in the shrinking of the summer Arctic sea ice and the melting of the Greenland glaciers. That melt, in turn, has been partly responsible for the three-inch rise in sea levels since 1992. The Earth is warming, the ice is melting, and sea level is rising. These are observed facts.

Are we humans the cause of these changes? The answer is an emphatic yes. Many climate-research groups around the world have calculated the various contributions to climate change, including those not related to humans, like volcanic ash. It has been shown repeatedly that it is just not possible to explain the recent warming without factoring in the rise in anthropogenic greenhouse gases. If you left the increase in carbon dioxide out of your calculations, you would see a wobbly but, on average, level temperature trend from the eighteen-nineties to today. But the record—the reality—shows a steeply rising temperature curve which closely matches the observed rise in carbon dioxide. The global community of climate scientists, endorsed by their respective National Academies of Science or equivalents, is solid in attributing the warming to fossil-fuel emissions. Humans are the cause of the accelerating warming. You can bet your life—or, more accurately, your descendants’ lives—on it.
Newton’s ideas, and those of his successors, are all-pervasive in our modern culture. When you walk down a street, the buildings you see are concrete and steel molded to match theory; the same is true of bridges and machines. We don’t build different designs of buildings, wait to see if they fall down, and then try another design anymore. Engineering theory, based on Newton’s work, is so accepted and reliable that we can get it right the first time, almost every time. The theory of aerodynamics is another perfect example: the Boeing 747 jumbo-jet prototype flew the first time it took to the air—that’s confidence for you. So every time you get on a commercial aircraft, you are entrusting your life to a set of equations, albeit supported by a lot of experimental observations. A jetliner is just aluminum wrapped around a theory.

Climate models are made out of theory. They are huge assemblies of equations that describe how sunlight warms the Earth, and how that absorbed energy influences the motion of the winds and oceans, the formation and dissipation of clouds, the melting of ice sheets, and many other things besides. These equations are all turned into computer code, linked to one another, and loaded into a supercomputer, where they calculate the time-evolution of the Earth system, typically in time steps of a few minutes. On time scales of a few days, we use these models for weather prediction (which works very well these days), and, on longer time scales, we can explore how the climate of the next few decades will develop depending on how much carbon dioxide we release into the atmosphere. There are three items of good news about this modelling enterprise: one, we can check on how well the models perform against the historical record, including the satellite data archive; two, we can calculate the uncertainty into the predictions; and three, there are more than twenty of these models worldwide, so we can use them as checks on one another. The conclusions drawn from a careful study of thousands of model runs are clear: the Earth is rapidly warming, and fossil-fuel burning is the principal driver.

But theories are abstract, after all, so it’s easy for people to get tricked into thinking that because something is based on theory, it could very likely be wrong or is debatable in the same way that a social issue is debatable. This is incorrect. Almost all the accepted theories that we use in the physical and biological sciences are not open to different interpretations depending on someone’s opinion, internal beliefs, gut feelings, or lobbying. In the science world, two and two make four. To change or modify a theory, as Einstein’s theories modified Newton’s, takes tremendous effort and a huge weight of experimental evidence.

Dr Piers Sellers space flight experience.

Dr Piers Sellers in Greenland:

We need more knowledge in and of Space and Earth, not more force.

Posted in American Association for the Advancement of Science, Anthropocene, climate change, climate data, climate education, climate models, Eaarth, Earth Day, environment, global warming, Hyper Anthropocene, NASA, oceanic eddies, Piers Sellers, Principles of Planetary Climate | Leave a comment

“What follows is not a set of predictions of what will happen …”

Posted in adaptation, Anthropocene, anti-intellectualism, Carbon Worshipers, climate change, global blinding, global warming | 2 Comments

Love means nothing, without understanding, and action

Can’t get enough of this video. It may be a corporate, Ørsted promotion, but it is beautiful.

And I continue to believe, that, as the original sense of the corporation, or benefit society suggested, contrary to (U.S.) popular progressive belief, corporations can be agencies for good.

The place we all call home needs love, but love means nothing — without action.

Posted in Aldo Leopold, American Solar Energy Society, American Statistical Association, Ørsted, Bloomberg, Bloomberg New Energy Finance, bridge to somewhere, Canettes Blues Band, climate, climate business, climate economics, corporate citizenship, corporate litigation on damage from fossil fuel emissions, corporate responsibility, corporate supply chains, corporations, destructive economic development, distributed generation, economics, emergent organization, fossil fuel divestment, global warming, green tech, Green Tech Media, Humans have a lot to answer for, Hyper Anthropocene, investing, investment in wind and solar energy, investments, Joseph Schumpeter, liberal climate deniers, reasonableness, Sankey diagram, solar democracy, solar domination, Spaceship Earth, stranded assets, the energy of the people, the green century, the value of financial assets, wind energy, wind power, wishful environmentalism | Leave a comment

big generation day … first complete with WSS II online

Our additional 3.45 kW solar PV is up and generating today, collecting substantial numbers of photons (500 kWh) by 0800 ET.

(Click on figure to see a larger image and use browser Back Button to return to blog.)

(Click on figure to see a larger image and use browser Back Button to return to blog.)

It’s a good thing! Today is a peak day! At present we are using 1.3 kW for cooling and general house stuff, and generating 9.5 kW, in net pushing 8.2 kW to help our non-solar neighbors to cool their homes without Eversource needing to supply them.

See here and here to follow these data series yourself.

Posted in electricity, engineering, RevoluSun, solar democracy, solar domination, solar energy, solar power,, SunPower, sustainability, the energy of the people, the green century, the value of financial assets, Tony Seba, utility company death spiral | Leave a comment

+10 PV panels! Now at 13.45 kW nameplate capacity

In addition to our 10.0 kW PV generation, we just added an additional 3.45 kW, via 10 additional SunPower X21-345 panels. The new panels are tied to a separate SolarEdge inverter, an SE3800H-US. (The older inverter is an SE10000A-US. The old panels are also X21-345s.) These were all designed and installed by RevoluSun of Burlington, MA. (They are great.)

Here’s are some photographs of the panels, with the new ones marked:

We also got consumption monitoring with the new inverter, although that’s not yet set up in my software.

Overall, the layout now looks like:

Inside, the inverters look like:

This additional increment is intended to offset our air source heat pump hot water heater and especially the charging of our Chevy Volt.

Posted in resiliency, RevoluSun, solar democracy, solar domination, solar energy, solar power,, SunPower, sustainability, the right to know, the value of financial assets, Tony Seba, utility company death spiral, Westwood, zero carbon | 2 Comments

typical streamflow series, and immersive Science

What is that impulse in streamflow actually about? How typical is it? How frequently has it happened in past? How often will it reoccur? What are it’s implications for floodplain planning?

There’s been a bit of discussion, here and there, about what we should or can expect the electorate of a representative democracy to know about Science. Surely, there’s “school learnin”’, and that’s valuable, but some of the most meaningful interactions with Science come from an individual’s experience of it, in vivo if you will. I recently described, in a comment at a blog how certain experiments as an undergraduate Physics student meant an awful lot to me, even if I had mastered the theory in a book. These were emotional connections. Sure, I had been prepared for this, and had already exhibited some kind of emotional commitment in my desire to remain up, late at night, our in winter, in the cold, in order to observe various stellar things, as part of a local Astronomy club. It’s hand in hand: You can’t do decent amateur Astronomy in New England except in frigid winter, because of the Summer humidity and the associated skyglow from places like Providence and Boston. I’m sure it’s worse now. Going deep north in New England is a help, and I’ve sometimes wondered why people there haven’t tried to capitalize on that.

But, I digress.

There’s something about this, whether it’s streamflow measurements, or taking your own weather measurements at home, or amateur Astronomy which bonds a body to the phenomena and to the process of knowing.

The Web and Internet interactions, despite offering superior measurement technology, never quite replace this experience. There is, I think, something to be argued for this kind of immersive experience in Science.

Posted in American Association for the Advancement of Science, science | Leave a comment

Leaders (say they) Don’t Know About Lags

Maybe they don’t. Most people don’t. On the other hand, there’s little more to them than understanding skeet, realizing aiming where the clay pigeon is now is a useless tactic for hitting it. Aim where the pigeon will be is more useful.

Consider one Massachusetts state Representative Thomas Golden, present House Chair of the Telecommunications, Energy and Utilities Committee. During an SOS Climate Disaster: Vigil & Rally on 26th July 2018, addressed the assembled and the public, saying “The Climate Crisis is not an emergency, only a ‘situation.”’

Perhaps he believes that.

Then again, perhaps he is being disingenuous and simply saying, instead, This is not my problem. This is someone else’s problem. I can only respond to what I see. Why might that be disingenuous? Because any child knows there are optimal points to push a swing to kick it higher. If a forest fire is going to consume an island, it’s not effective or even useful to wait until the island is half consumed to act.

Even a rudimentary understanding of causation implies delays are important and a consequence of it.

University of Massachusetts, Boston and their Sustainable Solutions Laboratory has been commissioned to tell leadership what they see. They are apparently not listening, choosing to go with the short term easy road of insubstantial, pretended actions.

This is not anything new. If a leader doesn’t know how to deal with it, they should ask someone who does, unless, of course, Representative Golden, you really want to be a Champion of Ignorance.

Praised Be the Flood.”


(Hat tip to Tamino.)

Posted in adaptation, American Association for the Advancement of Science, American Meteorological Association, AMETSOC, anti-intellectualism, anti-science, atmosphere, attribution, Boston Ethical Society, carbon dioxide, Carbon Worshipers, climate change, climate disruption, climate economics, climate education, Commonwealth of Massachusetts, Cult of Carbon, dynamical systems, environment, ethics, evidence, forecasting, geophysics, global warming, Massachusetts Interfaith Coalition for Climate Action, moral leadership, Our Children's Trust, Principles of Planetary Climate, rights of the inhabitants of the Commonwealth, the right to be and act stupid, the tragedy of our present civilization, tragedy of the horizon, unreason, UU, UU Humanists | Leave a comment

“The Unchained Goddess”, Bell Science Hour, 1958

A tad nostalgic, for the day where humanity could have stopped a bunch of harm from climate change. Also, although from a STEM perspective, the entire show is worthwhile, only the last seven minutes are pertinent to climate change. Moreover, they address it in a flippant kind of way. I’d say this is not a definitive presentation.

Still, collectively, maybe we deserve what’s going to happen, even if those most responsible are likely to dodge the harm of the early repercussions. But, eventually, they or their children or their grandchildren will feel the full wrath of their poor choices.

The physical, material world does not read or believe in that misleading, pathetic thing, called the Bible.

Yeah, I’m an atheist. Better (or worse?), a physical materialist and a humanist. And all I’ve learned and been taught about Christianity suggests to me it is a fundamental charade. Buddhism is better, but it has its problems.

But I mostly abhor the sense of domination people have had over other species, something I think is at root representative of all that is ill with humanity, in its ignorance of its tiny place in the Universe. If want to hear more of that perspective, I advise Carl Sagan’s dramatic, narrated video.

Posted in atheism, climate change, Cosmos, global warming | Leave a comment

`On Records`

This is a reblog from Eli Rabett, one of the post On Records, with additional comments and material from the author-moderator of this blog,

A distinguishing mark of a new record in a time series is that it exceeds all previous values another is that the first value in a time series is always a record.

Given a stationary situation with nothing except chance, aka natural variation, the number of new records should decline to zero, or pretty close, as the series extends in time.

(Click on figure to get a larger image, and use browser Back Button to return to blog.)

The above is from:

S. I. Seneviratne, R. Wartenburger, B. P. Guillod, A. L. Hirsch, M. M. Vogel, V. Brovkin, D. P. van Vuuren, N. Schaller, L. Boysen, K. V.
Calvin, J.n Doelman, P. Greve, P. Havlik, F. Humpenöder, T. Krisztin, D. Mitchell, A.r Popp, K. Riahi, J. Rogelj, C.-F.h Schleussner, J. Sillmann, E. Stehfest, ``Climate extremes, land–climate feedbacks and land-use forcing at 1.5°C'', Philosophical Transactions of the Royal Society A, Mathematical, Physical and Engineering Sciences, 2nd April 2018, \texttt{DOI: 10.1098/rsta.2016.0450}.

While Seneviratne, et al approach the estimates with a deeper and far more nuanced analysis, it’s been known that as the globe warms, land will warm faster:

And the travesty and tragedy are we’ve known about this a damn long time and have done nothing:

That’s from 1958.

See also:

H. D. Matthews, K. Zickfeld, R. Knutti, M. R. Allen, ``Focus on cumulative emissions, global carbon budgets and the implications for climate mitigation targets'', Environmental Research Letters, January 2018, 13(1).

Welcome to your choice:

Posted in Anthropocene, being carbon dioxide, carbon dioxide, Carbon Worshipers, civilization, climate change, climate disruption, Cult of Carbon, ecology, Eli Rabett, ethics, global warming, greenhouse gases, Humans have a lot to answer for, Hyper Anthropocene, liberal climate deniers, Massachusetts Interfaith Coalition for Climate Action, meteorology, Our Children's Trust, planning, pollution, quantitative ecology, radiative forcing, rights of the inhabitants of the Commonwealth, Spaceship Earth, temporal myopia, the right to be and act stupid, the right to know, the tragedy of our present civilization, the value of financial assets, tragedy of the horizon, Victor Brovkin, wishful environmentalism | Leave a comment

CO2 efficiency as a policy concept

I listened to the following talk, featuring Professor Kevin Anderson, who I have mentioned many times here before:

While I continue to be hugely supportive of distributed PV as an energetic and democratic solution, as inspired by John Farrell at ILSR, there is something to be said, in my thinking, for migrating a version of electrical energy efficiency to the CO2 realm. What does that mean?

What it means is choosing to not use a kWh of electrical energy, or a kJ (kiloJoule) when one could. It also means being sensitive to energy intensity. Sure, if I drive to the store up the street, or I walk to the store up the street, the point is that I get me from here to there and back. But the overhead and speed of using the automobile, in contrast with walking or using a bicycle or taking the electrical streetcar that runs down the route is so much higher than do any of those, is something where I, as a responsible member of a climate sensitive society, need to properly evaluate the value of my personal time against polluting The Commons.

Unfortunately, the technological Zeitgeist is to fix all these problems by slathering on additional techno-fixes, and justifying them with techno-casuistry so the lifestyles are preserved. I’m suggesting that a full systems analysis suggests little beats simply choosing to slow down and not use the energy to save time or feed a personal impatience to begin with.

This is worth a look at, both for practical and personal reasons. I, for instance, as a matter of personal discipline, always cross walks at corners now only when the urban lightings permit me to do so.

Posted in Bloomberg New Energy Finance, Boston Ethical Society, climate disruption, ILSR, John Farrell, Kevin Anderson, lifestyle changes, local self reliance, moral leadership, naturalism, personal discipline, Spaceship Earth, Unitarian Universalism, UU, UU Humanists | Leave a comment

One of the most interesting things about the MIP ensembles is that the mean of all the models generally has higher skill than any individual model.

We hold these truths to be self-evident, that all models are created equal, that they are endowed by their Creators with certain unalienable Rights, that among these are a DOI, Runability and Inclusion in the CMIP ensemble mean. Well, not quite. But it is Independence Day in the US, and coincidentally there is a new discussion paper [cite ref=(Abramowitz et al)]10.5194/esd-2018-51[/cite] (direct link) posted on model independence just posted at Earth System Dynamics. …

Source: Model Independence Day

Link | Posted on by | Leave a comment

These are ethical “AI Principles” from Google, but they might as well be `technological principles’

This is entirely adapted from this link, courtesy of Google and Alphabet.


  1. Be socially beneficial.
  2. Avoid creating or reinforcing unfair bias.
  3. Be built and tested for safety.
  4. Be accountable to people.
  5. Incorporate privacy design principles.
  6. Uphold high standards of scientific excellence.
  7. Be made available for uses that accord with these principles. (See important additional explanation at the primary source.)


  1. Technologies that cause or are likely to cause overall harm.
  2. Weapons or other technologies whose principal purpose or implementation is to cause or directly facilitate injury to people.
  3. Technologies that gather or use information for surveillance violating internationally accepted norms.
  4. Technologies whose purpose contravenes widely accepted principles of international law and human rights.

Google does qualify:

We want to be clear that while we are not developing AI for use in weapons, we will continue our work with governments and the military in many other areas. These include cybersecurity, training, military recruitment, veterans’ healthcare, and search and rescue. These collaborations are important and we’ll actively look for more ways to augment the critical work of these organizations and keep service members and civilians safe.

It’s clear this is dancing on a fence, but that uncomfortable position is inevitable in any optimization problem.

It’s curious, but these wouldn’t be bad principles for governments and polities to follow, either.

Posted in American Statistical Association, artificial intelligence, basic research, Bayesian, Boston Ethical Society, complex systems, computation, corporate citizenship, corporate responsibility, deep recurrent neural networks, emergent organization, ethical ideals, ethics, extended producer responsibility, friends and colleagues, Google, Google Pixel 2, humanism, investments, machine learning, mathematics, moral leadership, natural philosophy, politics, risk, science, secularism, technology, The Demon Haunted World, the right to know, Unitarian Universalism, UU, UU Humanists | Leave a comment

National Academies Statement on Harmful Consequences of Separating Families at the U.S. Border


“We urge the U.S. Department of Homeland Security to immediately stop separating migrant children from their families, based on the body of scientific evidence that underscores the potential for lifelong, harmful consequences for these children and based on human rights considerations.

“Reports from the National Academies of Sciences, Engineering, and Medicine contain an extensive body of evidence on the factors that affect the welfare of children – evidence that points to the danger of current immigration enforcement actions that separate children from their parents. Research indicates that these family separations jeopardize the short- and long-term health and well-being of the children involved. In addition, the Committee on Human Rights of the National Academies, which has a long history of addressing issues at the intersection of human rights, science, and health, stresses that the practice of separating parents from their children at the border is inconsistent with U.S. obligations under the International Covenant on Civil and Political Rights.

“Parents’ impact on their children’s well-being may never be greater than during the earliest years of life, when a child’s brain is developing rapidly and when nearly all of her or his experiences are shaped by parents and the family environment (NASEM, 2016, p. 1). Young children who are separated from their primary caregivers may potentially suffer mental health disorders and other adverse outcomes over the course of their lives (NASEM, 2016, p. 21-22). Child development involves complex interactions among genetic, biological, psychological, and social processes (NRC and IOM, 2009, p. 74), and a disruption in any of these – such as family disruption – hinders healthy development and increases the risk for future disorders (NRC and IOM, 2009, p.102-104). Young children are capable of deep and lasting sadness, grief, and disorganization in response to trauma and loss (NRC and IOM, 2000, p. 387). Indeed, most mental, emotional, and behavioral disorders have their roots in childhood and adolescence (NRC and IOM, 2009, p. 1), and childhood trauma has emerged as a strong risk factor for later suicidal behavior (IOM, 2002, p. 3).

“Decades of research have demonstrated that the parent-child relationship and the family environment are at the foundation of children’s well-being and healthy development. We call upon the Department of Homeland Security to stop family separations immediately based on this evidence.”

Marcia McNutt
President, National Academy of Sciences

C. D. Mote, Jr.
President, National Academy of Engineering

Victor J. Dzau
President, National Academy of Medicine

Source: Statement on Harmful Consequences of Separating Families at the U.S. Border

Posted in an ignorant American public, an uncaring American public, anti-intellectualism, anti-science, children as political casualties, compassion, Donald Trump, humanism, Humans have a lot to answer for, immigration, military inferiority, moral leadership, sadism, the right to be and act stupid, the right to know, torture, Unitarian Universalism, United States Government | Leave a comment

Professor Tony Seba, of late

I love it.

Professor Tony Seba, Stanford, 1 week ago.

It means anyone who continues to invest in or support the fossil fuels hegemony will be fundamentally disappointed by the markets. And it serves them right. By efficiency, or momentum, there is no beating energy that has a marginal cost of zero.

As someone once said, in a movie, oppose solar energy, “Go ahead. Make my day.”

Gasoline-powered autos won’t be sidelined because gasoline costs too much, it’ll be because gasoline costs too little. No one will want it, so service stations won’t be able to cover their overheads, and they’ll close: It won’t be available, because no one will care about it any more.

And, as for the homes and businesses who continue to buy into the “presently wise choice” of natural gas? Hah! What happens when they can no longer get it, their pipeline companies shutting down flows?

It’s a beautiful thing.

Oh, sure, they’ll try to socialize their losses. Hopefully the electorate isn’t foolish enough to accept that.

But, then, they are the electorate and are highly gullible.

Posted in American Statistical Association, anti-intellectualism, anti-science, Bloomberg New Energy Finance, BNEF, bridge to nowhere, Buckminster Fuller, Carbon Tax, Carbon Worshipers, causation, central banks, children as political casualties, citizen science, citizenship, clean disruption, climate, climate business, climate change, climate data, climate disruption, climate economics, Climate Lab Book, Climate Science Legal Defense Fund, coastal communities, coastal investment risks, coasts, Commonwealth of Massachusetts, Constitution of the Commonwealth of Massachusetts, consumption, corporate responsibility, corporations, corruption, critical slowing down, ctDNA, Cult of Carbon, David Archer, David Spiegelhalter, decentralized electric power generation | Leave a comment

This flooding can’t be stopped. What about the rest?

Tamino is writing about this subject, too. That entirely makes complete sense as it is the biggest geophysical and environmental story out there right now. I’ve included an update at this post’s end discussing the possible economic impacts.

It’s been known for a couple of years that the West Antarctic ice sheet was destabilizing and that this would result in appreciable sea-level rise. What wasn’t known was how widespread this was in Anarctica, and how fast this might proceed.

Well, we’re beginning to find out, and the news isn’t good.

Data below from “Mass balance of the Antarctic ice sheet from 1992 to 2017”, The IMBIE Team, Nature, 2018, 558, 219-222.


The Antarctic Ice Sheet is an important indicator of climate change and driver of sea-level rise. Here we combine satellite observations of its changing volume, flow and gravitational attraction with modelling of its surface mass balance to show that it lost 2,720 ± 1,390 billion tonnes of ice between 1992 and 2017, which corresponds to an increase in mean sea level of 7.6 ± 3.9 millimetres (errors are one standard deviation). Over this period, ocean-driven melting has caused rates of ice loss from West Antarctica to increase from 53 ± 29 billion to 159 ± 26 billion tonnes per year; ice-shelf collapse has increased the rate of ice loss from the Antarctic Peninsula from 7 ± 13 billion to 33 ± 16 billion tonnes per year. We find large variations in and among model estimates of surface mass balance and glacial isostatic adjustment for East Antarctica, with its average rate of mass gain over the period 1992–2017 (5 ± 46 billion tonnes per year) being the least certain.

For every centimeter [of sea-level rise] from West Antarctica, Boston feels one and a quarter centimeters. And that extends down the East Coast.

Professor Robert M DeConto, University of Massachusetts, Amherst, Geosciences, as quoted in The Atlantic, 13th June 2018, “After decades of losing ice, Antarctica is now hemorrhaging it”.

See also

S. Kruel, “The impacts of sea-level rise on tidal flooding in Boston, Massachusetts”, Journal of Coastal Research, 2016, 32(6), 1302-1309.

which I have already written about here.

It is important to understand that it is too late to stop this part of the effects of climate change: Boston and coasts will flood. We can hope that by the world cutting back on emissions it might slow. But reversing it is out of the question. And, to the degree the world is not keeping on schedule, even the slowing looks out of reach.

But, seriously, it’s unrealistic to think anything else. We have important groups of people (like those who elect Congress and the President) who don’t consider these risks serious, even doubt them, or think that Archangel Michael will come riding down on a big white horse and save us collectively because of Manifest Destiny or some other pious rubbish.

Unfortunately, we did not fund the research to ascertain how fast this could go until late, and we’ve done essentially nothing so far on a serious scale to try to stop it, setting the impossible condition of having to maintain an American economic boom. That might prove to be the most expensive economic expansion the world has ever seen.

Update, 2018-06-17

Beyond the geophysical impact of impending ice sheet collapse, there’s the economic one: If insurance prices don’t head upwards quickly, and real estate prices for expensive homes on the coasts don’t come under downward pressure, the only reason can be is that they expect the U.S. federal government to continue to fund their rebuilding through the Biggert-Waters Act (2012), the Homeowner Flood Insurance Affordability Act (2014), the Stafford isaster Relief and Emergency Assistance Act (1988), the Disaster Mitigation Act (2000), and the Pets Evacuation and Transportation Standards Act (2006). The wisdom of continuing these in the face of increasing storm costs is being questioned with greater ferocity. (See also.) It’s not difficult to see why:

And, courtesy of the New York Times:

And, courtesy of NOAA:

While some critics (e.g., Pielke, et al, “Normalized hurricane damage in the United States — 1900-2005”) have claimed the increases in losses is because there is more expensive property being damaged by otherwise ordinary storms, correction of losses by the Consumer Price Index (CPI) controls for some of that, and the rates of inflation in damage exceed rates of even the most appreciating real estate values. Moreover, if that’s the reason for the losses, nothing is being done to discourage the practice. Sure, inflation might not be able to be controlled, but (a) it has been very low in recent years and, (b), the CPI-adjusted values from NOAA show that’s not the explanation. The amount loss to disasters is climbing and the claim that’s all it’s about is disingenuous at best.

At some point the federal government will stop or significantly limit and curtail bailouts of rebuilds, like the affluent homes on Alabama’s Dauphin Island. At that point the value of coastal real estate will crest, and it may possibly plummet: A classic Minsky moment. It would be inadvisable to own coastal real estimate when that happens, particularly in towns like Falmouth, Massachusetts. See an article from Forbes which reports climate change is already depressing coastal real estate values by 7%.

Posted in adaptation, Antarctica, Anthropocene, bridge to nowhere, Carbon Worshipers, citizenship, civilization, climate, climate change, climate disruption, climate economics, climate justice, coastal communities, coastal investment risks, coasts, Commonwealth of Massachusetts, corporate litigation on damage from fossil fuel emissions, corporate responsibility, Cult of Carbon, environment, Eric Rignot, flooding, floods, glaciers, glaciology, global warming, greenhouse gases, hydrology, Hyper Anthropocene, ice sheet dynamics, icesheets, investing, investments, John Englander, living shorelines, Massachusetts, New England, real estate values, rights of the inhabitants of the Commonwealth, Robert M DeConto, Scituate, sea level rise, seawalls, shorelines, Stefan Rahmstorf, the right to be and act stupid, the right to know, the tragedy of our present civilization, wishful environmentalism, ``The tide is risin'/And so are we'' | Leave a comment

“Will climate change bring benefits from reduced cold-related mortality? Insights from the latest epidemiological research”

From RealClimate, and referring to article in Lancet :

Guest post by Veronika Huber Climate skeptics sometimes like to claim that although global warming will lead to more deaths from heat, it will overall save lives due to fewer deaths from cold. But is this true? Epidemiological studies suggest the opposite. Mortality statistics generally show a distinct seasonality. More people die in the colder winter months than in the warmer summer months. In European countries, for example, the difference between the average number of …

Source: Will climate change bring benefits from reduced cold-related mortality? Insights from the latest epidemiological research

Posted in Anthropocene, climate, climate change, climate disruption, epidemiology, evidence, global warming | Leave a comment

When linear systems can’t be solved by linear means

Linear systems of equations and their solution form the cornerstone of much Engineering and Science. Linear algebra is a paragon of Mathematics in the sense that its theory is what mathematicians try to emulate when they develop theory for many other less neat subjects. I think Linear Algebra ought to be required mathematics for any scientist or engineer. (For example, I think Quantum Mechanics makes a lot more sense when taught in terms of inner products than just some magic which drops from the sky.) Unfortunately, in many schools, it is not. You can learn it online, and Professor Gilbert Strang’s lectures and books are the best. (I actually prefer the second edition of his Linear Algebra and Its Applications, but I confess I haven’t looked at the fourth edition of the text, just the third, and I haven’t looked at his fifth edition of Introduction to Linear Algebra.)

There’s a lot to learn about numerical methods for linear systems, too, and Strang’s Applications teaches a lot that’s very good there, including the SVD of which Professor Strang writes “it is not nearly as famous as it should be.” I very much agree. You’ll see it usable everywhere, from dealing with some of the linear systems I’ll mention below to support for Principal Components Analysis in Statistics, to singular-spectrum analysis of time series, to Recommender Systems, a keystone algorithm in so-called Machine Learning work.

The study of numerical linear algebra is widespread and taught in several excellent books. My favorites are Golub and Van Loan’s Matrix Computations, Björck’s Numerical Methods for Least Squares Problems, and Trefethen and Bau’s Numerical Linear Algebra. But it’s interesting how fragile these solution methods are, and how quickly one needs to appeal to Calculus directly with but small changes in these problems. That’s what this post is about.

So what am I talking about? I’ll use small systems of linear equations as examples, despite it being entirely possible and even common to work with systems which have thousands or millions of variables and equations. Here’s a basic one:

(1)\,\,\,\left[ \begin{array} {c} b_{1} \\ b_{2} \\ b_{3} \end{array} \right] = \left[ \begin{array} {ccc} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{array} \right] \left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right]

written for brevity

(2)\,\,\,\mathbf{b} = \mathbf{A} \mathbf{x}

Of course, in any application the equation looks more like:

(3)\,\,\,\left[ \begin{array} {c} 12 \\ 4 \\ 16 \end{array} \right] = \left[ \begin{array} {ccc} 1 & 2 & 3 \\ 2 & 1 & 4 \\ 3 & 4 & 1 \end{array} \right] \left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right]

In R or MATLAB the result is easily obtained. I work and evangelize R, so any computation here will be recorded in it. Doing



lm(b ~ A + 0)

will produce

(4)\,\,\,\left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right] = \left[ \begin{array} {c} -3 \\ 6 \\ 1 \end{array} \right]

It’s also possible to solve several at once, for example, from:

(5)\,\,\,\left[ \begin{array} {cccc} 12 & 20 & 101 & 200 \\ 4 & 11 & -1 & 3 \\ 16 & 99 & 10 & 9 \end{array} \right] = \left[ \begin{array} {ccc} 1 & 2 & 3 \\ 2 & 1 & 4 \\ 3 & 4 & 1 \end{array} \right] \left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right]

(6)\,\,\,\left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right] = \left[ \begin{array} {c} -3 \\ 6 \\ 1 \end{array} \right], \left[ \begin{array} {c} 15.25 \\ 15.50 \\ -8.75 \end{array} \right], \left[ \begin{array} {c} -73.75 \\ 51.90 \\ 23.65 \end{array} \right], \left[ \begin{array} {c} -146.25 \\ 99.70 \\ 48.95 \end{array} \right]

And, of course, having an unknown \mathbf{b} but a known \mathbf{x} is direct, just using matrix multiplication:

(7)\,\,\,\left[ \begin{array} {c} b_{1} \\ b_{2} \\ b_{3} \end{array} \right] = \left[ \begin{array} {ccc} 1 & 2 & 3 \\ 2 & 1 & 4 \\ 3 & 4 & 1 \end{array} \right] \left[ \begin{array} {c} -73.75 \\ 51.90 \\ 23.65 \end{array} \right]


(8)\,\,\,\left[ \begin{array} {c} b_{1} \\ b_{2} \\ b_{3} \end{array} \right] = \left[ \begin{array} {c} -101 \\ -1 \\ 10 \end{array} \right]

Linear Algebra gives us sensible ways to interpret inconsistent systems like:

(9)\,\,\,\left[ \begin{array} {c} 12 \\ 4 \\ 16 \\ 23 \end{array} \right] = \left[ \begin{array} {ccc} 1 & 2 & 3 \\ 2 & 1 & 4 \\ 3 & 4 & 1 \\    17 & -2 & 11 \end{array} \right] \left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right]

by making reasonable assumptions about what the solution to such a system should mean. R via lm(.) gives:

(10)\,\,\,\left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right] = \left[ \begin{array} {c} 1.46655646 \\ 3.00534079  \\ 0.34193795 \end{array} \right]

Sets of solutions to things like

(11)\,\,\,\left[ \begin{array} {c} 12 \\ 4 \end{array} \right] = \left[ \begin{array} {ccc} 1 & 2 & 3 \\ 2 & 1 & 4 \end{array} \right] \left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right]

can be countenanced and there is even a way which I’ll talk about below for picking out a unique one: the minimum norm solution. This is where the SVD comes in. To learn about all the ways these things can be thought about and done, I recommend:

D. D. Jackson, “Interpretation of inaccurate, insufficient and inconsistent data”, Geophysical Journal International, 1972, 28(2), 97-109.

(That’s an awesome title for a paper, by the way.)

What if there are holes?

Going back to (3), however, suppose instead it looks something like this:

(12)\,\,\,\left[ \begin{array} {c} 12 \\ 4 \\ 16 \end{array} \right] = \left[ \begin{array} {ccc} 1 & 2 & 3 \\ 2 & 1 & a_{23} \\ 3 & 4 & 1 \end{array} \right] \left[ \begin{array} {c} -3 \\ 6 \\ 1 \end{array} \right]

and we don’t know what a_{23} is. Can it be calculated?

Well, it has to be able to be calculated: It’s the only unknown in this system, with the rules of matrix multiplication just being a shorthand for combining things. So, it’s entirely correct to think that the constants could be manipulated algebraically so they all show up on one side of equals, and a_{23} on the other. That’s a lot of algebra, though.

We might guess that \mathbf{A} was symmetric so think a_{23} = 4. But what about the following case, in (12)?

(12)\,\,\,\left[ \begin{array} {c} 12 \\ 4 \\ 16 \end{array} \right] = \left[ \begin{array} {ccc} a_{11} & 2 & 3 \\ 2 & 1 & a_{23} \\ a_{31} & a_{23} & 1 \end{array} \right] \left[ \begin{array} {c} -3 \\ 6 \\ 1 \end{array} \right]

Now there are 3 unknowns, a_{11}, a_{23}, and a_{31}. The answer is available in (3), but suppose that wasn’t known?

This problem is one of finding those parameters, searching for them if you like. To search, it helps to have a measure of how far away from a goal one is, that is, some kind of score. (14) is what I propose as a score, obtained by taking (12) and rewriting it as below, (13):

(13)\,\,\,0 = \left[ \begin{array} {c} 12 \\ 4 \\ 16 \end{array} \right] - \left[ \begin{array} {ccc} a_{11} & 2 & 3 \\ 2 & 1 & a_{23} \\ a_{31} & a_{23} & 1 \end{array} \right] \left[ \begin{array} {c} -3 \\ 6 \\ 1 \end{array} \right]

(14)\,\,\,\left|\left|\left[ \begin{array} {c} 12 \\ 4 \\ 16 \end{array} \right] - \left[ \begin{array} {ccc} a_{11} & 2 & 3 \\ 2 & 1 & a_{23} \\ a_{31} & a_{23} & 1 \end{array} \right] \left[ \begin{array} {c} -3 \\ 6 \\ 1 \end{array} \right]\right|\right|_{2}

(15)\,\,\,||\mathbf{z}||_{2} is an L_{2} norm, and ||\mathbf{z}||_{2} = \sqrt{(\sum_{i=1}^{n} z_{i}^2)}.

In other words, ||\mathbf{z}||_{2} is the length of the vector \mathbf{z}. It’s non-negative. Accordingly, making (14) as small as possible means pushing the left and right sides of (12) towards each other. When (14) is zero the left and right sides are equal.

Now, there are many possible values for a_{11}, a_{23}, and a_{31}. In most applications, considering all flonum values for these is not necessary. Typically, the application suggests a reasonable range for each of them, from a low value to a high value. Let

(\alpha_{11}, \beta_{11})

be the range of values for a_{11},

(\alpha_{23}, \beta_{23})

be the range of values for a_{23}, and

(\alpha_{31}, \beta_{31})

be the range of values for a_{31}, each dictated by the application. If \sigma_{11}, \sigma_{23}, and \sigma_{31} are each randomly but independently chosen from the unit interval, then a particular value of (14) can be expressed

(16)\,\,\,\left|\left|\left[ \begin{array} {c} 12 \\ 4 \\ 16 \end{array} \right] - \left[ \begin{array} {ccc} r(\sigma_{11}, \alpha_{11}, \beta_{11}) & 2 & 3 \\ 2 & 1 & r(\sigma_{23}, \alpha_{23}, \beta_{23}) \\ r(\sigma_{31}, \alpha_{31}, \beta_{31}) & r(\sigma_{23}, \alpha_{23}, \beta_{23}) & 1 \end{array} \right] \left[ \begin{array} {c} -3 \\ 6 \\ 1 \end{array} \right]\right|\right|_{2}


(17)\,\,\,r(\sigma, v_{\text{low}}, v_{\text{high}}) \triangleq v_{low}(1 - \sigma) + \sigma v_{\text{high}}

So, this is an optimization problem where what’s wanted is to make (16) as small as possible, searching among triplets of values for a_{11}, a_{23}, and a_{31}. How does that get done? R package nloptr. This is a package from CRAN which does a rich set of numerical nonlinear optimizations, allowing the user to choose the algorithm and other controls, like ranges of search and constraints upon the control parameters.

Another reason why these techniques are interesting is it is intriguing and fun to see how far one can get knowing very little. And when little is known, letting algorithms run for a while to make up for that ignorance doesn’t seem like such a bad trade.

An illustration

In order to illustrate the I don’t know much case, I’m opting for:

\alpha_{11} = -2
\beta_{11} = 2
\alpha_{23} = -1
\beta_{23} = 8
\alpha_{31} = -6
\beta_{31} = 6

What a run produces is:

nloptr(x0 = rep(0.5, 3), eval_f = objective1, lb = rep(0, 3), ub = rep(1, 3), opts = nloptr.options1, alpha.beta = alpha.beta)

Minimization using NLopt version 2.4.2

NLopt solver status: 5 ( NLOPT_MAXEVAL_REACHED: Optimization stopped because maxeval (above) was reached. )

Number of Iterations....: 100000
Termination conditions: xtol_rel: 1e-04 maxeval: 1e+05
Number of inequality constraints: 0
Number of equality constraints: 0
Current value of objective function: 0.000734026668840609
Current value of controls: 0.74997066329 0.5556247383 0.75010835335

Y1 resulting estimates for a_{11}, a_{23}, and a_{31} are: 1.00, 4.00, 3

That’s nloptr-speak for reporting on the call, the termination conditions and result. The bottom line in bold tells what was expected, that a_{11} = 1, a_{23} = 4, a_{31} = 3.

What about the code? The pertinent portion is shown below, and all the code is downloadable as a single R script from here. There’s also a trace of the execution of that script available as well.

L2norm<- function(x)
sqrt( sum(x*x) )

r<- function(sigma, alpha, beta)
stopifnot( (0 <= sigma) && (sigma <= 1) )
stopifnot( alpha < beta )
alpha*(1 - sigma) + beta*sigma

# Recall original was:
# A<- matrix(c(1, 2, 3, 2, 1, 4, 3, 4, 1), 3, 3, byrow=TRUE)

P1.func<- function(x, alpha.beta)
stopifnot( is.vector(x) )
stopifnot( 3 == length(x) )
sigma11<- x[1]
sigma23<- x[2]
sigma31<- x[3]
alpha11<- alpha.beta[1]
beta11<- alpha.beta[2]
alpha23<- alpha.beta[3]
beta23<- alpha.beta[4]
alpha31<- alpha.beta[5]
beta31<- alpha.beta[6]
P1<- matrix( c( r(sigma11,alpha11,beta11), 2, 3,
2, 1, r(sigma23,alpha23,beta23),
r(sigma31,alpha31,beta31), r(sigma23,alpha23,beta23), 1
nrow=3, ncol=3, byrow=TRUE )

objective1<- function(x, alpha.beta)
stopifnot( is.vector(x) )
stopifnot( 3 == length(x) )
b<- matrix(c(12,4,16),3,1)
x.right<- matrix(c(-3,6,1),3,1)
P1<- P1.func(x, alpha.beta)
d<- b - P1 %*% x.right
# L2 norm
return( L2norm(d) )

nloptr.options1<- list("algorithm"="NLOPT_GN_ISRES", "xtol_rel"=1.0e-6, "print_level"=0, "maxeval"=100000, "population"=1000)

alpha.beta<- c(-2, 2, -1, 8, -6, 6)

Y1<- nloptr(x0=rep(0.5,3),
lb=rep(0,3), ub=rep(1,3),


cat(sprintf("Y1 resulting estimates for a_{11}, a_{23}, and a_{31} are: %.2f, %.2f, %2.f\n",
r(Y1$solution[1], alpha.beta[1], alpha.beta[2]), r(Y1$solution[2], alpha.beta[3], alpha.beta[4]),
r(Y1$solution[3], alpha.beta[5], alpha.beta[6])))

But what is it good for? Case 1: Markov chain transition matrices

Consider again (1):

(1')\,\,\,\left[ \begin{array} {c} b_{1} \\ b_{2} \\ b_{3} \end{array} \right] = \left[ \begin{array} {ccc} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{array} \right] \left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right]

This happens also to be the template for a 3-state Markov chain with their many applications.

The following example is taken from the famous paper by Rabiner, as presented by Resch:

  • L. R. Rabiner, “A tutorial on Hidden Markov Models and selected applications in speech recognition”, Proceedings of the IEEE, February 1989, 77(2), DOI:10.1109/5.18626.
  • B. Resch, “Hidden Markov Models”, notes for the course Computational Intelligence, Graz University of Technology, 2011.
  • They begin with the transition diagram:

    which if cast into the form of (1′) and (2) looks like:

    (18)\,\,\,\left[ \begin{array} {c} b_{1} \\ b_{2} \\ b_{3} \end{array} \right] = \left[ \begin{array} {ccc} 0.8 & 0.05 & 0.15 \\ 0.2 & 0.6  & 0.2 \\ 0.2 & 0.3 & 0.5 \end{array} \right] \left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right]

    The rows, top-to-bottom, are labeled sunny, rainy, and foggy, as are the columns, left-to-right. Cell (i,j) gives the probability for going from state i to state j. For example, the probability of going from sunny to foggy is 0.15. Here’s a prettier rendition from Resch:

    Resch and Rabiner go on to teach Hidden Markov Models (“HMMs”) where \mathbf{A} is not known and, moreover, the weather is not directly observed. Instead, information about the weather is obtained by observing whether or not a third party takes an umbrella to work or not. Here, however, suppose the weather is directly known. And suppose \mathbf{A} is known except nothing is known about what happens after foggyexcept when it remains foggy. Symbolically:

    (19)\,\,\,\left[ \begin{array} {c} b_{1} \\ b_{2} \\ b_{3} \end{array} \right] = \left[ \begin{array} {ccc} 0.8 & 0.05 & 0.15 \\ 0.2 & 0.6  & 0.2 \\ a_{31} & a_{32} & 0.5 \end{array} \right] \left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right]

    Note in (18) or Resch’s tableau how the rows each sum to one. This is a characteristic of first order Markov models: Once in a state, the transition has to go somewhere, even if to stay in that state. Transitions can’t just cause the system to disappear, so all the outgoing probabilities need to sum to one. This means, however, that when what happens when it is foggy is introduced, there aren’t two unconstrained parameters, there is only one. Accordingly, rather than introducing a_{32}, I could write 1 - a_{31}. As it turns out, in my experience with nloptr, it is often better to specify this constraint explicitly so the optimizer
    knows about it rather than building it implicitly into the objective function, even at the price of introducing another parameter and its space to explore.

    The challenge I’ll pose here is somewhat tougher than that faced by HMMs. The data in hand is not a series of sunny, rainy, or foggy weather records but, because, say, the records were jumbled, all that’s in hand is a count of how many sunny, rainy, and foggy days there were, and what the count of days were following them. In particular:

    (20)\,\,\,\left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right] = \left[ \begin{array} {c} 1020 \\ 301 \\ 155 \end{array} \right]

    meaning that the first day of a set of pairs began where the first day was sunny 1020 times, rainy 301 times, and foggy 155 times. Statistical spidey sense wonders about how many observations are needed to pin town transition probabilities well, but let’s set that aside for now. (At least it’s plausible that if ordering information is given up, there might be a need for more count information.) And the count of what the weather was on the second days is:

    (21)\,\,\,\left[ \begin{array} {c} b_{1} \\ b_{2} \\ b_{3} \end{array} \right] = \left[ \begin{array} {c} 854 \\ 416 \\ 372 \end{array} \right]

    or 854 sunny days, 416 rainy days, and 372 foggy foggy days.

    Note that unlike in (16) here in (19) there is no need to pick upper and lower bounds on the value: This is a probability so it is by definition limited to the unit interval. But a_{31} + a_{32} + 0.5 = 1 always so that constraint needs to be stated.

    Here’s the code:

    P2.func<- function(x)
    # Sunny, Rainy, Foggy
    stopifnot( is.vector(x) )
    stopifnot( 2 == length(x) )
    a.31<- x[1]
    a.32<- x[2]
    P2<- matrix( c( 0.8, 0.05, 0.15,
    0.2, 0.6, 0.2,
    a.31, a.32, 0.5
    nrow=3, ncol=3, byrow=TRUE )

    objective2<- function(x)
    stopifnot( is.vector(x) )
    stopifnot( 2 == length(x) )
    x.right<- matrix(c(1020, 301, 155), 3, 1)
    b<- matrix(c(854, 416, 372),3,1)
    P2<- P2.func(x)
    d<- b - P2 %*% x.right
    # L2 norm
    return( L2norm(d) )

    constraint2<- function(x)
    return( (x[1] + x[2] - 0.5 ))

    nloptr.options2<- list("algorithm"="NLOPT_GN_ISRES", "xtol_rel"=1.0e-4, "print_level"=0, "maxeval"=100000, "population"=1000)

    Y2<- nloptr(x0=rep(0.5,2),
    lb=rep(0,2), ub=rep(1,2),


    cat(sprintf("Y2 resulting estimates for a_{31}, a_{32} are: %.2f, %.2f\n",
    Y2$solution[1], Y2$solution[2]))

    This run results in:

    nloptr(x0 = rep(0.5, 2), eval_f = objective2, lb = rep(0, 2), ub = rep(1, 2), eval_g_eq = constraint2, opts = nloptr.options2)

    Minimization using NLopt version 2.4.2

    NLopt solver status: 5 ( NLOPT_MAXEVAL_REACHED: Optimization stopped because maxeval (above) was reached. )

    Number of Iterations....: 100000
    Termination conditions: xtol_rel: 1e-04 maxeval: 1e+05
    Number of inequality constraints: 0
    Number of equality constraints: 1
    Current value of objective function: 0.500013288564363
    Current value of controls: 0.20027284199 0.29972776012

    Y2 resulting estimates for a_{31}, a_{32} are: 0.20, 0.30

    Suppose some of the data is missing? In particular, suppose instead:

    (20a)\,\,\,\left[ \begin{array} {c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right] = \left[ \begin{array} {c} 1020 \\ r(\eta, 155, 1020) \\ 155 \end{array} \right]

    where \eta is on the unit interval and so all that’s known is that x_{2} is between 155 and 1020, that is, bounded by the other two terms in \mathbf{x}.

    Now there are two parameters to search, but they are unconstrained, apart from being on the unit interval. The code for this is:

    P3.func<- function(x)
    # Sunny, Rainy, Foggy
    stopifnot( is.vector(x) )
    stopifnot( 3 == length(x) )
    a.31<- x[1]
    a.32<- x[2]
    # There's an x[3] but it isn't used in the P3.func. See
    # the objective3.
    P3<- matrix( c( 0.8, 0.05, 0.15,
    0.2, 0.6, 0.2,
    a.31, a.32, 0.5
    nrow=3, ncol=3, byrow=TRUE )

    objective3<- function(x)
    stopifnot( is.vector(x) )
    stopifnot( 3 == length(x) )
    x.right<- matrix(c(1020, r(x[3], 155, 1020), 155), 3, 1)
    b<- matrix(c(854, 416, 372),3,1)
    P3<- P3.func(x)
    d<- b - P3 %*% x.right
    # L2 norm
    return( L2norm(d) )

    constraint3<- function(x)
    stopifnot( 3 == length(x) )
    return( (x[1] + x[2] - 0.5 ))

    nloptr.options3<- list("algorithm"="NLOPT_GN_ISRES", "xtol_rel"=1.0e-4, "print_level"=0, "maxeval"=100000, "population"=1000)

    Y3<- nloptr(x0=rep(0.5,3),
    lb=rep(0,3), ub=rep(1,3),


    cat(sprintf("Y3 resulting estimates for a_{31}, a_{32}, and eta are: %.2f, %.2f, %.2f\n",
    Y3$solution[1], Y3$solution[2], Y3$solution[3]))

    The results are:

    nloptr(x0 = rep(0.5, 3), eval_f = objective3, lb = rep(0, 3), ub = rep(1, 3), eval_g_eq = constraint3, opts = nloptr.options3)

    Minimization using NLopt version 2.4.2

    NLopt solver status: 5 ( NLOPT_MAXEVAL_REACHED: Optimization stopped because maxeval (above) was reached. )

    Number of Iterations....: 100000
    Termination conditions: xtol_rel: 1e-04 maxeval: 1e+05
    Number of inequality constraints: 0
    Number of equality constraints: 1
    Current value of objective function: 0.639962390444759
    Current value of controls: 0.20055501795 0.29944464945 0.16847867543

    Y3 resulting estimates for a_{31}, a_{32}, and \eta are: 0.20, 0.30, 0.17, with that \eta corresponding to 301

    That 301 versus the true 372 isn’t too bad.

    An example of where this kind of estimation is done more generally, see:

    But what is it good for? Case 2: Learning prediction matrices

    When systems like (2) arise in cases of statistical regression, the matrix \mathbf{A} is called a prediction or design matrix. The idea is that its columns represent sequences of predictions for the response, represented by the column vector \mathbf{b}, and the purpose of regression is to find the best weights, represented by column vector \mathbf{x}, for predicting the response.

    Consider (2) again but instead of \mathbf{b} and \mathbf{x} being column vectors, as in (5), they are matrices, \mathbf{B} and \mathbf{X}, respectively. In other words, the situation is that there are lots of (\mathbf{b}_{k}, \mathbf{x}_{l}) pairs available. And then suppose nothing is known about \mathbf{A}, that is, it just contains nine unknown parameters:

    (22)\,\,\,\mathbf{A} = \left[ \begin{array} {ccc} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{array} \right]

    There are, of course, questions about how many (\mathbf{b}_{k}, \mathbf{x}_{l}) pairs are needed in tandem with choice of number of iterations (see maxeval discussion in Miscellaneous Notes below.) Here, 8 pairs were used for purposes of illustration.

    (23)\,\,\,\mathbf{X} = \left[\begin{array}{cccccccc}   1356 & 7505 & 4299 & 3419 & 7132 & 1965 & 8365 & 8031 \\   5689 & 8065 & 7001 & 638  & 8977 & 1088 & 3229 & 1016 \\   3777 & 8135 & 3689 & 1993 & 3635 & 9776 & 8967 & 7039   \end{array}   \right]


    (24)\,\,\,\mathbf{B} = \left[\begin{array}{cccccccc}   5215 & 13693 & 7265 &  4217 &  9367 & 10588 & 14372 & 12043 \\  7528 & 17825 & 11024 & 4989 & 14860 &  9447 & 16162 & 13087 \\   6161 & 12798 & 7702 & 3023  &  9551 &  8908 & 11429 &  8734   \end{array}   \right]

    The code for this case is:

    objective4<- function(x)
    stopifnot( is.vector(x) )
    stopifnot( 9 == length(x) )
    B<- matrix(c(5215, 13693, 7265, 4217, 9367, 10588, 14372, 12043,
    7528, 17825, 11024, 4989, 14860, 9447, 16162, 13087,
    6161, 12798, 7702, 3023, 9551, 8908, 11429, 8734
    ), 3, 8, byrow=TRUE)
    X.right<- matrix(c(1356, 7505, 4299, 3419, 7132, 1965, 8365, 8031,
    5689, 8065, 7001, 638, 8977, 1088, 3229, 1016,
    3777, 8135, 3689, 1993, 3635, 9776, 8967, 7039
    ), 3, 8, byrow=TRUE)
    P4<- matrix(x, nrow=3, ncol=3, byrow=TRUE)
    d<- B - P4 %*% X.right
    # L2 norm for matrix
    return( L2normForMatrix(d, scaling=1000) )

    nloptr.options4<- list("algorithm"="NLOPT_GN_ISRES", "xtol_rel"=1.0e-6, "print_level"=0, "maxeval"=300000, "population"=1000)

    Y4<- nloptr(x0=rep(0.5,9),
    lb=rep(0,9), ub=rep(1,9),


    cat("Y4 resulting estimates for \mathbf{A}:\n")
    print(matrix(Y4$solution, 3, 3, byrow=TRUE))

    The run results are:

    nloptr(x0 = rep(0.5, 9), eval_f = objective4, lb = rep(0, 9), ub = rep(1, 9), opts = nloptr.options4)

    Minimization using NLopt version 2.4.2

    NLopt solver status: 5 ( NLOPT_MAXEVAL_REACHED: Optimization stopped because maxeval (above) was reached. )

    Number of Iterations....: 300000
    Termination conditions: xtol_rel: 1e-06 maxeval: 3e+05
    Number of inequality constraints: 0
    Number of equality constraints: 0
    Current value of objective function: 0.0013835300300619
    Current value of controls: 0.66308125177 0.13825982301 0.93439957114 0.92775614187 0.63095968859 0.70967190127 0.3338899268 0.47841968691 0.79082981177

    Y4 resulting estimates for mathbf{A}:
    [,1] [,2] [,3]
    [1,] 0.66308125177 0.13825982301 0.93439957114
    [2,] 0.92775614187 0.63095968859 0.70967190127
    [3,] 0.33388992680 0.47841968691 0.79082981177

    In fact, the held back version of \mathbf{A} used to generate these test data sets was:

    (25)\,\,\,\mathbf{A} = \left[\begin{array}{ccc} 0.663 & 0.138 & 0.934 \\                                                         0.928 & 0.631 & 0.710 \\                                                         0.334 & 0.478 & 0.791  \end{array} \right]

    and that matches the result rather well. So, in a sense, the algorithm has “learned” \mathbf{A} from the 8 data pairs presented.

    Miscellaneous notes

    All the runs of nloptr here were done with the following settings. The algorithm is always ISRES. The parameters xrel_tol = 1.0e-4 and population = 1000. maxeval, the number of iterations, varied depending upon the problem. For Y1, Y2, Y3, and Y4 it was 100000, 100000, 100000, and 300000, respectively In all instances, the appropriate optimization controls are given by the nloptr.optionsn variable, where n \in \{1,2,3,4\}.

    Per the description, ISRES, which is an acronym for Improved Stochastic Ranking Evolution Strategy:

    The evolution strategy is based on a combination of a mutation rule (with a log-normal step-size update and exponential smoothing) and differential variation (a Nelder–Mead-like update rule). The fitness ranking is simply via the objective function for problems without nonlinear constraints, but when nonlinear constraints are included the stochastic ranking proposed by Runarsson and Yao is employed. The population size for ISRES defaults to 20×(n+1) in n dimensions, but this can be changed with the nlopt_set_population function.

    This method supports arbitrary nonlinear inequality and equality constraints in addition to the bound constraints, and is specified within NLopt as NLOPT_GN_ISRES.

    Further notes are available in:

    • T. P. Runarsson, X. Yao, “Search biases in constrained evolutionary optimization,” IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 205, 35(2), 233-243.
    • T. P. Runarsson, X. Yao, “Stochastic ranking for constrained evolutionary optimization,” IEEE Transactions on Evolutionary Computation, 2000, 4(3), 284-294.
Posted in Calculus, dynamic linear models, mathematics, maths, nloptr, numerical algorithms, numerical analysis, numerical linear algebra, numerics, SVD | Leave a comment