“Full-depth Ocean Heat Content” reblog

This is a re-blog of an excellent post at And Then There’s Physics, titled Full-depth OHC or, expanded, “full-depth ocean heat content”.

Since my holiday is now over, I thought I might briefly comment on a recent paper by Cheng et al., called Observed and simulated full-depth ocean heat-content changes for 1970–2005. John Abraham, o…

Source: Full-depth OHC

Posted in Anthropocene, climate, climate change, climate data, climate disruption, climate models, computation, differential equations, ensembles, environment, fluid dynamics, forecasting, geophysics, global warming, greenhouse gases, Hyper Anthropocene, Lorenz, Mathematics and Climate Research Network, model comparison, NOAA, oceanography, physics, science, statistics, theoretical physics, thermodynamics, time series | Leave a comment

Richard Somerville, UCSD, Scripps: “The science is becoming more widely accepted”

By Richard Somerville, emiritus professor of Oceanography from Scripps Institution of Oceanography. See the site he helps build and run regarding communication regarding change.

Somerville_IPCC_summary_2016-07-24_154819

Posted in adaptation, American Meteorological Association, AMETSOC, Anthropocene, atmosphere, citizenship, civilization, climate, climate change, climate disruption, environment, forecasting, global warming, meteorology, oceanography, Principles of Planetary Climate, science, Scripps Institution of Oceanography, the right to know, the tragedy of our present civilization, University of California, zero carbon | Leave a comment

“Stochastic Parameterization: Towards a new view of weather and climate models”

Judith Berner, Ulrich Achatz, Lauriane Batté, Lisa Bengtsson, Alvaro De La Cámara, Hannah M. Christensen, Matteo Colangeli, Danielle R. B. Coleman, Daan Crommelin, Stamen I. Dolaptchiev, Christian L.E. Franzke, Petra Friederichs, Peter Imkeller, Heikki Järvinen, Stephan Juricke, Vassili Kitsios, François Lott, Valerio Lucarini, Salil Mahajan, Timothy N. Palmer, Cécile Penland, Mirjana Sakradzija, Jin-Song Von Storch, Antje Weisheimer, Michael Weniger, Paul D. Williams, Jun-Ichi Yano, Stochastic Parameterization: Towards a new view of weather and climate models, Bulletin of the American Meteorological Society, published online 19^{th} July 2016,

Abstract

Stochastic parameterizations — empirically derived, or based on rigorous mathematical and statistical concepts — have great potential to increase the predictive capability of next generation weather and climate models.

The last decade has seen the success of stochastic parameterizations in short-term, medium-range and seasonal forecasts: operational weather centers now routinely use stochastic parameterization schemes to better represent model inadequacy and improve the quantification of forecast uncertainty. Developed initially for numerical weather prediction, the inclusion of stochastic parameterizations not only provides better estimates of uncertainty, but it is also extremely promising for reducing longstanding climate biases and relevant for determining the climate response to external forcing.

This article highlights recent developments from different research groups which show that the stochastic representation of unresolved processes in the atmosphere, oceans, land surface and cryosphere of comprehensive weather and climate models (a) gives rise to more reliable probabilistic forecasts of weather and climate and (b) reduces systematic model bias.

We make a case that the use of mathematically stringent methods for the derivation of stochastic dynamic equations will lead to substantial improvements in our ability to accurately simulate weather and climate at all scales. Recent work in mathematics, statistical mechanics and turbulence is reviewed, its relevance for the climate problem demonstrated, and future research directions outlined.

And five related papers, from another field:

Posted in biology, climate models, complex systems, convergent cross-mapping, data science, dynamical systems, ecology, Ethan Deyle, Floris Takens, George Sughihara, Hao Ye, likelihood-free, Lorenz, mathematics, meteorological models, model-free forecasting, physics, population biology, population dynamics, quantitative biology, quantitative ecology, Scripps Institution of Oceanography, state-space models, statistical dependence, statistics, stochastic algorithms, stochastic search, stochastics, Takens embedding theorem, time series, Victor Brovkin | 4 Comments

Natural gas: The Zaphod Beeblebrox of energy

Amber Lin at The Bulletin of the Atomic Scientists describes the two-headed character of natural gas plants needed to implement “natural gas as a bridge fuel”, and sketches the stark reality proponents of that argument are embracing if they are serious about using natural gas, whether for electricity or heating, to reduce greenhouse gas emissions.

The basic fact is that in order to serve as a proper “bridge”, natural gas infrastructure would need to be decommissioned by 2050, including ceasing flows of the gas through the elaborate pipelines which criss-cross the United States. That’s because emission limits for CO2 dictated by Nature cannot be met otherwise, with 450 ppm CO2, just 40 ppm higher than where we are now, corresponding to the widely accepted +2°C warming limit. And, as that is unlikely, if we want to limit warming to +3°C 650 ppm is the overall limit, and +3°C brings us into a highly uncertain, dangerous, and eventually ice-free world. In particular, we might lose control of a portion of the warming process, since large natural stores of CO2 are quite likely to be breached and begin leaking at those temperatures.

The Presidential commission on the matter also sketched the key problem with using a “bridge fuel” mechanism to reach targets like this, namely, “A slow start leads to a crash finish”, meaning that to hit these targets, the abandonment of fossil fuels and their infrastructure must be pursued much more quickly than if we start early. I daresay, none of the proposals for new natural gas generation have incorporated operating lifetimes which abruptly end in 2050, or depreciation schedules which reflect that. In fact, the new Massachusetts Salem Harbor gas-powered electricity generator has a lifetime up through 2080.

mixedZG

Amber Lin tells how there are really two incompatible kinds of natural gas plants for electricity generation:

When constructing a new natural gas power plant, there are two options: a combined cycle or an open cycle. A combined-cycle power plant produces electricity with relatively high efficiency and low carbon emissions: When the gas burns, it heats and compresses air to spin a turbine and power a generator. A heat recovery system captures waste heat, which is routed to a nearby steam turbine to generate even more power. Combined-cycle plants have low operating costs, but because high capital costs must be offset, these plants are built to produce baseload power—available 24 hours a day. Open-cycle gas turbine plants lack the steam cycle, so their thermal efficiency is much lower, and their carbon emissions per unit of electricity generated are slightly higher. Their running costs are much higher than a combined-cycle plant, but they have a much lower start-up cost, so they are often built as “peakers,” plants that run only to support other power infrastructure during hours of high demand or when solar or wind isn’t available.

Considering the two choices in the larger context of natural gas as a “transition fuel,” a dilemma appears: To build the bridge, combined-cycle is what is needed—a consistent, efficient, power source that can effectively replace coal. But for a combined-cycle natural gas plant to be economically feasible, it would typically need 15 to 20 years to make up for start-up costs, and even longer to become profitable. This means that a combined-cycle plant built in 2016 would break even no sooner than 2031, and would have to run for several more decades to be a worthwhile investment. Levi’s 2030 limit for peak emissions, and roughly 2050 limit for zero emissions, translate to major fossil fuel reductions after 2030. Owners and backers, however, will not want to shut down gas plants that are just beginning to generate a profit. Thus, building combined-cycle plants in 2016 without an explicit understanding of their necessarily temporary nature—and with no financial incentives for early closures in the future—defeats the purpose of natural gas as a “transition fuel.”

Why not focus on open-cycle plants instead? While “peakers” make sense as backups for future renewable energy sources, they don’t make sense right now. In the current infrastructure, they can only run for a couple hundred hours a year before they cost more than they can earn; this is not nearly enough to displace coal. Closed-cycle plants can help build the bridge but cannot close it, and open-cycle plants can help close the bridge but cannot build it. Neither type of plant is both economically feasible in the long run, and powerful enough to meet today’s demand while cutting emissions in time to mitigate climate change. However, when natural gas is branded as a “transition fuel” in politics and in popular media, this crucial detail is rarely mentioned.

(Emphasis added by blog author.)

So natural gas plants are the Zaphod Beeblebrox of electricity generation, as they are duplicitous and their purpose is to distract from the true goals of natural gas infrastructure expansion, to prolong the day when fossil fuel assets are stranded because of government action to mitigate climate change, or, as increasingly plausible, it is taxed for its Carbon.

Luckily Arthur’s Betelgeusean friend, Ford Prefect, a roving researcher for that illustrious interstellar travel almanac The Hitchhikers Guide to the Galaxy, was more of an optimist. Ford saw silver linings where Arthur saw only clouds and so between them they made one prudent space traveller, unless their travels led them to the planet Junipella where the clouds actually did have silver linings. Arthur would have doubtless steered the ship straight into the nearest cloud of gloom and Ford would have almost certainly attempted to steal the silver, which would have resulted in the catastrophic combustion of the natural gas inside the lining. The explo­sion would have been pretty, but as a heroic ending it would lack a certain something, i.e. a hero in one piece.

(An extract from The Hitchhikers Guide to the Galaxy.)

Posted in adaptation, Anthropocene, atmosphere, Bloomberg, Bloomberg New Energy Finance, BNEF, bridge to nowhere, bridge to somewhere, carbon dioxide, Carbon Tax, Carbon Worshipers, citizenship, civilization, climate, climate change, climate disruption, climate economics, consumption, decentralized electric power generation, decentralized energy, distributed generation, electricity, electricity markets, energy, energy utilities, explosive methane, fossil fuel divestment, fossil fuels, fracking, gas pipeline leaks, global warming, greenhouse gases, greenwashing, Hyper Anthropocene, investment in wind and solar energy, leaving fossil fuels in the ground, Massachusetts, Massachusetts Clean Energy Center, methane, natural gas, networks, petroleum, pipelines, planning, politics, public utility commissions, PUCs, rate of return regulation, rationality, reason, reasonableness, regulatory capture, Sankey diagram, solar domination, stranded assets, supply chains, the energy of the people, the green century, the right to be and act stupid, the right to know, the tragedy of our present civilization, the value of financial assets, zero carbon | Leave a comment

The Presidential betting markets

Someone blatantly misrepresented the U.S. Presidential election betting markets in a Google+ comment thread tonight, and I wanted to bring these forward, here.

See the latest odds and assessments from the prediction markets.

Done.

2016-07-22_010019

No doubt some supporters of Trump will argue “God is on our side, and so these heathen markets cannot be correct”. I’ll bet.

Update, 2016-07-24

Current odds on Betfair.

Posted in forecasting, investing, politics, prediction markets, rationality, reasonableness, statistics | Leave a comment

Now, if we could only say the same thing about Massachusetts …

Massachusetts is supposed to be a Blue State.

Massachusetts is supposed to be concerned about the environment, full of tree-hugging eco-weenies (like myself!), and sprouting solar panels from every other rooftop.

Massachusetts is supposed to have aggressive support for zero Carbon energy, including incentives, SRECs, and so on.

But facts are different.

59% of Massachusetts electricity comes from explosive methane (“natural gas” to those of you who prefer industry adverts). This is a potent greenhouse gas which, in 20 year timeframes, is 90x worse than CO2 for climate disrupting radiative forcing. (See https://667-per-cm.net/about if you have doubts.) Natural gas ain’t granola. And the calculations which suggest it is better for the environment are, in my opinion, whacked and bupkis. Set aside upstream impacts from fracking. Not all methane is burnt when it goes up your chimney for heating, nor in generating plants. There are big leaks throughout the Boston metropolitan area which the utilities will fix “if they are dangerous”, but they don’t consider greenhouse gas emissions dangerous. And we all known we have to transition off of fossil fuels, for the good of ourselves, a coastal state as we are, and the moral good of the planet and people on it, not to mention the recently affirmed requirements of the Global Warming Solutions Act (“GWSA”). The Union of Concerned Scientists says we are getting overdependent upon natural gas. And the comparison with coal as a benefit is a logical fallacy of “the worst negates the bad”.

That’s quite different than Texas. Yes, Texas. Home of cowboy boots, and guns, and Spectra Energy.

climatehopesg

  1. Texas holds the record for all-time wind energy production.
  2. The benefits of wind are estimated at $3.3 billion annually.
  3. Texas was the first US state to reach 10,000 megawatts of wind power generating capacity.
  4. Texas was One of the First U.S. States to Require a Certain Amount of Electricity Come from Renewable Energy Sources. (As was Massachusetts, to its credit.)
  5. Texas wind power is cheaper than fossil fuels.
  6. The Texas wind industry employs more than 24,000 workers.

Without going really big on offshore wind and solar, Massachusetts could be being just being a bunch of chumps. And I often wonder if Spectra Energy isn’t trying to dump their explosive methane here, because people at home know better. Or it could be Massachusetts citizens are hypocrites, claiming to be for something, until it affects their own back yards. Or it could be, Massachusetts leadership is having $100,000 spent on them, just in 2016.

windpower

0413bus_wind

AlamoIV_2016-07-21_230755

Alamo-I-Aerial-1024x683

Posted in Anthropocene, Bloomberg New Energy Finance, bridge to nowhere, bridge to somewhere, Buckminster Fuller, Cape Wind, Carbon Worshipers, citizenship, clean disruption, climate business, corruption, decentralized electric power generation, decentralized energy, demand-side solutions, destructive economic development, distributed generation, economics, electricity, electricity markets, energy, energy utilities, explosive methane, fossil fuel divestment, fossil fuels, gas pipeline leaks, Green Tea Coalition, greenhouse gases, Hyper Anthropocene, investment in wind and solar energy, ISO-NE, Joseph Schumpeter, leaving fossil fuels in the ground, local generation, MA, Mark Jacobson, Massachusetts, Massachusetts Clean Energy Center, Massachusetts Interfaith Coalition for Climate Action, Michael Osborne, natural gas, New England, Nikola Tesla, pipelines, politics, public utility commissions, PUCs, rate of return regulation, rationality, reason, reasonableness, regulatory capture, risk, Sankey diagram, solar energy, solar power, Spaceship Earth, supply chains, Texas, the energy of the people, the green century, the tragedy of our present civilization, the value of financial assets, Tony Seba, wind energy, wind power, zero carbon | Leave a comment

David Spiegelhalter on `how to spot a dodgy statistic’

In this political season, it’s useful to brush up on rhetorical skills, particularly ones involving numbers and statistics, or what John Allen Paulos called numeracy. Professor David Spiegelhalter has written a guide to some of these tricks. Read the whole thing. Highlights, though, of devices used to produce statistics which aren’t-quite-right (that is, wrong):

  • Use a real number, but change its meaning
  • Make the number look big (but not too big)
  • Casually imply causation from correlation
  • Choose your definitions carefully
  • Use total numbers rather than proportions (or whichever way suits your argument)
  • Don’t provide any relevant context
  • Exaggerate the importance of a possibly illusory change
  • Prematurely announce the success of a policy initiative using unofficial selected data
  • If all else fails, just make the numbers up

David Spiegelhalter is the Winton Professor of the Public Understanding of Risk at the University of Cambridge and president elect of the Royal Statistical Society. Among many other things, he’s an advocate for expressing life risks as micromorts.

Posted in abstraction, anemic data, Bayes, Bayesian, chance, citizenship, civilization, corruption, Daniel Kahneman, disingenuity, Donald Trump, education, games of chance, ignorance, maths, moral leadership, obfuscating data, open data, perceptions, politics, rationality, reason, reasonableness, rhetoric, risk, sampling, science, sociology, statistics, the right to know | Leave a comment

Rushing the +2 degree Celsius boundary

I made a comment on Google+ pertaining to a report of a recent NOAA finding.

Enjoy.

But remember that COP21 boundary is equivalent to 450 ppm CO2.

Posted in adaptation, AMETSOC, Anthropocene, atmosphere, Bill Nye, bridge to nowhere, carbon dioxide, Carbon Tax, Carbon Worshipers, citizenship, civilization, clean disruption, climate, climate disruption, COP21, corporate litigation on damage from fossil fuel emissions, differential equations, disruption, distributed generation, Donald Trump, ecology, El Nina, El Nino, energy, energy reduction, engineering, environment, environmental law, Epcot, explosive methane, forecasting, fossil fuel divestment, fossil fuels, geophysics, global warming, greenhouse gases, greenwashing, Hyper Anthropocene, investment in wind and solar energy, IPCC, local generation, Mark Jacobson, Martyn Plummer, microgrids, Miguel Altieri, philosophy, physical materialism, R, resiliency, Ricky Rood, risk, Sankey diagram | Leave a comment

BOYCOTT natural gas, American or otherwise

It’s one thing to oppose pipelines and continued use of fossil fuels, but there is little as effective as a boycott of the key product. This is certainly not a new idea. (I don’t do Facebook. See this 2001 article as well.) So if you want to nudge in the direction of renewables, please consider boycotting natural gas. If you want to save money in the long term, please consider leaving natural gas. Natural gas and other fossil fuel prices are inherently volatile. Complaints of their being too high some times are really complaints about this volatility. Renewable energy produces electricity at the same price decade after decade.

Natural gas ain’t granola. Despite company advertisements to the contrary, drilling and fracking natural gas wells and associated infrastructure, including pipelines, compressor stations, and piping and metering stations are invasive, disruptive, expensive, and harmful to people, the environment, and the climate. Methane, the chief component of natural gas, is many times more powerful as a greenhouse gas than is CO2, and even at the burning end, combustion of natural gas is not complete, so there is leakage, even if the raw chemistry of all the component that is burnt is much cleaner than coal. Moreover, gas leaks, from pipelines, from nearly every step along the way, and especially in distribution networks near homes. And don’t think that because you hear reassuring things from utilities and gas companies and engineers that there’s safety there. It may be out of sight, but the political process and the Natural Gas Act of 1938 rigs the federal system against all opponents of natural gas, from cities and towns, down to localities and homeowners and farmers.

Co-constituents of natural gas with methane are carcinogenic and are powerfully harmful of human breathing and lungs. Even the odorants which are added to facilitate detection of leaks are themselves harmful.

  • The easiest way is to design your new home with solar PV, energy efficiency, and electric heating/cooling in mind. Induction stoves are wonderful devices, bringing most of the benefits of gas stoves and energy efficiency to an all-electric footprint. Many new homes, particularly large ones, have excellent roofs and yards for solar PV arrays. Our home has appreciable tree shading, especially in summer, but we solved that by oversizing the array we installed, and, so, generating like crazy for the parts of the day and year we do see unimpeded Sun. And those puffy cumulus clouds are truly awesome helpers.
  • Consider refitting your home and getting off natural gas. We were never on natural gas, but we once did heat our home with oil heated forced hot water and got our hot water that way. Now, our home is zero Carbon, since we heat and cool with ductless minisplits, and have an electric air heat pump hot water heater. We even have an electric, battery-powered lawnmower. Payback times are better each year. (Our solar array will pay for itself in 7 years, this being in Massachusetts.) Your state may have an incentive program and, from what I have studied, it is a win in any case: You’ll just not earn as much back as people do in states that have strong support of solar PV, like New York. The town of Minster, OH went big on solar despite their state’s punitive measures upon solar owners.
  • If you cannot afford solar, or your roof or yard is shielded from Sun, or you live in an apartment, consider the Relay Power community solar program, and switch over from natural gas to at least one ductless minisplit system. Relay Power is available to anyone in the Eversource electricity region in Massachusetts, and there are community solar programs elsewhere in the United States. Check yours.
  • Make your house more efficient! The less heat you need, the less gas you use. If you have a gas-powered clothes dryer, seriously consider drying your clothes by hanging them on racks on your deck or from a clothesline. Dampness in a house is not good for the house, and clothes smell better and feel better, in my opinion, when dried outdoors. If you use a gas oven, consider getting a much smaller top-of-counter electric oven. In our experience, most cooking for singles and couples does not need the big oven. Most of these small ovens are big enough to bake pizza. We have a Breville Convection Smart Oven and we love it.
  • Support your local environmental organization and political action committee to push your state governor and legislature for better energy policy, one which advocates for zero Carbon energy. The more of this there is, the cheaper is electric power in your state, and the lower the price of natural gas, even if you continue to use it.
  • Prohibit reimbursements like the Pipeline Tax. Utilities and gas companies are for-profit corporations. They should be able to get loans to build infrastructure from the private sector. They don’t need taxpayers to share the risk.

Natural gas … WE DON’T WANT YOUR PIPELINE We don’t want your damn gas.

Posted in Anthropocene, atmosphere, Bloomberg New Energy Finance, BNEF, bridge to nowhere, Carbon Worshipers, citizenship, civilization, climate change, climate disruption, climate economics, climate justice, decentralized electric power generation, decentralized energy, denial, destructive economic development, distributed generation, ecology, economics, electricity, electricity markets, energy, environment, explosive methane, fossil fuel divestment, fossil fuels, fracking, gas pipeline leaks, global warming, greenhouse gases, greenwashing, Hyper Anthropocene, ignorance, ISO-NE, leaving fossil fuels in the ground, methane, natural gas, New England, politics, public utility commissions, rationality, regulatory capture, Sankey diagram, the energy of the people, the problem of evil, the right to be and act stupid, the stack of lies, the tragedy of our present civilization, zero carbon, ``The tide is risin'/And so are we'' | Leave a comment

Mark Carney: Why are financial regulators and central bank governors looking at climate?

http://www.cbc.ca/i/caffeine/syndicate/?mediaId=725874755644

“We don’t want a Minsky moment about climate.”

Update, 2016-07-19

Interesting that Carney talks about “stabilizing at a temperature” when emissions are stabilized using a Carbon tax. He agrees with a Carbon tax, but he seems to have his science wrong. I did not get the impression he understands that to stabilize at any temperature, Carbon emissions need to go to zero. In his world, I wonder, does that mean that a price on Carbon needs to go to infinity? From my perspective, there is an implicit ceiling on Carbon price, and that is the realistic price per tonne to exact a unit of Carbon from atmosphere. Perhaps it would be more, before it’s not just about extracting this tonne of Carbon but this one, and another, and more. But, still, there is a kind of ceiling.

Posted in adaptation, Anthropocene, Bloomberg, Bloomberg New Energy Finance, BLUE, central banks, civilization, climate, climate business, climate change, climate disruption, climate economics, climate education, climate justice, corporate litigation on damage from fossil fuel emissions, corporate supply chains, demand-side solutions, ecology, economics, education, environment, false advertising, finance, fossil fuel divestment, fossil fuels, global warming, greenwashing, grid defection, insurance, investing, Joseph Schumpeter, liberal climate deniers, local generation, organizational failures, rate of return regulation, rationality, reasonableness, solar domination, Spaceship Earth, stranded assets, sustainability, the right to know, the value of financial assets, zero carbon | Leave a comment

JASA demands code and data be supplied as a condition of publication

The Journal of the American Statistical Association (“JASA”) has announced in this month’s Amstat News that effective 1st September 2016 “… will require code and data as a minimum standard for reproducibility of statistical scientific research.” Trends were heading this way, but it is excellent to see a major journal insisting upon it as standard practice.

There appear to be some weasel words allowing publications having “proprietary data” to move forward, insisting upon code nevertheless. I can only imagine that publications opting for that path will be seen as less established, solid, or compelling.

Posted in American Association for the Advancement of Science, American Statistical Association, citizen science, engineering, ethics, evidence, new forms of scientific peer review, numerical software, planning, rationality, reasonableness, resiliency, science, statistics, stochastic algorithms, testing, the right to know | Leave a comment

Newt Gingrich and Van Jones. Right on.

It’s the thing. And it addresses how media and people forget about the actual statistics, and focus on the White Hot Bright Light.

What’s striking is that people have apparently forgotten that Statistics, as a field and a profession, was created principally as a vehicle for betterment of society. Not only was this true of creative originators like Florence Nightingale, but the very idea of creating mortality tables for mutual insurance companies.

Posted in American Statistical Association, Bayes, Bayesian, citizen science, criminal justice, Daniel Kahneman, ethics, evidence, fear uncertainty and doubt, humanism, Lives Matter, logistic regression, Markov Chain Monte Carlo, MCMC, organizational failures, population biology, rationality, reasonableness, risk, statistics, Susan Jacoby, the right to know | Leave a comment

What makes me nervous

With regard to my comment at hypergeometric | July 13, 2016 at 3:50 pm on Tamino’s blog, someone challenged me on my assertion “Believe me, the +3C-+4C worlds are not places we want to go!” there. I have replied at Tamino’s blog, and I recommend reading the excellent comments there for context, but I thought it worthwhile to state my position here as well. Below is the quote. And we are apparently heading to the +3C to +4C region, even if COP21 is fully implemented. (See NCAR’s full report.)
Sanderson-ONeill-Tebaldi_fig1_2016-07-14_163539
Sanderson-ONeill-Tebaldi_fig4_2016-07-14_164002


While a detailed substantiation is inappropriate for this space, I’ll just mention three:

  1. a World Bank report on the risks of a +4C degree world
  2. The 2013 paper by Caballero and Huber indicating Eocene warming for atmospheric CO2 concentrations comparable to those we may see in the next 100 years and based upon paleoclimate records was +13C +- 3C warmer than today, not merely +4C
  3. An introduction to and interpretation of Caballero and Huber by Ray Pierrehumbert (“Hot climates, high sensitivity”) in terms of non-linear, state-dependent climate sensitivity, described there as a Figure 1

The basis for my assertion is that we definitely do not want to approach the region [Professor] Ray [Pierrehumbert] writes “Here there (may) be dragons.”

While I am not a climate scientist, I am enough of a dynamicist to look at the rate with which we are introducing greenhouse gases compared to natural processes we can read in the paleorecord (with the possible exception of the Permian extinction event) to wonder whether or not we are actively exploring the climate state space for dynamical bifurcations. People who have examined the question suggest we would probably never know if we were approaching one. While there’s little that can be done except to press on for rapid reductions in CO2 emissions, I can only be honest and say these realizations make me very nervous.



Update, 2016-07-16

From Richard Pauli. See his blog for more.

Posted in adaptation, AMETSOC, Anthropocene, bifurcations, bollocks, bridge to nowhere, carbon dioxide, Carbon Worshipers, civilization, climate, climate change, climate disruption, climate education, complex systems, differential equations, dynamical systems, Eaarth, ecology, environment, fossil fuels, games of chance, geophysics, global warming, Hyper Anthropocene, meteorology, methane, natural gas, oceanography, Principles of Planetary Climate, Ray Pierrehumbert, science, the problem of evil, the right to be and act stupid, the tragedy of our present civilization | 1 Comment

Three stories of solar energy domination: Which outcome would YOU prefer?

(Updated, 2016-07-14)

See Shayle Kann’s great piece at GreenTech media. The choices:

  1. “Version one: Aimless transformation”
  2. “Version two: The balkanized grid”
  3. “Version three: Embracing the transformation”

In addition to solar PV, wind energy of all forms (especially underutilized local wind turbines), and energy storage, Kann is right on, in my opinion, emphasizing the great potential of blockchain technology. See here for a primer.

Also, the supposed need for base load is a chimera and just as mythical, and concerns about duck curves are misplaced. The grid is a network, and like any network, including the communications network known as the Internet, load needs to be shaped some times. That’s part of what demand response is about, but what some fail to see is that, on this point, a grid is better off having a large number of spatially separated, small generators than a few large generators, even if the large generators are all zero Carbon. This is particularly true if some of the generators have their own energy storage or are entirely energy storage centers.

Whatever you may think of nuclear power, an article from the Rocky Mountain Institute on the closing of the Diablo Canyon Nuclear Plant pointed out another thing I did not know about these large “base load” generation facilities:

As Germany found, integrating varying solar and wind power with steady “baseload” plants can present challenges for the opposite of the reason originally supposed: not because wind and solar power vary (demand varies even less predictably), but because “baseload” plants are too inflexible.


That’s interesting.

On the nuclear plant issue, I would personally vastly prefer keeping old and existing nuclear plants open for longer than bringing on new natural gas generation facilities and especially new natural gas pipelines. Admittedly, as noted, nuclear power cannot respond as quickly as can these “peaking gas plants”. However, I have never seen a convincing argument on why this is the only way this can be done, rather than using smart controls and allocation on the grid. We know the grid needs to be made smarter. Why does dumping billions of dollars into peaking plants and pipelines make sense, and not into modernizing a grid that needs to go there? I seriously doubt the reason is a technical one. I seriously suspect it has to do with business models, profitability, and policies people are comfortable with because that’s how they’ve always done them. They are squirrels who have crossed a road three-fourths of the way, and now that there’s an onrushing car, they turn around and want to go back to where they came from.

And, finally, whatever the road chosen on the grid, as I’ve emphasized here repeatedly, even the conservative (*) and Carbon worshipping U.S. Energy Information Administration is now projecting a great role for zero Carbon energy by 2030. And the news from REN continues to be excellent.

And remember, all forms of electrical energy are intermittent.


* By “conservative” I do not mean political conservativism but, rather, the tendency for the EIA to most reluctantly budge from extrapolation of past data onto what would be considered reasonable economic forecasting in nearly any other field.

Posted in adaptation, Anthropocene, biofuels, Bloomberg New Energy Finance, BNEF, bridge to somewhere, Buckminster Fuller, business, Chris Goodall, citizenship, civilization, clean disruption, conservation, consumption, decentralized electric power generation, decentralized energy, demand-side solutions, destructive economic development, disruption, distributed generation, ecology, Ecology Action, economics, efficiency, electricity, electricity markets, energy, energy reduction, energy storage, energy utilities, engineering, environment, fossil fuel divestment, green tech, grid defection, Hermann Scheer, Hyper Anthropocene, Internet, investing, investment in wind and solar energy, ISO-NE, Joseph Schumpeter, leaving fossil fuels in the ground, local generation, microgrids, public utility commissions, PUCs, rationality, reasonableness, regime shifts, Sankey diagram, solar domination, solar energy, solar power, Spaceship Earth, stranded assets, the energy of the people, the green century, the right to know, the tragedy of our present civilization, the value of financial assets, Tony Seba, utility company death spiral, wind energy, wind power, zero carbon | Leave a comment

“What phony op-eds about climate change have in common” (reblog of Tamino’s post)

Open Mind

Senator Sheldon Whitehouse has published an opinion piece in the Columbia Journalism Review about the “web of denial” a number of senators have turned their focus on this week. It’s so worth reading, I reproduce it here:

View original post 995 more words

Posted in adaptation, American Association for the Advancement of Science, AMETSOC, Anthropocene, citizen science, citizenship, climate change, climate disruption, Eaarth, environment, global warming, greenwashing, Hyper Anthropocene, meteorology, Tamino, zero carbon | Leave a comment

Want your democracy back? Take back control of your energy supply

Some progressives lament the loss of Bernie Sanders’ run for President, arguing “we need to get our democracy back.” A necessary step in order to get your democracy back is to take back control of your energy supply. Centralized energy means centralized political power. Residential solar PV power, possibly with energy storage, in individual homes or in local communities, is a political force for good, not only because it is an element of a plan to mitigate greenhouse gas emissions. The energy supply needs to be decentralized. The late Hermann Scheer understood this perfectly, as captured in his talk above, and spells it out in his book. The late Buckminster Fuller alludes to it in his Operating Manual for Spaceship Earth.

Not only does absolute power corrupt absolutely, centralized power corrupts and gives some members of the polity and some politicians undo influence, not only because of the associated monies.

german-solar-paneled-houses
(For larger impact, click on figure, and use browser Back Button to return to blog.)

ciudad-solar-autosostenible
(For larger impact, click on figure, and use browser Back Button to return to blog.)

Posted in adaptation, Anthropocene, Bloomberg New Energy Finance, BNEF, bridge to somewhere, Buckminster Fuller, citizenship, civilization, clean disruption, climate disruption, conservation, consumption, decentralized electric power generation, decentralized energy, demand-side solutions, distributed generation, ecology, economics, efficiency, electricity, electricity markets, energy, energy storage, energy utilities, environment, Epcot, extended supply chains, fossil fuel divestment, global warming, green tech, grid defection, Hermann Scheer, Hyper Anthropocene, investment in wind and solar energy, ISO-NE, Joseph Schumpeter, leaving fossil fuels in the ground, liberal climate deniers, local generation, microgrids, public utility commissions, PUCs, rate of return regulation, rationality, reasonableness, Sankey diagram, solar domination, solar energy, solar power, SolarPV.tv, Spaceship Earth, the energy of the people, the green century, the right to know, zero carbon | 4 Comments

Climate Lamentation: Call & Response

(At the site of the West Roxbury, MA, Spectra/Algonquin explosive pipeline)

Cry out! Cry out! Wail in lamentation for all that climate change has wrought … And will wreak upon the children and grandchildren, upon the poor and the disenfranchised, upon the dreams of families who wanted coastal homes to pass to future generations, upon communities, once thriving, decimated because Nature took their land back, according to Her inexorable Law.

The most foolish response to the fact of climate disruption is continuing to expand fossil fuel infrastructure. It is imperative, and it is morally imperative this be stopped.

(In the pictures below, simply clicking upon them will produce a larger image. Return to the presentation by using your browser Back Button.)

A moral group, a brave group, choose to say No, “You … shall not … pass!”. And, on 29th June 2016, a group, including my beloved Claire, were the latest in hundreds to risk arrest, to use arrest to stop a malevolent intrusive upon a peaceful community, one completely opposed to the act.

WRL NVDA 6-29-16 MaryB, Buddhist nun

WRL NVDA 6-29-16 Rabbi F

WRL NVDA 6-29-16 2

WRL NVDA 6-29-16 funeral 5

WRL NVDA 6-29-16 funeral 4

In the morning, an attempt to stop construction was met with a wall of police presence. This could not be challenged, since touching a police officer is a very serious offense.

WRL NVDA 6-29-16 funeral2

Accordingly, the group retreated to a nearby church to regroup, and to plan.

WRL NVDA 6-29-16 regroup

And in the afternoon …

13501796_860340564109300_9181178893876821754_n

13533018_860340580775965_6751404415778684243_n

WRL NVDA 6-29-16 the hole

My beautiful wife, Claire, stopping construction, ready to get arrested (in red shirt, left):

IMG9544461

Claire’s Support, Andrea, and Claire’s arresting officer:

WRL NVDA 6-29-16 Andrea and my arresting officer

In the holding cell:

WRL NVDA 6-29-16 holding cell

WRL NVDA 6-29-16 holding cell Jane, Rev Kate, Rev Lindsay, Rev Rali, Paul A

WRL NVDA 6-29-16 KathyH Jane, Revs kate, Heather, Lindsay

And upon bail payment and release, later in the day, the brave group:

0629162026

Update, 2016-06-30, 21:04 EDT

An excellent Vimeo video of the entire day:

Update, 2016-07-01

I neglected to mention yesterday, that Karenna Gore, the daughter of former Vice President Al Gore, whose movie, “An Inconvenient Truth” was the first popular introduction to many people regarding the climate crisis, took part and was arrested at this action along with 22 others.

Update, 2016-07-01, 17:47 EDT and 23:40 EDT

Sweeet! News coverage:

http://bcove.me/48xq5i7s

http://www.democracynow.org/2016/6/30/karenna_gore_daughter_of_al_gore

http://www.democracynow.org/2016/6/30/tim_dechristopher_on_climate_activism_in

Posted in Anthropocene, Bloomberg New Energy Finance, BNEF, bollocks, bridge to nowhere, Carbon Worshipers, corporate litigation on damage from fossil fuel emissions, destructive economic development, electricity, electricity markets, energy utilities, environment, ethics, explosive methane, false advertising, fossil fuel divestment, fossil fuels, fracking, global warming | 2 Comments

Massachusetts Bill S.2372 (“An Act to promote energy diversity”), Amendment 1

Ms. Jehlen, Messrs. Keenan, Montigny, Timilty and Joyce, Ms. Creem, Mr. Brady, Ms. L’Italien, Ms. Gobi, Ms. O’Connor Ives, Ms. Chang-Diaz, Messrs. Lewis, Pacheco, Moore and Ross moved that the bill be amended by inserting the following section:-

SECTION X. Section 94A of chapter 164 of the general laws, as appearing in the 2014 official edition, is hereby amended by adding the following paragraph:-

Nothing in this section shall be construed to authorize the department to review and approve contracts for natural gas pipeline capacity filed by electric companies.

This Amendment was adopted by the Massachusetts Senate, today, unanimously.

Posted in Bloomberg New Energy Finance, BNEF, bridge to nowhere, bridge to somewhere, citizenship, clean disruption, decentralized electric power generation, destructive economic development, disruption, electricity, electricity markets, energy utilities, explosive methane, fossil fuel divestment, fossil fuels, gas pipeline leaks, greenhouse gases, investment in wind and solar energy, Joseph Schumpeter, leaving fossil fuels in the ground, Massachusetts, Massachusetts Interfaith Coalition for Climate Action, methane, microgrids, moral leadership, natural gas, pipelines, public utility commissions, PUCs, rate of return regulation, rationality, reasonableness, regulatory capture, solar domination, solar energy, solar power, SolarPV.tv, Spaceship Earth, stranded assets, sustainability, the energy of the people, the green century, the value of financial assets, wind energy, wind power, zero carbon, ``The tide is risin'/And so are we'' | Leave a comment

“The climate talks [in Paris] were a fraud”

Dr James Hansen on The Open Mind.

Signatures won’t save the climate”, writes Danielle Ola at PVTech.

And, despite the good news below, Bloomberg New Energy Finance warns:

The 2⁰C scenario would require much more money. On top of the $7.8 trillion, the world would need to invest another $5.3 trillion in zero-carbon power by 2040 to prevent CO2 in the atmosphere rising above the Intergovernmental Panel on Climate Change’s ‘safe’ limit of 450 parts per million.

Update, 2016-06-30

Investment in renewables required to achieve global climate goals is ‘entirely possible’”. That’s from IRENA (International Renewable Energy Agency).

Update, 2016-07-14

We are apparently heading to the +3C to +4C region, even if COP21 is fully implemented. (See NCAR’s full report.)
Sanderson-ONeill-Tebaldi_fig1_2016-07-14_163539
Sanderson-ONeill-Tebaldi_fig4_2016-07-14_164002

Posted in adaptation, Anthropocene, Carbon Worshipers, citizenship, climate change, climate disruption, Daniel Kahneman, Eaarth, ecology, Ecology Action, environment, environmental law, fossil fuel divestment, fossil fuels, geophysics, global warming, greenhouse gases, Hyper Anthropocene, James Hansen, leaving fossil fuels in the ground, liberal climate deniers, meteorology, mitigation, natural gas, Neill deGrasse Tyson, oceanography, Our Children's Trust, petroleum, pipelines, rationality, reasonableness, science, Spaceship Earth, sustainability, the energy of the people, the green century, the right to be and act stupid, UU Humanists, Warren Buffett, zero carbon | 4 Comments

Still think exponential growth in global Solar adoption is mere extrapolation?

Story here. Graphic:
SolarPower-Europe-1
(Click on image to see a larger figure, and use browser Back Button to return to blog.)

Update, 2016-06-28

Bloomberg: “Solar Power to Grow Sixfold as Sun Becoming Cheapest Resource”. Excerpt:

The amount of electricity generated using solar panels stands to expand as much as sixfold by 2030 as the cost of production falls below competing natural gas and coal-fired plants, according to the International Renewable Energy Agency.

Solar plants using photovoltaic technology could account for 8 percent to 13 percent of global electricity produced in 2030, compared with 1.2 percent at the end of last year, the Abu Dhabi-based industry group said in a report Wednesday. The average cost of electricity from a photovoltaic system is forecast to plunge as much as 59 percent by 2025, making solar the cheapest form of power generation “in an increasing number of cases,” it said.

Renewables are replacing nuclear energy and curbing electricity production from gas and coal in developed areas such as Europe and the U.S., according to Bloomberg New Energy Finance. California’s PG&E Corp. is proposing to close two nuclear reactors as wind and solar costs decline. Even as supply gluts depress coal and gas prices, solar and wind technologies will be the cheapest ways to produce electricity in most parts of the world in the 2030s, New Energy Finance said in a report this month.

Update, 30th June 2016

AnnualInstallationOfSolarPanelsInUSA
(Click on image to see a larger figure, and use browser Back Button to return to blog.)

Solar panel installations in the United States in 2016 will be 94% higher than they were in 2015.

Update, 3rd July 2016

Past references on this blog to the same subject matter:

  1. Tony Seba’s latest
  2. The storage necessity myth — how to choreograph high renewables electricity systems
  3. Disruption to global electricity production during the next 25 years
  4. Five seismic shifts to shake global electricity over the next 25 years
  5. The turning point: New hope for the climate
  6. How cheap can solar get?
  7. Worldwide growth of photovoltaics
  8. Pricing sunshine
  9. Solar industry data
  10. OilPrice.com attempts to dispute the continued dominance of Swanson’s law, and, judging by the comments, falls short of the mark. In fact, it focuses entirely upon solar cell modules and fails to note, apparently based upon 2005 data, that the trend in total system installations is also exponentially increasing
  11. The late Hermann Scheer at Google. See also his book (also available on Amazon, and an older but similar one by Buckminster Fuller.
Posted in adaptation, Anthropocene, Bloomberg New Energy Finance, BNEF, Buckminster Fuller, clean disruption, decentralized electric power generation, decentralized energy, demand-side solutions, destructive economic development, disruption, distributed generation, efficiency, electricity, electricity markets, exponential growth, fossil fuel divestment, green tech, grid defection, Hermann Scheer, Hyper Anthropocene, investing, investment in wind and solar energy, Joseph Schumpeter, leaving fossil fuels in the ground, local generation, microgrids, rate of return regulation, rationality, Ray Kurzweil, reasonableness, resiliency, RevoluSun, Sankey diagram, solar domination, solar energy, solar power, SolarPV.tv, Spaceship Earth, SunPower, the energy of the people, the green century, the value of financial assets, Tony Seba, zero carbon | 2 Comments

“I need to wake up”

Now, more than ever.


(The above was published in September 2015.)

Posted in adaptation, Antarctica, Anthropocene, Arctic, carbon dioxide, citizenship, civilization, climate, climate change, climate disruption, decentralized electric power generation, decentralized energy, denial, destructive economic development, disruption, distributed generation, ecology, Ecology Action, economics, environment, evidence, feed-in tariff, fossil fuel divestment, fossil fuels, gas pipeline leaks, geophysics, global warming, greenhouse gases, grid defection, Hyper Anthropocene, investment in wind and solar energy, James Hansen, leaving fossil fuels in the ground, meteorology, oceanography, physics, pipelines, politics, Principles of Planetary Climate, rationality, reasonableness, science, sea level rise, solar domination, solar energy, solar power, SolarPV.tv, Spaceship Earth, Stefan Rahmstorf, Tamino, temporal myopia, the energy of the people, the green century, the problem of evil, the right to be and act stupid, the right to know, the stack of lies, the tragedy of our present civilization, Unitarian Universalism, utility company death spiral, UU Humanists, WAIS, Wally Broecker, wind energy, wind power, zero carbon | 1 Comment

Elegy for the Arctic

And for the Antarctic
antarctica_oli_2015061

Hat tip to Climate Denial Crock of the Week.

Posted in Antarctica, Arctic, carbon dioxide, climate change, climate disruption, environment, glaciers, glaciology, global warming, greenhouse gases, Hyper Anthropocene, ice sheet dynamics | Leave a comment

SolarIMPULSE: The Atlantic crossing

Controls panel monitoring.

Best of luck to pilot Bertrand Piccard and the entire SolarIMPULSE team!

http://www.solarimpulse.com/widgets/rtw_wrapup/1

http://www.solarimpulse.com/widgets/energy

http://www.solarimpulse.com/widgets/aircraft

http://www.solarimpulse.com/widgets/instruments/1

Update, 1253 EDT, 20th June 2016

WindChange_SI_2016-06-20_125354
(Click on image for a larger figure, and use browser Back Button to return to blog.)

Update, 1407 EDT, 20th June 2016

In addition to winds, some difficulties with Oxygen flow, although Bertrand Piccard is using a backup, and he reports there are clouds to the southeast.
Clouds_SI_2016-06-20_140902

Update, 1748 EDT, 20th June 2016

The weather pattern over the Atlantic has changed, and there is a low pressure “nor’easter” moving up the coast. I have not followed the flight consistently, but it looks like SolarIMPULSE is being apparently being diverted to a landing in Nova Scotia. It’s possible that they may try to skirt the weather and try to push on. Not clear at the moment what they intend. They probably are avoiding a premature commitment to an end. They can retract their request for landing clearance to Moncton Center later.

SolarIMPULSE has cancelled a request for approach and is pushing on into the night, continuing its crossing of the Atlantic.

Weather at 850 hPa:
850hPa_weather_2016-06-20_175122

At 1000 hPa:
weather_2016-06-20_175034

At surface:
surface_weather_2016-06-20_175255

And SolarIMPULSE is presently flying in clouds.

Approximate current location:
850hPa_weather_2016-06-20_175122_annotated
2016-06-20_182018

Update, 2016-06-20, 20:16 EDT

SolarIMPULSE is continuing its trans-Atlantic flight, dodging developing weather to its southwest. It is being acquired by Gandar, Newfoundland. It’s present location:
IntoTheNightSouthOfNewfoundland_2016-06-20_201740

Update, 2016-06-21, 11:06 EDT

Through the night! Now east of St John’s, Newfoundland, and south of Greenland:
SolarIMPULSE_2016-06-21_110351

Update, 2016-06-23, 01:38 EDT

SolarIMPULSE arrives in Seville, Spain!
SolarIMPULSE_arrives_in_Seville

Posted in adaptation, Anthropocene, Bloomberg New Energy Finance, clean disruption, decentralized electric power generation, efficiency, electricity, Hermann Scheer, solar domination, solar energy, solar power, SolarPV.tv, SunPower, zero carbon | Leave a comment

Plunging costs for Solar PV

Offered without comment. Hat tip to Bloomberg and to the Massachusetts Clean Energy Center.

UtilityScaleSolarPVTotalInstalledCosts2009-2025
(Click on image to see a larger figure. Use your browser Back Button to return to blog.)

Posted in adaptation, Bloomberg, Bloomberg New Energy Finance, BNEF, Buckminster Fuller, clean disruption, decentralized electric power generation, decentralized energy, destructive economic development, distributed generation, electricity markets, energy, energy utilities, fossil fuel divestment, green tech, grid defection, Hermann Scheer, Hyper Anthropocene, investing, investment in wind and solar energy, Massachusetts Clean Energy Center, rate of return regulation, rationality, reasonableness, regime shifts, regulatory capture, Sankey diagram, solar domination, solar energy, solar power, SolarPV.tv, Spaceship Earth, stranded assets, the value of financial assets, Tony Seba, utility company death spiral, wind energy, wind power, zero carbon | Leave a comment

Stephen Chu: `Put a global price on Carbon’

In the Spring 2016 edition of Catalyst, a periodical of the Union of Concerned Scientists, Nobel laureate and former U.S. DOE head Professor Stephen Chu offers a suggestion on what the world should do after the COP21 meeting in Paris. Below is an excerpt, with emphasis added, and slightly edited for completeness where noted:

More than half of [the allotment of global Carbon emissions needed to have a 50% chance of remaining below the 2°C target] has already been used up since the beginning of the Industrial Revolution and, at our current emissions rate, the remainder will be gone in about 30 years. Clearly, the remaining carbon budget is a precious resource, but cap-and-trade allocations start from existing levels of emissions. It is prima facie unfair to allow developed countries to pollute more because they were historically the biggest polluters.

A global carbon tax avoids the intractable problem of how to allocate carbon emissions credits between developed and developing countries, and levies the highest taxes on the biggest emitters. If countries are unwilling to levy a cost on carbon, the playing field can be leveled with suitable border tariffs on goods imported into participating countries. In addition, the wealthiest countries still need to help less developed countries in this transition.

Posted in adaptation, Anthropocene, carbon dioxide, Carbon Tax, Carbon Worshipers, citizenship, civilization, climate disruption, consumption, ecology, economics, energy, environment, environmental law, fossil fuel divestment, fossil fuels, global warming, greenhouse gases, Hyper Anthropocene, leaving fossil fuels in the ground, politics, rationality, reasonableness, the green century, zero carbon | Leave a comment

China as a zero Carbon energy powerhouse

solar-energy-chart-IEA-3
(Click on image for a larger figure, and use your browser Back Button to return to blog.)

Figure courtesy of the International Energy Agency.


cumulative_emissions_2011_2016-06-16_105547
(Click on image for a larger figure, and use your browser Back Button to return to blog.)

Posted in adaptation, Anthropocene, Bloomberg New Energy Finance, BNEF, bridge to somewhere, Buckminster Fuller, clean disruption, decentralized electric power generation, decentralized energy, distributed generation, economics, electricity, energy, engineering, environment, fossil fuel divestment, green tech, Hermann Scheer, Hyper Anthropocene, investing, investment in wind and solar energy, leaving fossil fuels in the ground, public utility commissions, PUCs, regime shifts, solar domination, solar energy, solar power, SolarPV.tv, Spaceship Earth, the energy of the people, the green century, the right to be and act stupid, the right to know, the tragedy of our present civilization, the value of financial assets, Tony Seba, zero carbon | 2 Comments

data.table

R provides a helpful data structure called the “data frame” that gives the user an intuitive way to organize, view, and access data.  Many of the functions that you would us…

Source: Intro to The data.table Package

Posted in big data, data science, engineering, numerical analysis, numerical software, numerics, open source scientific software, R, smart data, statistics | Leave a comment

France, and Mathematics

Cédric Villani, does Mathematics.

“Problems worthy of attack, prove their worth by hitting back.” — Piet Hein

Posted in abstraction, Google, mathematics, mathematics education, maths, networks, Pagerank, percolation theory, point pattern analysis, probability, rationality, reasonableness, stochastic algorithms | Leave a comment

On Smart Data

One of the things I find surprising, if not astonishing, is that in the rush to embrace Big Data, a lot of learning and statistical technique has been left apparently discarded along the way. I’m hardly the first to point this out. Moreover, there are remedies available. Still, there are books on predictive analytics published which, while they collect a set of interesting ad hoc techniques for rapid inference together, leave out a lot of traditional techniques and wise concerns. Many of the major software plexes, like Weka, Spark MLib, or the map-reduce framework with its strong algorithmic constraints, facilitate the organization of large datasets and their retrieval, but the standard practices necessarily omit things like subsampling, and assessing what your real sample size is. To be crude, 100 tonnes of crap is still crap, and a billion replicas of exactly the same record don’t give you any more information than what’s in the same record.

Fundamentally there seems to be this idea that traditional statistical methods are too slow for the world of Big Data. I think that’s meant in two different ways. The first, which almost everyone addresses when the matter is discussed, is that the data set sizes or the rates of streaming are so large it’s not possible to apply batch-oriented or heavy computation to them. The second, which seldom gets mentioned in my experience, is that there’s an emphasis by organizations on rapidly producing apparent results, and, so, it’s perceived that deep thinking or care in sampling is not consistent with the business mission (*). I say “apparent results” because often there are only poor ways to tell if results are adequate. Itemset methods, sometimes called association rules, which I have used in an application, have a number of frequentist statistics offered which are used for diagnosis. One, for instance, is called Confidence and is a very poor man’s estimate of a conditional probability. It’s recognized that it has limitations, but to treat the finding limitations as if they are a research result is, in my opinion, to feign ignorance. So, to me, the rapid production of apparent results is simply looking like keeping busy, without a quantitative way of knowing. And “the customers seem to continue to be happy” or “sales keep going up” which, while important of course, don’t necessary have any causal connection to what’s being done.

Facts are, of course, that there are lots of ways traditional statistics can inform efforts involving large datasets. Moreover, there are, indeed, techniques which have been known since the 1960s for keeping up with the onslaught of a large data stream, these have been greatly improved, and they are very much in use by people who know how to use them.

And I suspect that’s another thing: Most of the traditional techniques for doing prediction and inference on streams, namely dynamic linear models, state-space methods, and dynamic generalized linear models all use more mathematics, specifically, numerical linear algebra and basic multivariable calculus to do what they do. And the population of developers and managers and, even, engineers eschew use of these methods whenever they can, because they are perceived to be hard. Instead, things like inference based upon ad hoc methods of locally sensitive hashing are used because it is “standard practice”, and generalizations of clustering methods, without even inquiring if the topology of the problem admits use of these. Sure, I can see these methods have their place, but it’s not like these are the only way things can be done.

Rather than some kind of leap of faith that more and more data can make up for things like poor statistical power, I think smart organizations realize that sampling matters, and traditional critiques of business processes are important. Accordingly, I’m not interested in Big Data, I’m interested in Smart Data, no matter what its size. I think anyone who cares about what their results mean should be interested in Smart Data, too.

In addition to simply representing good practice, Smart Data techniques do things easier (in the big picture sense) than do ad hoc collections of ad hoc techniques, no matter how many times they are cross-validated. For instance, predicting consumer behavior on hypothetical products, or products they have never experienced, or products no one has ever experienced, is not something which extrapolations of existing evidence compendia can ever dream of doing. There needs to be a well-wrought model in order to do that. And the model needs to be evidence-based, too. In other words, if there’s no training data, or truth data to score it, simply observations, many of the present Big Data methods are hopeless.


(*) This is exemplified by the many competitions or hackathons where tough problems are expected to be solved in a short time by adversarial teams. Sure, speed is sometimes necessary, but does anyone seriously expect every business can be run that way and last? “Internet time” is and always was ridiculous. At least, they’ll lose their people. They could go bankrupt after making a big mistake that was not noticed due to the rush. Flexibility, yes. But careful competence akin to skunkworks, definitely. Death marches can’t work, nor can projects which feature any two or more of their characteristics: wishful thinking, escalation of commitment, optimism bias, and planning fallacy.

Posted in Akaike Information Criterion, Bayes, Bayesian, Bayesian inversion, big data, bigmemory package for R, changepoint detection, data science, data streams, dlm package, dynamic generalized linear models, dynamic linear models, dynamical systems, Generalize Additive Models, generalized linear models, information theoretic statistics, Kalman filter, linear algebra, logistic regression, machine learning, Markov Chain Monte Carlo, mathematics, mathematics education, maths, maximum likelihood, MCMC, Monte Carlo Statistical Methods, multivariate statistics, numerical analysis, numerical software, numerics, quantitative biology, quantitative ecology, rationality, reasonableness, sampling, smart data, state-space models, statistical dependence, statistics, the right to know, time series | Leave a comment

“Catching long tail distribution” (Ted Dunning)

One of the best presentations on what can happen if someone takes a naive approach to network data. It also highlights what is, to my mind, the greatly underappreciated t-distribution, which is typically only used in connection with frequentist Student t-tests, but serves as a generator someplace between the Gaussian and the crazy Cauchy distribution. Also relevant is the Lévy flight which has significance in biology. (See also.)

The message is that a combination of multiple paths, sampling rate changes, and a glitch on one of the paths can make an event appear to occur where there is none.

Posted in Cauchy distribution, complex systems, data science, Lévy flights, leptokurtic, mathematics, maths, networks, physics, population biology, population dynamics, regime shifts, sampling, statistics, Student t distribution, time series | Leave a comment