Category Archives: state-space models

Repaired R code for Markov spatial simulation of hurricane tracks from historical trajectories

I’m currently studying random walk and diffusion processes and their connections with random fields. I’m interested in this because at the core of dynamic linear models, Kalman filters, and state-space methods there is a random walk in a parameter space. … Continue reading

Posted in American Meteorological Association, American Statistical Association, AMETSOC, Arthur Charpentier, atmosphere, diffusion, diffusion processes, dynamic linear models, dynamical systems, environment, geophysics, hurricanes, Kalman filter, Kerry Emanuel, Lévy flights, Lorenz, Markov chain random fields, mathematics, mathematics education, maths, MCMC, mesh models, meteorological models, meteorology, model-free forecasting, Monte Carlo Statistical Methods, numerical analysis, numerical software, oceanography, open data, open source scientific software, physics, random walk processes, random walks, science, spatial statistics, state-space models, statistical dependence, statistics, stochastic algorithms, stochastics, time series | Leave a comment

A model of an electrical grid: A vision

Many people seem to view the electrical grid of the future being much like the present one. I think a lot about networks, because of my job. And I especially think a lot about network topologies, although primarily concerning the … Continue reading

Posted in abstraction, American Meteorological Association, anomaly detection, Anthropocene, Bloomberg New Energy Finance, BNEF, Boston, bridge to somewhere, Buckminster Fuller, Canettes Blues Band, clean disruption, climate business, climate economics, complex systems, corporate supply chains, decentralized electric power generation, decentralized energy, demand-side solutions, differential equations, distributed generation, efficiency, EIA, electricity, electricity markets, energy, energy reduction, energy storage, energy utilities, engineering, extended supply chains, green tech, grid defection, Hermann Scheer, Hyper Anthropocene, investment in wind and solar energy, ISO-NE, Kalman filter, kriging, Lawrence Berkeley National Laboratory, leaving fossil fuels in the ground, Lenny Smith, local generation, marginal energy sources, Massachusetts Clean Energy Center, Mathematics and Climate Research Network, mesh models, meteorology, microgrids, networks, New England, New York State, open data, organizational failures, pipelines, planning, prediction markets, public utility commissions, PUCs, rate of return regulation, rationality, reason, reasonableness, regime shifts, regulatory capture, resiliency, risk, Sankey diagram, smart data, solar domination, solar energy, solar power, Spaceship Earth, spatial statistics, state-space models, statistical dependence, statistics, stochastic algorithms, stochastics, stranded assets, supply chains, sustainability, the energy of the people, the green century, the value of financial assets, thermodynamics, time series, Tony Seba, utility company death spiral, wave equations, wind energy, wind power, zero carbon | Leave a comment

“Stochastic Parameterization: Towards a new view of weather and climate models”

Judith Berner, Ulrich Achatz, Lauriane Batté, Lisa Bengtsson, Alvaro De La Cámara, Hannah M. Christensen, Matteo Colangeli, Danielle R. B. Coleman, Daan Crommelin, Stamen I. Dolaptchiev, Christian L.E. Franzke, Petra Friederichs, Peter Imkeller, Heikki Järvinen, Stephan Juricke, Vassili Kitsios, François … Continue reading

Posted in biology, climate models, complex systems, convergent cross-mapping, data science, dynamical systems, ecology, Ethan Deyle, Floris Takens, George Sughihara, Hao Ye, likelihood-free, Lorenz, mathematics, meteorological models, model-free forecasting, physics, population biology, population dynamics, quantitative biology, quantitative ecology, Scripps Institution of Oceanography, state-space models, statistical dependence, statistics, stochastic algorithms, stochastic search, stochastics, Takens embedding theorem, time series, Victor Brovkin | 4 Comments

On Smart Data

One of the things I find surprising, if not astonishing, is that in the rush to embrace Big Data, a lot of learning and statistical technique has been left apparently discarded along the way. I’m hardly the first to point … Continue reading

Posted in Akaike Information Criterion, Bayes, Bayesian, Bayesian inversion, big data, bigmemory package for R, changepoint detection, data science, data streams, dlm package, dynamic generalized linear models, dynamic linear models, dynamical systems, Generalize Additive Models, generalized linear models, information theoretic statistics, Kalman filter, linear algebra, logistic regression, machine learning, Markov Chain Monte Carlo, mathematics, mathematics education, maths, maximum likelihood, MCMC, Monte Carlo Statistical Methods, multivariate statistics, numerical analysis, numerical software, numerics, quantitative biology, quantitative ecology, rationality, reasonableness, sampling, smart data, state-space models, statistical dependence, statistics, the right to know, time series | Leave a comment

Six cases of models

The previous post included an attempt to explain land surface temperatures as estimated by the BEST project using a dynamic linear model including regressions on both quarterly CO2 concentrations and ocean heat content. The idea was to check the explanatory … Continue reading

Posted in AMETSOC, anemic data, Anthropocene, astrophysics, Bayesian, Berkeley Earth Surface Temperature project, BEST, carbon dioxide, climate, climate change, climate data, climate disruption, climate models, dlm package, dynamic linear models, dynamical systems, environment, fossil fuels, geophysics, Giovanni Petris, global warming, greenhouse gases, Hyper Anthropocene, information theoretic statistics, maths, maximum likelihood, meteorology, model comparison, numerical software, Patrizia Campagnoli, Rauch-Tung-Striebel, Sonia Petrone, state-space models, stochastic algorithms, stochastic search, SVD, time series | 1 Comment

On Munshi mush

(Slightly updated on 2016-06-11.) Professor Emeritus Jamal Munshi of Sonoma State University has papers recently cited in science denier circles as evidence that the conventional associations between mean global surface temperature and cumulative carbon emissions are, well, bunk, due to … Continue reading

Posted in Bayes, Bayesian, Berkeley Earth Surface Temperature project, BEST, carbon dioxide, cat1, climate, climate change, climate data, climate education, climate models, convergent cross-mapping, dynamic linear models, ecology, ENSO, environment, Ethan Deyle, evidence, geophysics, George Sughihara, global warming, greenhouse gases, information theoretic statistics, Kalman filter, mathematics, maths, meteorology, model comparison, NOAA, oceanography, prediction, state-space models, statistics, Takens embedding theorem, Techno Utopias, the right to know, theoretical physics, time series, zero carbon | 1 Comment

Cory Lesmeister’s treatment of Simson’s Paradox (at “Fear and Loathing in Data Science”)

(Updated 2016-05-08, to provide reference for plateaus of ML functions in vicinity of MLE.) Simpson’s Paradox is one of those phenomena of data which really give Statistics a substance and a role, beyond the roles it inherits from, say, theoretical … Continue reading

Posted in Akaike Information Criterion, approximate Bayesian computation, Bayes, Bayesian, evidence, Frequentist, games of chance, information theoretic statistics, Kalman filter, likelihood-free, mathematics, maths, maximum likelihood, Monte Carlo Statistical Methods, probabilistic programming, rationality, Rauch-Tung-Striebel, Simpson's Paradox, state-space models, statistical dependence, statistics, stochastics | Leave a comment

high dimension Metropolis-Hastings algorithms

If attempting to simulate from a multivariate standard normal distribution in a large dimension, when starting from the mode of the target, i.e., its mean γ, leaving the mode γis extremely unlikely, given the huge drop between the value of the density at the mode γ and at likely realisations Continue reading

Posted in Bayes, Bayesian, Bayesian inversion, boosting, chance, Christian Robert, computation, ensembles, Gibbs Sampling, James Spall, Jerome Friedman, Markov Chain Monte Carlo, mathematics, maths, MCMC, Monte Carlo Statistical Methods, multivariate statistics, numerical software, numerics, optimization, reasonableness, Robert Schapire, SPSA, state-space models, statistics, stochastic algorithms, stochastic search, stochastics, Yoav Freund | Leave a comment

Generating supports for classification rules in black box regression models

Inspired by the extensive and excellent work in approximate Bayesian computation (see also), especially that done by Professors Christian Robert and colleagues (see also), and Professor Simon Wood (see also), it occurred to me that the complaints regarding lack of … Continue reading

Posted in approximate Bayesian computation, Bayes, Bayesian, Bayesian inversion, generalized linear models, machine learning, numerical analysis, numerical software, probabilistic programming, rationality, reasonableness, state-space models, statistics, stochastic algorithms, stochastic search, stochastics, support of black boxes | Leave a comment

“Causal feedbacks in climate change”

Today I was reviewing and re-reading the nonlinear time series technical literature I have, seeking ideas on how to go about using the statistical ensemble learning technique called “boosting” with them. (See the very nice book, R. E. Schapire, Y. … Continue reading

Posted in Anthropocene, boosting, Carbon Cycle, carbon dioxide, Carbon Worshipers, cat1, climate, climate change, climate data, climate disruption, complex systems, convergent cross-mapping, denial, differential equations, diffusion processes, dynamical systems, ecology, Egbert van Nes, empirical likelihood, ensembles, environment, Ethan Deyle, Floris Takens, forecasting, fossil fuels, geophysics, George Sughihara, global warming, greenhouse gases, Hao Ye, machine learning, Maren Scheffer, mathematics, maths, meteorology, physics, rationality, reasonableness, science, state-space models, Takens embedding theorem, time series, Timothy Lenton, Victor Brovkin | 1 Comment

After the Decade of Dithering, the Deadly Twenties

In a recent post, after reviewing the extreme Arctic warming event of late 2015, Professor John Baez quotes an earlier interview with Dr Gregory Benford, who is arguing for a geoengineering effort to restore the frozen Arctic. I do not … Continue reading

Posted in adaptation, AMOC, Arctic, chance, changepoint detection, climate, climate change, climate disruption, critical slowing down, ecology, engineering, geoengineering, global warming, greenhouse gases, Hyper Anthropocene, ignorance, James Hansen, MIchael Mann, mitigation, oceanography, physics, politics, rationality, reasonableness, regime shifts, science, science education, state-space models, statistics, the right to know, thermohaline circulation, time series | Leave a comment

dynamic linear model applied to sea-level-rise anomalies

I spent much of the data working up a function for level+trend dynamic linear modeling based upon the dlm package by Petris, Petrone, and Campagnoli, while trying some calculations and code for regime shift detection. One of the test cases … Continue reading

Posted in Bayesian, citizen science, climate change, climate data, climate disruption, dynamic linear models, floods, forecasting, Frequentist, global warming, icesheets, information theoretic statistics, Kalman filter, meteorology, open data, sea level rise, state-space models, statistics, time series | Leave a comment

Thoughts on “Regime Shift?”

John Baez at The Azimuth Project opened a discussion on the recent paper by Reid, et al Philip C. Reid et al, Global impacts of the 1980s regime shift on the Earth’s climate and systems, Global Change Biology, 2015. I … Continue reading

Posted in Bayesian, changepoint detection, climate change, climate disruption, climate models, dynamic linear models, ecology, ensembles, environment, global warming, population biology, Rauch-Tung-Striebel, regime shifts, state-space models, stochastic algorithms, time series | Leave a comment

Southern Oscillation (SOI) correlated with Outgoing Longwave Radiation (OLR)

To the climate community this is nothing at all new, but I spotted these time series today and thought they would make a nice exhibit on how something people have direct control over, greenhouse gas emissions, affect a “teleconnection mechanism” … Continue reading

Posted in AMETSOC, bifurcations, carbon dioxide, climate, climate change, climate disruption, climate models, Dan Satterfield, differential equations, dynamic linear models, dynamical systems, ENSO, environment, forecasting, generalized linear models, geophysics, global warming, greenhouse gases, IPCC, Mathematica, mathematics, maths, meteorology, NCAR, NOAA, numerical software, oceanography, open data, physics, population biology, Principles of Planetary Climate, rationality, reasonableness, science, Spaceship Earth, state-space models, thermodynamics, time series | Leave a comment

Southern New England Meteorology Conference, 24th October 2015

I attending the 2015 edition of the Southern New England Meteorology Conference in Milton, MA, near the Blue Hill, and its Blue Hill Climatological Observatory, of which I am a member as we as of the American Meteorological Society. I … Continue reading

Posted in Anthropocene, capricious gods, climate, Dan Satterfield, dynamical systems, ensembles, ENSO, environment, floods, forecasting, geophysics, Hyper Anthropocene, information theoretic statistics, mesh models, meteorology, model comparison, NCAR, NOAA, nor'easters, oceanography, probability, science, spatial statistics, state-space models, statistics, stochastic algorithms, stochastic search, stochastics, time series | 1 Comment

STUFF IN PROGRESS

It’s a good time to reconnoiter and review the things I have in progress and are planned, both as a teaser, and as a promise. I am currently working the following technical projects, entirely on my personal time outside of … Continue reading

Posted in numerical analysis, planning, R, rationality, reasonableness, state-space models, statistics | Leave a comment

Your future: Antarctica, in detail

Climate and geophysical accuracy demands fine modeling grids, and very large supercomputers. The best and biggest supercomputers have not been available for climate work, until recently. Watch how results differ if fine meshes and big supercomputers are used. Why haven’t … Continue reading

Posted in Antarctica, Anthropocene, bridge to nowhere, climate, climate change, climate disruption, climate zombies, disingenuity, ecology, ensembles, forecasting, geophysics, global warming, Hyper Anthropocene, ignorance, IPCC, Lawrence Berkeley National Laboratory, LBNL, living shorelines, mathematics, mathematics education, maths, mesh models, meteorology, multivariate statistics, numerical software, optimization, physics, rationality, reasonableness, risk, science, science education, sea level rise, spatial statistics, state-space models, statistics, stochastic algorithms, stochastics, supercomputers, temporal myopia, the right to know, thermodynamics, time series, University of California Berkeley, WAIS | Leave a comment

SCIENCE OF DOOM takes on assessing zero Carbon power and a zero Carbon grid

Updated, 2127 EDT, 10th August 2015 The blog, Science of Doom, has taken on a new thread discussing the technical feasibilities and problems associated with building out zero Carbon energy in the context of an electric grid. As such, it … Continue reading

Posted in adaptation, Anthropocene, clean disruption, climate data, climate disruption, conservation, consumption, decentralized electric power generation, decentralized energy, demand-side solutions, destructive economic development, dynamic linear models, dynamical systems, economics, efficiency, energy, energy reduction, engineering, environment, exponential growth, forecasting, fossil fuel divestment, global warming, Hyper Anthropocene, investing, investment in wind and solar energy, microgrids, open data, optimization, prediction, rationality, reasonableness, risk, solar power, state-space models, stochastics, sustainability, the right to know, time series, wind power, Wordpress, zero carbon | 4 Comments

Comprehensive and compact tutorial on Petris’ DLM package in R; with an update about Helske’s KFAS

A blogger named Lalas produced on Quantitative Thoughts a very comprehensive and compact tutorial on the R package dlm by Petris. I use dlm a lot. Unfortunately, Lalas does not give details on how the SVD is used. They do … Continue reading

Posted in Bayes, Bayesian, dynamic linear models, dynamical systems, forecasting, Kalman filter, mathematics, maths, multivariate statistics, numerical software, open source scientific software, prediction, R, Rauch-Tung-Striebel, state-space models, statistics, stochastic algorithms, SVD, time series | Leave a comment

Why decentralized electrical power has to win, no matter what Elon Musk says, and utilities are doomed

Image | Posted on by | 3 Comments

On the Climate Club

But if the other advanced nations had a stick — a tariff of 4 percent on the imports from countries not in the “climate club” — the cost-benefit calculation for the United States would flip. Not participating in the club … Continue reading

Quote | Posted on by | 2 Comments

The CWSLab workflow tool: an experiment in community code development

Originally posted on Dr Climate:
Give anyone working in the climate sciences half a chance and they’ll chew your ear off about CMIP5. It’s the largest climate modelling project ever conducted and formed the basis for much of the IPCC…

Posted in climate, climate education, climate models, computation, differential equations, dynamical systems, environment, forecasting, geophysics, global warming, IPCC, mathematics, mathematics education, maths, meteorology, model comparison, NCAR, oceanography, open source scientific software, physics, Principles of Planetary Climate, Python 3, rationality, reasonableness, science, science education, state-space models, statistics, time series, transparency | Leave a comment

A promise forward …

I’ve made a commitment at Google Plus to detail the implications of underestimated rainfall in terms of precipitation risk. I’m planning to tie this up with my informal work on the Town of Sharon’s water supply, in Sharon, MA.

Posted in chance, citizenship, ecology, ENSO, environment, floods, forecasting, geophysics, meteorology, politics, precipitation, rationality, state-space models, statistics, stochastics | Leave a comment

Dynamic Linear Models package, dlmodeler

I’m checking out the dlmodeler package in R for a work project. It is accompanied by textbooks, G. Petris, S. Petrone, P. Campagnoli, Dynamic Linear Models with R, Springer, 2009 and J. Durbin, S. J. Koopman, Time Series Analysis by … Continue reading

Posted in Bayesian, geophysics, mathematics, maths, oceanography, open source scientific software, Python 3, R, science, sea level rise, state-space models, statistics, stochastic algorithms, time series | Leave a comment