Category Archives: stochastics

The Johnson-Lindenstrauss Lemma, and the paradoxical power of random linear operators. Part 1.

Updated, 2018-12-04 I’ll be discussing the ramifications of: William B. Johnson and Joram Lindenstrauss, “Extensions of Lipschitz mappings into a Hilbert space, Contemporary Mathematics, 26:189–206, 1984. for several posts here. Some introduction and links to proofs and explications will be … Continue reading

Posted in clustering, data science, dimension reduction, information theoretic statistics, Johnson-Lindenstrauss Lemma, k-NN, Locality Sensitive Hashing, mathematics, maths, multivariate statistics, non-parametric model, numerical algorithms, numerical linear algebra, point pattern analysis, random projections, recommender systems, science, stochastic algorithms, stochastics, subspace projection methods | 1 Comment

Repaired R code for Markov spatial simulation of hurricane tracks from historical trajectories

(Slight update, 28th June 2020.) I’m currently studying random walk and diffusion processes and their connections with random fields. I’m interested in this because at the core of dynamic linear models, Kalman filters, and state-space methods there is a random … Continue reading

Posted in American Meteorological Association, American Statistical Association, AMETSOC, Arthur Charpentier, atmosphere, diffusion, diffusion processes, dynamic linear models, dynamical systems, environment, geophysics, hurricanes, Kalman filter, Kerry Emanuel, Lévy flights, Lorenz, Markov chain random fields, mathematics, mathematics education, maths, MCMC, mesh models, meteorological models, meteorology, model-free forecasting, Monte Carlo Statistical Methods, numerical analysis, numerical software, oceanography, open data, open source scientific software, physics, random walk processes, random walks, science, spatial statistics, state-space models, statistical dependence, statistics, stochastic algorithms, stochastics, time series | 1 Comment

A model of an electrical grid: A vision

Many people seem to view the electrical grid of the future being much like the present one. I think a lot about networks, because of my job. And I especially think a lot about network topologies, although primarily concerning the … Continue reading

Posted in abstraction, American Meteorological Association, anomaly detection, Anthropocene, Bloomberg New Energy Finance, BNEF, Boston, bridge to somewhere, Buckminster Fuller, Canettes Blues Band, clean disruption, climate business, climate economics, complex systems, corporate supply chains, decentralized electric power generation, decentralized energy, demand-side solutions, differential equations, distributed generation, efficiency, EIA, electricity, electricity markets, energy, energy reduction, energy storage, energy utilities, engineering, extended supply chains, green tech, grid defection, Hermann Scheer, Hyper Anthropocene, investment in wind and solar energy, ISO-NE, Kalman filter, kriging, Lawrence Berkeley National Laboratory, leaving fossil fuels in the ground, Lenny Smith, local generation, marginal energy sources, Massachusetts Clean Energy Center, Mathematics and Climate Research Network, mesh models, meteorology, microgrids, networks, New England, New York State, open data, organizational failures, pipelines, planning, prediction markets, public utility commissions, PUCs, rate of return regulation, rationality, reason, reasonableness, regime shifts, regulatory capture, resiliency, risk, Sankey diagram, smart data, solar domination, solar energy, solar power, Spaceship Earth, spatial statistics, state-space models, statistical dependence, statistics, stochastic algorithms, stochastics, stranded assets, supply chains, sustainability, the energy of the people, the green century, the value of financial assets, thermodynamics, time series, Tony Seba, utility company death spiral, wave equations, wind energy, wind power, zero carbon | Leave a comment

“Stochastic Parameterization: Towards a new view of weather and climate models”

Judith Berner, Ulrich Achatz, Lauriane Batté, Lisa Bengtsson, Alvaro De La Cámara, Hannah M. Christensen, Matteo Colangeli, Danielle R. B. Coleman, Daan Crommelin, Stamen I. Dolaptchiev, Christian L.E. Franzke, Petra Friederichs, Peter Imkeller, Heikki Järvinen, Stephan Juricke, Vassili Kitsios, François … Continue reading

Posted in biology, climate models, complex systems, convergent cross-mapping, data science, dynamical systems, ecology, Ethan Deyle, Floris Takens, George Sughihara, Hao Ye, likelihood-free, Lorenz, mathematics, meteorological models, model-free forecasting, physics, population biology, population dynamics, quantitative biology, quantitative ecology, Scripps Institution of Oceanography, state-space models, statistical dependence, statistics, stochastic algorithms, stochastic search, stochastics, Takens embedding theorem, time series, Victor Brovkin | 4 Comments

Cory Lesmeister’s treatment of Simson’s Paradox (at “Fear and Loathing in Data Science”)

(Updated 2016-05-08, to provide reference for plateaus of ML functions in vicinity of MLE.) Simpson’s Paradox is one of those phenomena of data which really give Statistics a substance and a role, beyond the roles it inherits from, say, theoretical … Continue reading

Posted in Akaike Information Criterion, approximate Bayesian computation, Bayes, Bayesian, evidence, Frequentist, games of chance, information theoretic statistics, Kalman filter, likelihood-free, mathematics, maths, maximum likelihood, Monte Carlo Statistical Methods, probabilistic programming, rationality, Rauch-Tung-Striebel, Simpson's Paradox, state-space models, statistical dependence, statistics, stochastics | Leave a comment

“Solar power is contagious. These maps show how it spreads.” (from Vox)

Brad Plumer at Vox writes on take-up patterns of rooftop solar based upon a large dataset from SolarCity. The full article is available at the SolarCity site. Mr Plumer combines it with a report on other studies of solar adoption … Continue reading

Posted in Bloomberg New Energy Finance, BNEF, business, clean disruption, decentralized electric power generation, decentralized energy, demand-side solutions, destructive economic development, diffusion, diffusion processes, disruption, distributed generation, economics, electricity markets, energy, energy utilities, exponential growth, grid defection, investment in wind and solar energy, local generation, Peter Diggle, point pattern analysis, public utility commissions, PUCs, rate of return regulation, regulatory capture, sociology, solar domination, solar energy, solar power, SolarPV.tv, spatial statistics, statistics, stochastics, the energy of the people, the green century, utility company death spiral, zero carbon | Leave a comment

“Lucky d20” (by Tamino, with my reblogging comments)

Originally posted on Open Mind:
What with talk of killer heat waves, droughts, floods, etc. etc., this blog tends to get pretty serious. When it does, we don’t deal with happy prospects, but with the danger of worldwide catastrophe. But…

Posted in Bayes, Bayesian, card decks, card draws, card games, chance, D&D, Dungeons and Dragons, games of chance, mathematics, maths, Monte Carlo Statistical Methods, probability, statistical dependence, statistics, stochastic algorithms, stochastics, Wizards of the Coast | Leave a comment

p-values and hypothesis tests: the Bayesian(s) rule

The American Statistical Association of which I am a longtime member issued an important statement today which will hopefully move statistical practice in engineering and especially in the sciences away from the misleading practice of using p-values and hypothesis tests. … Continue reading

Posted in approximate Bayesian computation, arXiv, Bayes, Bayesian, Bayesian inversion, bollocks, Christian Robert, climate, complex systems, data science, Frequentist, information theoretic statistics, likelihood-free, Markov Chain Monte Carlo, MCMC, Monte Carlo Statistical Methods, population biology, rationality, reasonableness, science, scientific publishing, statistical dependence, statistics, stochastics, Student t distribution | Leave a comment

patents disincentivize progress

Very interesting.

Posted in business, data centers, destructive economic development, economics, high frequency trading, humanism, Hyper Anthropocene, mathematics, maths, rationality, reasonableness, risk, selfishness, stochastics | 1 Comment

In celebration of the unification of Gravity with the other fundamental forces …

It was announced today that LIGO found definitive evidence of gravity waves last September. Excellent descriptions at IOP, including a beginners guide. No better way to celebrate than with the Canettes Blues Band, from CERN: A group of physicists who … Continue reading

Posted in Bulletin of the Atomic Scientists, Canettes Blues Band, geophysics, gravity waves, LHC, LIGO, physics, quantum, rationality, reasonableness, statistics, stochastics, theoretical physics | Leave a comment

high dimension Metropolis-Hastings algorithms

If attempting to simulate from a multivariate standard normal distribution in a large dimension, when starting from the mode of the target, i.e., its mean γ, leaving the mode γis extremely unlikely, given the huge drop between the value of the density at the mode γ and at likely realisations Continue reading

Posted in Bayes, Bayesian, Bayesian inversion, boosting, chance, Christian Robert, computation, ensembles, Gibbs Sampling, James Spall, Jerome Friedman, Markov Chain Monte Carlo, mathematics, maths, MCMC, Monte Carlo Statistical Methods, multivariate statistics, numerical software, numerics, optimization, reasonableness, Robert Schapire, SPSA, state-space models, statistics, stochastic algorithms, stochastic search, stochastics, Yoav Freund | Leave a comment

Generating supports for classification rules in black box regression models

Inspired by the extensive and excellent work in approximate Bayesian computation (see also), especially that done by Professors Christian Robert and colleagues (see also), and Professor Simon Wood (see also), it occurred to me that the complaints regarding lack of … Continue reading

Posted in approximate Bayesian computation, Bayes, Bayesian, Bayesian inversion, generalized linear models, machine learning, numerical analysis, numerical software, probabilistic programming, rationality, reasonableness, state-space models, statistics, stochastic algorithms, stochastic search, stochastics, support of black boxes | Leave a comment

All I do is complain, complain …

I was reviewing a presentation given as part of a short course in the machine learning genre today, and happened across the following two bullets, under the heading “Strictly Stationary Processes”: Predicting a time series is possible if and only … Continue reading

Posted in bifurcations, chaos, citizen science, convergent cross-mapping, dynamic linear models, dynamical systems, engineering, Floris Takens, generalized linear models, geophysics, George Sughihara, ignorance, Lenny Smith, Lorenz, mathematics, maths, meteorology, prediction, probability, rationality, reasonableness, statistics, stochastic algorithms, stochastic search, stochastics, Takens embedding theorem, the right to know, time series | 1 Comment

Pale Blue Dot

Compassion, yes. Love, no.

Posted in astronomy, astrophysics, atheism, Bill Maher, Bill Nye, bollocks, Boston Ethical Society, Carl Sagan, citizenship, civilization, compassion, ecology, geophysics, humanism, NASA, physical materialism, physics, population biology, Sankey diagram, Spaceship Earth, statistics, stochastics | 1 Comment

El Nino In A Can – Dan’s Wild Wild Science Journal – AGU Blogosphere

Click the image above to see a video from the GFDL CM2.6 climate model. This is NOT this year’s El Nino. When you start a climate model in which the ocean and the land and atmosphere can inte… Source: El … Continue reading

Posted in AMETSOC, astrophysics, climate, climate change, climate models, computation, Dan Satterfield, differential equations, diffusion, diffusion processes, dynamical systems, ENSO, environment, forecasting, geophysics, global warming, Hyper Anthropocene, Kerry Emanuel, mathematics, maths, mesh models, meteorology, model comparison, NASA, NCAR, NOAA, numerical analysis, oceanography, physics, Principles of Planetary Climate, rationality, Ray Pierrehumbert, reasonableness, science, Spaceship Earth, stochastics, supercomputers, the right to know, thermodynamics, time series | Leave a comment

What the future of energy everywhere looks like

What will the energy landscape look like after utility companies are either dead, dying, or revert to a tiny portion of their territory? Silicon Valley CCE Partnership gives us all a clue. It’s been described in the San Francisco Chronicle, … Continue reading

Posted in adaptation, Anthropocene, capricious gods, chance, citizenship, civilization, clean disruption, conservation, consumption, decentralized electric power generation, decentralized energy, demand-side solutions, destructive economic development, dynamical systems, economics, efficiency, energy, energy reduction, energy utilities, engineering, environment, ethics, forecasting, fossil fuel divestment, geophysics, global warming, Hyper Anthropocene, investment in wind and solar energy, living shorelines, mesh models, meteorology, microgrids, mitigation, obfuscating data, oceanography, physical materialism, physics, pipelines, planning, politics, prediction, probabilistic programming, public utility commissions, PUCs, quantum, reasonableness, reproducible research, risk, Sankey diagram, science, sea level rise, selfishness, solar energy, solar power, SolarPV.tv, Spaceship Earth, statistics, stochastic algorithms, stochastics, Svante Arrhenius, taxes, temporal myopia, the right to know, the value of financial assets, transparency, UU Humanists, WHOI, wind energy, wind power, zero carbon | Leave a comment

Southern New England Meteorology Conference, 24th October 2015

I attending the 2015 edition of the Southern New England Meteorology Conference in Milton, MA, near the Blue Hill, and its Blue Hill Climatological Observatory, of which I am a member as we as of the American Meteorological Society. I … Continue reading

Posted in Anthropocene, capricious gods, climate, Dan Satterfield, dynamical systems, ensembles, ENSO, environment, floods, forecasting, geophysics, Hyper Anthropocene, information theoretic statistics, mesh models, meteorology, model comparison, NCAR, NOAA, nor'easters, oceanography, probability, science, spatial statistics, state-space models, statistics, stochastic algorithms, stochastic search, stochastics, time series | 1 Comment

Solar array with cloud predicting technology launched in WA

Australia’s first grid-connected solar power project with cloud predicting technology launched at Karratha Airport, WA, in bid to smooth solar supply. Source: Solar array with cloud predicting technology launched in WA

Posted in adaptation, Anthropocene, carbon dioxide, citizenship, civilization, clean disruption, climate, climate change, climate disruption, conservation, consumption, decentralized electric power generation, decentralized energy, demand-side solutions, dynamic linear models, efficiency, energy, energy reduction, energy utilities, engineering, environment, ethics, forecasting, geophysics, global warming, Hyper Anthropocene, investment in wind and solar energy, Kalman filter, mathematics, maths, meteorology, microgrids, mitigation, NCAR, numerical software, optimization, physics, prediction, probabilistic programming, rationality, reasonableness, risk, science, solar power, stochastics, sustainability, time series | Leave a comment

Your future: Antarctica, in detail

Climate and geophysical accuracy demands fine modeling grids, and very large supercomputers. The best and biggest supercomputers have not been available for climate work, until recently. Watch how results differ if fine meshes and big supercomputers are used. Why haven’t … Continue reading

Posted in Antarctica, Anthropocene, bridge to nowhere, climate, climate change, climate disruption, climate zombies, disingenuity, ecology, ensembles, forecasting, geophysics, global warming, Hyper Anthropocene, ignorance, IPCC, Lawrence Berkeley National Laboratory, LBNL, living shorelines, mathematics, mathematics education, maths, mesh models, meteorology, multivariate statistics, numerical software, optimization, physics, rationality, reasonableness, risk, science, science education, sea level rise, spatial statistics, state-space models, statistics, stochastic algorithms, stochastics, supercomputers, temporal myopia, the right to know, thermodynamics, time series, University of California Berkeley, WAIS | Leave a comment

“The Bayesian Second Law of Thermodynamics” (Sean Carroll, and collaborators)

http://www.preposterousuniverse.com/blog/2015/08/11/the-bayesian-second-law-of-thermodynamics/ See also.

Posted in approximate Bayesian computation, Bayesian, bifurcations, Boltzmann, capricious gods, dynamical systems, ensembles, games of chance, Gibbs Sampling, information theoretic statistics, Josiah Willard Gibbs, mathematics, maths, physics, probability, rationality, reasonableness, science, statistics, stochastic algorithms, stochastics, thermodynamics, Wordpress | Leave a comment

SCIENCE OF DOOM takes on assessing zero Carbon power and a zero Carbon grid

Updated, 2127 EDT, 10th August 2015 The blog, Science of Doom, has taken on a new thread discussing the technical feasibilities and problems associated with building out zero Carbon energy in the context of an electric grid. As such, it … Continue reading

Posted in adaptation, Anthropocene, clean disruption, climate data, climate disruption, conservation, consumption, decentralized electric power generation, decentralized energy, demand-side solutions, destructive economic development, dynamic linear models, dynamical systems, economics, efficiency, energy, energy reduction, engineering, environment, exponential growth, forecasting, fossil fuel divestment, global warming, Hyper Anthropocene, investing, investment in wind and solar energy, microgrids, open data, optimization, prediction, rationality, reasonableness, risk, solar power, state-space models, stochastics, sustainability, the right to know, time series, wind power, Wordpress, zero carbon | 4 Comments

disruption to global electricity production during the next 25 years

I am a huge fan of Tony Seba’s writings and work, primarily because I am an engineer, and I simply cannot accept that the situation with the impending climate catastrophe is hopeless. Engineers are eternal optimists. Not everyone will be … Continue reading

Posted in adaptation, bridge to nowhere, carbon dioxide, Carbon Tax, citizenship, clean disruption, climate, climate change, climate disruption, consumption, decentralized electric power generation, decentralized energy, dynamical systems, economics, energy, engineering, environment, ethics, exponential growth, fossil fuel divestment, fossil fuels, fracking, global warming, history, investment in wind and solar energy, mathematics, maths, methane, microgrids, natural gas, politics, rationality, reasonableness, risk, sociology, solar power, statistics, stochastics, sustainability, taxes, the right to know, Tony Seba, transparency, wind power, zero carbon | 6 Comments

“NOAA temperature record updates and the ‘hiatus’” (from Gavin at REALCLIMATE)

NOAA temperature record updates and the ‘hiatus’. No doubt there’ll be, as Dr Schmidt says, a howl of protests that the data are “being manipulated”. There’s more discussion by Professor Mann. But, more to the point, it looks like we’re … Continue reading

Posted in chance, citizen science, climate, climate change, climate disruption, climate education, climate models, diffusion processes, dynamical systems, energy, ensembles, forecasting, geophysics, global warming, hiatus, maths, meteorology, model comparison, NASA, NCAR, NOAA, oceanography, physics, rationality, spatial statistics, statistics, stochastics, temporal myopia, time series | 2 Comments

On the Climate Club

But if the other advanced nations had a stick — a tariff of 4 percent on the imports from countries not in the “climate club” — the cost-benefit calculation for the United States would flip. Not participating in the club … Continue reading

Posted in citizenship, civilization, climate, climate change, climate disruption, climate education, ecology, economics, education, environment, ethics, geophysics, global warming, humanism, investing, investment in wind and solar energy, IPCC, mathematics, mathematics education, maths, meteorology, NASA, NCAR, NOAA, open data, open source scientific software, politics, rationality, reasonableness, risk, science, science education, sociology, state-space models, statistics, stochastic search, stochastics, sustainability, temporal myopia, time series, transparency, Unitarian Universalism, UU Humanists, wind power, zero carbon | 2 Comments

A promise forward …

I’ve made a commitment at Google Plus to detail the implications of underestimated rainfall in terms of precipitation risk. I’m planning to tie this up with my informal work on the Town of Sharon’s water supply, in Sharon, MA. Update, … Continue reading

Posted in chance, citizenship, ecology, ENSO, environment, floods, forecasting, geophysics, meteorology, politics, precipitation, rationality, state-space models, statistics, stochastics | Leave a comment

“Cauchy Distribution: Evil or Angel?” (from Xian)

Cauchy Distribution: Evil or Angel?. From Professor Christian Robert.

Posted in arXiv, Bayes, Bayesian, Cauchy distribution, information theoretic statistics, mathematics, maths, optimization, probabilistic programming, probability, rationality, reasonableness, statistics, stochastic algorithms, stochastics, Student t distribution | Leave a comment

“el Niño and the Non-Spherical Cow”

Originally posted on Open Mind:
Most people who follow climate science are aware that one of the natural factors which affects global temperate is the el Niño Southern Oscillation (ENSO). It’s a mode of natural variation in the tropical eastern…

Posted in Anthropocene, bridge to nowhere, carbon dioxide, Carbon Tax, civilization, clean disruption, climate, climate change, climate disruption, climate education, ecology, economics, ENSO, ethics, forecasting, global warming, meteorology, NCAR, NOAA, oceanography, open data, open source scientific software, physics, politics, rationality, Ray Pierrehumbert, reasonableness, reproducible research, risk, science, science education, sea level rise, statistics, stochastics, temporal myopia, time series, transparency | Leave a comment

Stone STOCHASTICITY Project

(Click on image for a larger one.) See the write-up for details.

Posted in ales, capricious gods, chance, chemistry, games of chance, maths, probability, risk, statistics, stochastics, the right to know, Uncategorized | Leave a comment