Distributed Solar: The Democratizaton of Energy
Category Archives: stochastic algorithms
The Johnson-Lindenstrauss Lemma, and the paradoxical power of random linear operators. Part 1.
Updated, 2018-12-04 I’ll be discussing the ramifications of: William B. Johnson and Joram Lindenstrauss, “Extensions of Lipschitz mappings into a Hilbert space, Contemporary Mathematics, 26:189–206, 1984. for several posts here. Some introduction and links to proofs and explications will be … Continue reading
Posted in clustering, data science, dimension reduction, information theoretic statistics, Johnson-Lindenstrauss Lemma, k-NN, Locality Sensitive Hashing, mathematics, maths, multivariate statistics, non-parametric model, numerical algorithms, numerical linear algebra, point pattern analysis, random projections, recommender systems, science, stochastic algorithms, stochastics, subspace projection methods
Leave a comment
Sampling: Rejection, Reservoir, and Slice
An article by Suilou Huang for catatrophe modeler AIR-WorldWide of Boston about rejection sampling in CAT modeling got me thinking about pulling together some notes about sampling algorithms of various kinds. There are, of course, books written about this subject, … Continue reading
Posted in accept-reject methods, American Statistical Association, Bayesian computational methods, catastrophe modeling, data science, diffusion processes, empirical likelihood, Gibbs Sampling, insurance, Markov Chain Monte Carlo, mathematics, Mathematics and Climate Research Network, maths, Monte Carlo Statistical Methods, multivariate statistics, numerical algorithms, numerical analysis, numerical software, numerics, percolation theory, Python 3 programming language, R statistical programming language, Radford Neal, sampling, slice sampling, spatial statistics, statistics, stochastic algorithms, stochastic search
Leave a comment
Repaired R code for Markov spatial simulation of hurricane tracks from historical trajectories
I’m currently studying random walk and diffusion processes and their connections with random fields. I’m interested in this because at the core of dynamic linear models, Kalman filters, and state-space methods there is a random walk in a parameter space. … Continue reading
Posted in American Meteorological Association, American Statistical Association, AMETSOC, Arthur Charpentier, atmosphere, diffusion, diffusion processes, dynamic linear models, dynamical systems, environment, geophysics, hurricanes, Kalman filter, Kerry Emanuel, Lévy flights, Lorenz, Markov chain random fields, mathematics, mathematics education, maths, MCMC, mesh models, meteorological models, meteorology, model-free forecasting, Monte Carlo Statistical Methods, numerical analysis, numerical software, oceanography, open data, open source scientific software, physics, random walk processes, random walks, science, spatial statistics, state-space models, statistical dependence, statistics, stochastic algorithms, stochastics, time series
1 Comment
A model of an electrical grid: A vision
Many people seem to view the electrical grid of the future being much like the present one. I think a lot about networks, because of my job. And I especially think a lot about network topologies, although primarily concerning the … Continue reading
Posted in abstraction, American Meteorological Association, anomaly detection, Anthropocene, Bloomberg New Energy Finance, BNEF, Boston, bridge to somewhere, Buckminster Fuller, Canettes Blues Band, clean disruption, climate business, climate economics, complex systems, corporate supply chains, decentralized electric power generation, decentralized energy, demand-side solutions, differential equations, distributed generation, efficiency, EIA, electricity, electricity markets, energy, energy reduction, energy storage, energy utilities, engineering, extended supply chains, green tech, grid defection, Hermann Scheer, Hyper Anthropocene, investment in wind and solar energy, ISO-NE, Kalman filter, kriging, Lawrence Berkeley National Laboratory, leaving fossil fuels in the ground, Lenny Smith, local generation, marginal energy sources, Massachusetts Clean Energy Center, Mathematics and Climate Research Network, mesh models, meteorology, microgrids, networks, New England, New York State, open data, organizational failures, pipelines, planning, prediction markets, public utility commissions, PUCs, rate of return regulation, rationality, reason, reasonableness, regime shifts, regulatory capture, resiliency, risk, Sankey diagram, smart data, solar domination, solar energy, solar power, Spaceship Earth, spatial statistics, state-space models, statistical dependence, statistics, stochastic algorithms, stochastics, stranded assets, supply chains, sustainability, the energy of the people, the green century, the value of financial assets, thermodynamics, time series, Tony Seba, utility company death spiral, wave equations, wind energy, wind power, zero carbon
Leave a comment
JASA demands code and data be supplied as a condition of publication
The Journal of the American Statistical Association (“JASA”) has announced in this month’s Amstat News that effective 1st September 2016 “… will require code and data as a minimum standard for reproducibility of statistical scientific research.” Trends were heading this … Continue reading
Posted in American Association for the Advancement of Science, American Statistical Association, citizen science, engineering, ethics, evidence, new forms of scientific peer review, numerical software, planning, rationality, reasonableness, resiliency, science, statistics, stochastic algorithms, testing, the right to know
Leave a comment
France, and Mathematics
Cédric Villani, does Mathematics. “Problems worthy of attack, prove their worth by hitting back.” — Piet Hein
Six cases of models
The previous post included an attempt to explain land surface temperatures as estimated by the BEST project using a dynamic linear model including regressions on both quarterly CO2 concentrations and ocean heat content. The idea was to check the explanatory … Continue reading
Posted in AMETSOC, anemic data, Anthropocene, astrophysics, Bayesian, Berkeley Earth Surface Temperature project, BEST, carbon dioxide, climate, climate change, climate data, climate disruption, climate models, dlm package, dynamic linear models, dynamical systems, environment, fossil fuels, geophysics, Giovanni Petris, global warming, greenhouse gases, Hyper Anthropocene, information theoretic statistics, maths, maximum likelihood, meteorology, model comparison, numerical software, Patrizia Campagnoli, Rauch-Tung-Striebel, Sonia Petrone, state-space models, stochastic algorithms, stochastic search, SVD, time series
1 Comment
“Lucky d20” (by Tamino, with my reblogging comments)
Originally posted on Open Mind:
What with talk of killer heat waves, droughts, floods, etc. etc., this blog tends to get pretty serious. When it does, we don’t deal with happy prospects, but with the danger of worldwide catastrophe. But…
Of my favorite things …
(Clarifying language added 4 Apr 2016, 12:26 EDT.) I just watched an episode from the last season of Star Trek: The Next Generation entitled “Force of Nature.” As anyone who pays the least attention to this blog knows, opposing human … Continue reading
Posted in Anthropocene, bridge to somewhere, bucket list, Buckminster Fuller, Carl Sagan, climate, climate change, climate disruption, climate education, compassion, data science, Earle Wilson, ecology, Ecology Action, environment, evolution, geophysics, George Sughihara, global warming, Hyper Anthropocene, life purpose, mathematics, mathematics education, maths, numerical analysis, optimization, philosophy, physical materialism, physics, population biology, population dynamics, proud dad, quantitative biology, quantitative ecology, rationality, reasonableness, science, sociology, statistics, stochastic algorithms
5 Comments
“Grid shading by simulated annealing” [Martyn Plummer]
Source: Grid shading by simulated annealing (or what I did on my holidays), aka “fun with GCHQ job adverts”, by Martyn Plummer, developer of JAGS. Excerpt: I wanted to solve the puzzle but did not want to sit down with … Continue reading
Posted in approximate Bayesian computation, Bayesian, Bayesian inversion, Boltzmann, BUGS, Christian Robert, Gibbs Sampling, JAGS, likelihood-free, Markov Chain Monte Carlo, Martyn Plummer, mathematics, maths, MCMC, Monte Carlo Statistical Methods, optimization, probabilistic programming, SPSA, stochastic algorithms, stochastic search
Leave a comment
high dimension Metropolis-Hastings algorithms
If attempting to simulate from a multivariate standard normal distribution in a large dimension, when starting from the mode of the target, i.e., its mean γ, leaving the mode γis extremely unlikely, given the huge drop between the value of the density at the mode γ and at likely realisations Continue reading
Posted in Bayes, Bayesian, Bayesian inversion, boosting, chance, Christian Robert, computation, ensembles, Gibbs Sampling, James Spall, Jerome Friedman, Markov Chain Monte Carlo, mathematics, maths, MCMC, Monte Carlo Statistical Methods, multivariate statistics, numerical software, numerics, optimization, reasonableness, Robert Schapire, SPSA, state-space models, statistics, stochastic algorithms, stochastic search, stochastics, Yoav Freund
Leave a comment
Generating supports for classification rules in black box regression models
Inspired by the extensive and excellent work in approximate Bayesian computation (see also), especially that done by Professors Christian Robert and colleagues (see also), and Professor Simon Wood (see also), it occurred to me that the complaints regarding lack of … Continue reading
Posted in approximate Bayesian computation, Bayes, Bayesian, Bayesian inversion, generalized linear models, machine learning, numerical analysis, numerical software, probabilistic programming, rationality, reasonableness, state-space models, statistics, stochastic algorithms, stochastic search, stochastics, support of black boxes
Leave a comment
Thoughts on “Regime Shift?”
John Baez at The Azimuth Project opened a discussion on the recent paper by Reid, et al Philip C. Reid et al, Global impacts of the 1980s regime shift on the Earth’s climate and systems, Global Change Biology, 2015. I … Continue reading
reblog: “Tiny Data, Approximate Bayesian Computation and the Socks of Karl Broman”
It’s Rasmus Bååth, in a post and video of which I am very fond: http://www.sumsar.net/blog/2014/10/tiny-data-and-the-socks-of-karl-broman/.
What the future of energy everywhere looks like
What will the energy landscape look like after utility companies are either dead, dying, or revert to a tiny portion of their territory? Silicon Valley CCE Partnership gives us all a clue. It’s been described in the San Francisco Chronicle, … Continue reading
Posted in adaptation, Anthropocene, capricious gods, chance, citizenship, civilization, clean disruption, conservation, consumption, decentralized electric power generation, decentralized energy, demand-side solutions, destructive economic development, dynamical systems, economics, efficiency, energy, energy reduction, energy utilities, engineering, environment, ethics, forecasting, fossil fuel divestment, geophysics, global warming, Hyper Anthropocene, investment in wind and solar energy, living shorelines, mesh models, meteorology, microgrids, mitigation, obfuscating data, oceanography, physical materialism, physics, pipelines, planning, politics, prediction, probabilistic programming, public utility commissions, PUCs, quantum, reasonableness, reproducible research, risk, Sankey diagram, science, sea level rise, selfishness, solar energy, solar power, SolarPV.tv, Spaceship Earth, statistics, stochastic algorithms, stochastics, Svante Arrhenius, taxes, temporal myopia, the right to know, the value of financial assets, transparency, UU Humanists, WHOI, wind energy, wind power, zero carbon
Leave a comment
Southern New England Meteorology Conference, 24th October 2015
I attending the 2015 edition of the Southern New England Meteorology Conference in Milton, MA, near the Blue Hill, and its Blue Hill Climatological Observatory, of which I am a member as we as of the American Meteorological Society. I … Continue reading
Posted in Anthropocene, capricious gods, climate, Dan Satterfield, dynamical systems, ensembles, ENSO, environment, floods, forecasting, geophysics, Hyper Anthropocene, information theoretic statistics, mesh models, meteorology, model comparison, NCAR, NOAA, nor'easters, oceanography, probability, science, spatial statistics, state-space models, statistics, stochastic algorithms, stochastic search, stochastics, time series
1 Comment
On differential localization of tumors using relative concentrations of ctDNA. Part 1.
Like most mammalian tissue, tumors often produce shards of DNA as a byproduct of cell death and fracture. This circulating tumor DNA is being studied as a means of detecting tumors or their resurgence after treatment. (See also a Q&A … Continue reading
Posted in approximate Bayesian computation, Bayesian, Bayesian inversion, cardiovascular system, diffusion, dynamic linear models, eigenanalysis, engineering, forecasting, mathematics, maths, medicine, networks, prediction, spatial statistics, statistics, stochastic algorithms, stochastic search, wave equations
3 Comments
On Changing Things
You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete. That’s from Buckminster Fuller, a fellow Unitarian.
Posted in adaptation, Anthropocene, bifurcations, bridge to nowhere, Buckminster Fuller, Cauchy distribution, clean disruption, climate disruption, demand-side solutions, destructive economic development, Disney, dynamic linear models, dynamical systems, Epcot, exponential growth, fossil fuel divestment, global warming, Hyper Anthropocene, physical materialism, planning, rationality, reasonableness, Spaceship Earth, stochastic algorithms
Leave a comment
Your future: Antarctica, in detail
Climate and geophysical accuracy demands fine modeling grids, and very large supercomputers. The best and biggest supercomputers have not been available for climate work, until recently. Watch how results differ if fine meshes and big supercomputers are used. Why haven’t … Continue reading
Posted in Antarctica, Anthropocene, bridge to nowhere, climate, climate change, climate disruption, climate zombies, disingenuity, ecology, ensembles, forecasting, geophysics, global warming, Hyper Anthropocene, ignorance, IPCC, Lawrence Berkeley National Laboratory, LBNL, living shorelines, mathematics, mathematics education, maths, mesh models, meteorology, multivariate statistics, numerical software, optimization, physics, rationality, reasonableness, risk, science, science education, sea level rise, spatial statistics, state-space models, statistics, stochastic algorithms, stochastics, supercomputers, temporal myopia, the right to know, thermodynamics, time series, University of California Berkeley, WAIS
Leave a comment
“The Bayesian Second Law of Thermodynamics” (Sean Carroll, and collaborators)
http://www.preposterousuniverse.com/blog/2015/08/11/the-bayesian-second-law-of-thermodynamics/ See also.
Posted in approximate Bayesian computation, Bayesian, bifurcations, Boltzmann, capricious gods, dynamical systems, ensembles, games of chance, Gibbs Sampling, information theoretic statistics, Josiah Willard Gibbs, mathematics, maths, physics, probability, rationality, reasonableness, science, statistics, stochastic algorithms, stochastics, thermodynamics, Wordpress
Leave a comment
Comprehensive and compact tutorial on Petris’ DLM package in R; with an update about Helske’s KFAS
A blogger named Lalas produced on Quantitative Thoughts a very comprehensive and compact tutorial on the R package dlm by Petris. I use dlm a lot. Unfortunately, Lalas does not give details on how the SVD is used. They do … Continue reading
Posted in Bayes, Bayesian, dynamic linear models, dynamical systems, forecasting, Kalman filter, mathematics, maths, multivariate statistics, numerical software, open source scientific software, prediction, R, Rauch-Tung-Striebel, state-space models, statistics, stochastic algorithms, SVD, time series
Leave a comment
“Cauchy Distribution: Evil or Angel?” (from Xian)
Cauchy Distribution: Evil or Angel?. From Professor Christian Robert.
“… the most patronizing start to an answer I have ever received …”
Professor Christian Robert tries to help out a student of MCMC on Cross Validated and earns the comment that his help had “the most patronizing start to an answer I have ever received“. I learned a new term: primitivus petitor.
“A vignette on Metropolis” (Christian Robert)
Originally posted on Xi'an's Og:
Over the past week, I wrote a short introduction to the Metropolis-Hastings algorithm, mostly in the style of our Introduction to Monte Carlo with R book, that is, with very little theory and…
“Unbiased Bayes for Big Data: Path of partial posteriors” (Christian Robert)
Unbiased Bayes for Big Data: Path of partial posteriors.
Dynamic Linear Models package, dlmodeler
I’m checking out the dlmodeler package in R for a work project. It is accompanied by textbooks, G. Petris, S. Petrone, P. Campagnoli, Dynamic Linear Models with R, Springer, 2009 and J. Durbin, S. J. Koopman, Time Series Analysis by … Continue reading
Markov Chain Monte Carlo methods and logistic regression
This post could also be subtitled “Residual deviance isn’t the whole story.” My favorite book on logistic regression is by Dr Joseph Hilbe, Logistic Regression Models, CRC Press, 2009, Chapman & Hill. It is a solidly frequentist text, but its … Continue reading
Posted in Bayes, Bayesian, logistic regression, MCMC, notes, R, statistics, stochastic algorithms, stochastic search
3 Comments
Bayesian change-point analysis for global temperatures, 1850-2010
Professor Peter Congdon reports on two Bayesian models for global temperature shifts in his textbook, Applied Bayesian Modelling, as “Example 6.12: Global temperatures, 1850-2010”, on pages 252-253. A direct link is available online. The first is apparently original with Congdon, … Continue reading
