Category Archives: statistics

Papers of the day

From the Machine Learning and Computational Modeling Lab, School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran: A. Ahmadian, K. Fouladi, B. N. Araabi, “Writer identification using a probabilistic model of handwritten digits and Approximate Bayesian Computation,” International … Continue reading

Posted in American Association for the Advancement of Science, American Statistical Association, approximate Bayesian computation, Bayesian, civilization, computation, denial, education, engineering, evidence, free flow of labor, physics, science, science education, statistics | Leave a comment

The Azimuth Climate Data Backup Project, in association with ClimateMirror

(Updated the afternoon of 3rd February 2017.) The Azimuth Climate Data Backup Project, operating in association with ClimateMirror, is being funded via the Kickstarter available at this link. Give what you can. Thanks! See our goal statement. This is all … Continue reading

Posted in American Association for the Advancement of Science, citizen science, civilization, climate, climate change, climate data, climate disruption, climate education, climate justice, Climate Lab Book, cynicism, denial, Donald Trump, education, EIA, ethics, evidence, fear uncertainty and doubt, forecasting, fossil fuels, Global Carbon Project, global warming, greenhouse gases, Hyper Anthropocene, leaving fossil fuels in the ground, NASA, NOAA, open data, open source scientific software, rationality, reason, reasonableness, risk, science, science denier, science education, smart data, statistics, the right to know, the tragedy of our present civilization, UU, ``The tide is risin'/And so are we'' | Leave a comment

Energy Consumption with Air Source Heat Pumps and Water Heater

Once nice thing about having a net metered solar PV array is that, with a little diligence, you can figure out how much electricity your household is consuming each day, or at finer resolution if you like (*). Below is … Continue reading

Posted in Anthropocene, Bloomberg New Energy Finance, clean disruption, CleanTechnica, climate economics, conservation, consumption, decentralized electric power generation, decentralized energy, demand-side solutions, efficiency, energy reduction, engineering, global warming, Hyper Anthropocene, ISO-NE, local self reliance, New England, smart data, solar democracy, solar domination, solar energy, statistics, time series, Tony Seba, zero carbon | Leave a comment

“… The Green-Feminine stereotype and its effect on sustainable consumption”

Updated, 2016-11-28 I heard about this study earlier this year, and queued it up for a careful examination. I got to that today. The article is: A R Brough, J E B Wilkie, J Ma, M S Isaac, D Gal, … Continue reading

Posted in Frequentist, statistics | Leave a comment

“All models are wrong. Some models are useful.” — George Box

(Image courtesy of the Damien Garcia.) As a statistician and quant, I’ve thought hard about that oft-cited Boxism. I’m not sure I agree. It’s not that there is such a thing as a perfect model, or correct model, whatever in … Continue reading

Posted in abstraction, American Association for the Advancement of Science, astronomy, astrophysics, mathematics, model-free forecasting, numerics, perceptions, physical materialism, physics, rationality, reason, reasonableness, science, spatial statistics, splines, statistics, the right to know, theoretical physics, time series | Leave a comment

Just a lil’ bit o’ a drought … Nothing to be alarmed about … (!)

Posted in adaptation, American Meteorological Association, AMETSOC, Anthropocene, atmosphere, climate change, climate data, climate disruption, drought, environment, fluid dynamics, global warming, greenhouse gases, hydrology, Hyper Anthropocene, leaving fossil fuels in the ground, meteorology, quantitative ecology, Spaceship Earth, statistics, time series, water, water vapor, WHOI, zero carbon | Leave a comment

“Predicting annual temperatures a year ahead” (Dr Gavin Schmidt at REALCLIMATE)

Dr Schmidt is essentially betting that the trend, seen as a random variable, will regress towards the smooth mean. I have a post at Nate Silver’s 538 site on how we can predict annual surface temperature anomalies based on El … Continue reading

Posted in American Meteorological Association, American Statistical Association, AMETSOC, anomaly detection, Anthropocene, changepoint detection, climate change, climate data, climate disruption, climate education, climate models, ecology, environment, forecasting, geophysics, global warming, Hyper Anthropocene, meteorology, oceanography, physics, regression toward the mean, science, statistics, time series | Leave a comment

Bastardi’s Bust

Famous climate denialist Joe Bastari of WeatherBELL Analytics LLC, formerly of Accuweather.com made a prediction on Arctic ice recovery back in 2010 (when at AccuWeather), and observations have since made his “studies” laughable. I have heard his colleague, Joseph D’Aleo … Continue reading

Posted in Accuweather, American Meteorological Association, anomaly detection, Anthropocene, Arctic, climate change, climate disruption, climate models, coasts, ecology, environment, evidence, global warming, Hyper Anthropocene, ice sheet dynamics, meteorology, NOAA, science denier, shorelines, statistics, Stefan Rahmstorf, the right to be and act stupid, the right to know, the stack of lies, the tragedy of our present civilization, time series | Leave a comment

NextGen VOICES: `On data’, `On setbacks’, and `On discovery’

Science Magazine has a periodic column called Science in brief and occasionally that column features a set of what they call “NextGen VOICES”, meaning young scientists. They gather the survey using Twitter (of course) via the hashtag #NextGenSci. For the … Continue reading

Posted in American Association for the Advancement of Science, big data, data science, evidence, Mathbabe, maths, maxims, science, Science magazine, smart data, statistics | Leave a comment

“Sharon’s Water Problem” (by Paul Lauenstein)

(Click on image to see a bigger version of this figure. Use your browser Back Button to return to this blog.) The town of Sharon, MA, has a water problem. Click on the link and see Paul’s presentation about it. … Continue reading

Posted in adaptation, agriculture, American Meteorological Association, AMETSOC, Anthropocene, Boston, citizen science, climate change, climate disruption, diffusion processes, drought, ecology, Ecology Action, environment, forecasting, global warming, hydrology, Hyper Anthropocene, MA, New England, Paul Lauenstein, precipitation, quantitative ecology, science, statistics, the tragedy of our present civilization, water, Woods Hole Oceanographic Institution | Leave a comment

“Naïve empiricism and what theory suggests about errors in observed global warming”

A post from one of my favorite statistics-oriented bloggers, Variable Variability, dealing with a subject too casually passed over. See Naïve empiricism and what theory suggests about errors in observed global warming.

Posted in Anthropocene, climate, climate change, climate data, climate disruption, confirmation bias, geophysics, rationality, statistics, time series, Variable Variability, Victor Venema | Leave a comment

Can the City of Boston adapt to and help mitigate climate disruption?

(See the major update at the bottom of this post as well.) (On “Less Science and More Social Science” at And Then There’s Physics) And Then There’s Physics is one of my favorite blogs discussing climate disruption and related policy … Continue reading

Posted in adaptation, anomaly detection, Anthropocene, Bill Nye, Bloomberg New Energy Finance, BNEF, bollocks, Boston, bridge to somewhere, citizenship, civilization, clean disruption, climate business, climate change, climate disruption, climate economics, climate education, climate justice, Daniel Kahneman, destructive economic development, economics, engineering, environment, finance, floods, forecasting, Gaylord Nelson, global warming, Hermann Scheer, Hyper Anthropocene, insurance, investing, John Englander, Joseph Schumpeter, Kerry Emanuel, MA, Massachusetts, meteorology, Minsky moment, nor'easters, organizational failures, politics, risk, sea level rise, sociology, statistics, supply chains, sustainability, the right to be and act stupid, the right to know, the tragedy of our present civilization, the value of financial assets | 6 Comments

Repaired R code for Markov spatial simulation of hurricane tracks from historical trajectories

I’m currently studying random walk and diffusion processes and their connections with random fields. I’m interested in this because at the core of dynamic linear models, Kalman filters, and state-space methods there is a random walk in a parameter space. … Continue reading

Posted in American Meteorological Association, American Statistical Association, AMETSOC, Arthur Charpentier, atmosphere, diffusion, diffusion processes, dynamic linear models, dynamical systems, environment, geophysics, hurricanes, Kalman filter, Kerry Emanuel, Lévy flights, Lorenz, Markov chain random fields, mathematics, mathematics education, maths, MCMC, mesh models, meteorological models, meteorology, model-free forecasting, Monte Carlo Statistical Methods, numerical analysis, numerical software, oceanography, open data, open source scientific software, physics, random walk processes, random walks, science, spatial statistics, state-space models, statistical dependence, statistics, stochastic algorithms, stochastics, time series | Leave a comment

Energy Democracy

I’ve actually written about this before, but John Farrell of the ILSR (“Institute for Local Self-Reliance” a famous Emerson essay, by the way) presents an up-to-date synthesis of developments, incorporating policy as well as Tony Seba-like, Hermann Scheer-like, and Michael … Continue reading

Posted in abstraction, Anthropocene, Bloomberg New Energy Finance, BNEF, bridge to somewhere, Buckminster Fuller, citizenship, clean disruption, CleanTechnica, climate economics, corporate litigation on damage from fossil fuel emissions, decentralized electric power generation, decentralized energy, demand-side solutions, destructive economic development, distributed generation, economics, efficiency, electricity, electricity markets, energy, energy reduction, energy storage, energy utilities, engineering, evidence, extended supply chains, feed-in tariff, forecasting, fossil fuel divestment, fossil fuels, green tech, grid defection, Hermann Scheer, Hyper Anthropocene, ILSR, investment in wind and solar energy, John Farrell, Joseph Schumpeter, leaving fossil fuels in the ground, liberal climate deniers, life purpose, local generation, Massachusetts Clean Energy Center, Michael Osborne, microgrids, open data, public utility commissions, PUCs, rate of return regulation, rationality, reason, reasonableness, regime shifts, regulatory capture, risk, Sankey diagram, solar democracy, solar domination, solar energy, solar power, SolarPV.tv, Spaceship Earth, spatial statistics, statistics, stranded assets, sustainability, temporal myopia, the energy of the people, the green century, Tony Seba, utility company death spiral, wind energy, wind power, zero carbon | 1 Comment

“Holy crap – an actual book!”

Originally posted on mathbabe:
Yo, everyone! The final version of my book now exists, and I have exactly one copy! Here’s my editor, Amanda Cook, holding it yesterday when we met for beers: Here’s my son holding it: He’s offered…

Posted in American Association for the Advancement of Science, Buckminster Fuller, business, citizen science, citizenship, civilization, complex systems, confirmation bias, data science, data streams, deep recurrent neural networks, denial, economics, education, engineering, ethics, evidence, Internet, investing, life purpose, machine learning, mathematical publishing, mathematics, mathematics education, maths, moral leadership, multivariate statistics, numerical software, numerics, obfuscating data, organizational failures, politics, population biology, prediction, prediction markets, privacy, quantitative biology, quantitative ecology, rationality, reason, reasonableness, rhetoric, risk, Schnabel census, smart data, sociology, statistical dependence, statistics, the right to be and act stupid, the right to know, the value of financial assets, transparency, UU Humanists | Leave a comment

A model of an electrical grid: A vision

Many people seem to view the electrical grid of the future being much like the present one. I think a lot about networks, because of my job. And I especially think a lot about network topologies, although primarily concerning the … Continue reading

Posted in abstraction, American Meteorological Association, anomaly detection, Anthropocene, Bloomberg New Energy Finance, BNEF, Boston, bridge to somewhere, Buckminster Fuller, Canettes Blues Band, clean disruption, climate business, climate economics, complex systems, corporate supply chains, decentralized electric power generation, decentralized energy, demand-side solutions, differential equations, distributed generation, efficiency, EIA, electricity, electricity markets, energy, energy reduction, energy storage, energy utilities, engineering, extended supply chains, green tech, grid defection, Hermann Scheer, Hyper Anthropocene, investment in wind and solar energy, ISO-NE, Kalman filter, kriging, Lawrence Berkeley National Laboratory, leaving fossil fuels in the ground, Lenny Smith, local generation, marginal energy sources, Massachusetts Clean Energy Center, Mathematics and Climate Research Network, mesh models, meteorology, microgrids, networks, New England, New York State, open data, organizational failures, pipelines, planning, prediction markets, public utility commissions, PUCs, rate of return regulation, rationality, reason, reasonableness, regime shifts, regulatory capture, resiliency, risk, Sankey diagram, smart data, solar domination, solar energy, solar power, Spaceship Earth, spatial statistics, state-space models, statistical dependence, statistics, stochastic algorithms, stochastics, stranded assets, supply chains, sustainability, the energy of the people, the green century, the value of financial assets, thermodynamics, time series, Tony Seba, utility company death spiral, wave equations, wind energy, wind power, zero carbon | Leave a comment

Bayesian blocks via PELT in R

The Bayesian blocks algorithm of Scargle, Jackson, Norris, and Chiang has an enthusiastic user community in astrostatistics, in data mining, and among some in machine learning. It is a dynamic programming algorithm (see VanderPlas referenced below) and, so, exhibits optimality … Continue reading

Posted in American Statistical Association, AMETSOC, anomaly detection, astrophysics, Cauchy distribution, changepoint detection, engineering, geophysics, multivariate statistics, numerical analysis, numerical software, numerics, oceanography, population biology, population dynamics, Python 3, quantitative biology, quantitative ecology, R, Scargle, spatial statistics, square wave approximation, statistics, stepwise approximation, time series, Woods Hole Oceanographic Institution | Leave a comment

“Full-depth Ocean Heat Content” reblog

This is a re-blog of an excellent post at And Then There’s Physics, titled Full-depth OHC or, expanded, “full-depth ocean heat content”. Since my holiday is now over, I thought I might briefly comment on a recent paper by Cheng … Continue reading

Posted in Anthropocene, climate, climate change, climate data, climate disruption, climate models, computation, differential equations, ensembles, environment, fluid dynamics, forecasting, geophysics, global warming, greenhouse gases, Hyper Anthropocene, Lorenz, Mathematics and Climate Research Network, model comparison, NOAA, oceanography, physics, science, statistics, theoretical physics, thermodynamics, time series | Leave a comment

“Stochastic Parameterization: Towards a new view of weather and climate models”

Judith Berner, Ulrich Achatz, Lauriane Batté, Lisa Bengtsson, Alvaro De La Cámara, Hannah M. Christensen, Matteo Colangeli, Danielle R. B. Coleman, Daan Crommelin, Stamen I. Dolaptchiev, Christian L.E. Franzke, Petra Friederichs, Peter Imkeller, Heikki Järvinen, Stephan Juricke, Vassili Kitsios, François … Continue reading

Posted in biology, climate models, complex systems, convergent cross-mapping, data science, dynamical systems, ecology, Ethan Deyle, Floris Takens, George Sughihara, Hao Ye, likelihood-free, Lorenz, mathematics, meteorological models, model-free forecasting, physics, population biology, population dynamics, quantitative biology, quantitative ecology, Scripps Institution of Oceanography, state-space models, statistical dependence, statistics, stochastic algorithms, stochastic search, stochastics, Takens embedding theorem, time series, Victor Brovkin | 4 Comments

The Presidential betting markets

Someone blatantly misrepresented the U.S. Presidential election betting markets in a Google+ comment thread tonight, and I wanted to bring these forward, here. See the latest odds and assessments from the prediction markets. Done. No doubt some supporters of Trump … Continue reading

Posted in forecasting, investing, politics, prediction markets, rationality, reasonableness, statistics | Leave a comment

David Spiegelhalter on `how to spot a dodgy statistic’

In this political season, it’s useful to brush up on rhetorical skills, particularly ones involving numbers and statistics, or what John Allen Paulos called numeracy. Professor David Spiegelhalter has written a guide to some of these tricks. Read the whole … Continue reading

Posted in abstraction, anemic data, Bayes, Bayesian, chance, citizenship, civilization, corruption, Daniel Kahneman, disingenuity, Donald Trump, education, games of chance, ignorance, maths, moral leadership, obfuscating data, open data, perceptions, politics, rationality, reason, reasonableness, rhetoric, risk, sampling, science, sociology, statistics, the right to know | Leave a comment

JASA demands code and data be supplied as a condition of publication

The Journal of the American Statistical Association (“JASA”) has announced in this month’s Amstat News that effective 1st September 2016 “… will require code and data as a minimum standard for reproducibility of statistical scientific research.” Trends were heading this … Continue reading

Posted in American Association for the Advancement of Science, American Statistical Association, citizen science, engineering, ethics, evidence, new forms of scientific peer review, numerical software, planning, rationality, reasonableness, resiliency, science, statistics, stochastic algorithms, testing, the right to know | Leave a comment

Newt Gingrich and Van Jones. Right on.

It’s the thing. And it addresses how media and people forget about the actual statistics, and focus on the White Hot Bright Light. A study by Gelman, Fagan, and Kiss A study by Freyer A counterpoint to the Freyer study … Continue reading

Posted in American Statistical Association, Bayes, Bayesian, citizen science, criminal justice, Daniel Kahneman, ethics, evidence, fear uncertainty and doubt, humanism, Lives Matter, logistic regression, Markov Chain Monte Carlo, MCMC, organizational failures, population biology, rationality, reasonableness, risk, statistics, Susan Jacoby, the right to know | Leave a comment

data.table

R provides a helpful data structure called the “data frame” that gives the user an intuitive way to organize, view, and access data.  Many of the functions that you would us… Source: Intro to The data.table Package

Posted in big data, data science, engineering, numerical analysis, numerical software, numerics, open source scientific software, R, smart data, statistics | Leave a comment

On Smart Data

One of the things I find surprising, if not astonishing, is that in the rush to embrace Big Data, a lot of learning and statistical technique has been left apparently discarded along the way. I’m hardly the first to point … Continue reading

Posted in Akaike Information Criterion, Bayes, Bayesian, Bayesian inversion, big data, bigmemory package for R, changepoint detection, data science, data streams, dlm package, dynamic generalized linear models, dynamic linear models, dynamical systems, Generalize Additive Models, generalized linear models, information theoretic statistics, Kalman filter, linear algebra, logistic regression, machine learning, Markov Chain Monte Carlo, mathematics, mathematics education, maths, maximum likelihood, MCMC, Monte Carlo Statistical Methods, multivariate statistics, numerical analysis, numerical software, numerics, quantitative biology, quantitative ecology, rationality, reasonableness, sampling, smart data, state-space models, statistical dependence, statistics, the right to know, time series | Leave a comment

“Catching long tail distribution” (Ted Dunning)

One of the best presentations on what can happen if someone takes a naive approach to network data. It also highlights what is, to my mind, the greatly underappreciated t-distribution, which is typically only used in connection with frequentist Student … Continue reading

Posted in Cauchy distribution, complex systems, data science, Lévy flights, leptokurtic, mathematics, maths, networks, physics, population biology, population dynamics, regime shifts, sampling, statistics, Student t distribution, time series | Leave a comment

On Munshi mush

(Slightly updated on 2016-06-11.) Professor Emeritus Jamal Munshi of Sonoma State University has papers recently cited in science denier circles as evidence that the conventional associations between mean global surface temperature and cumulative carbon emissions are, well, bunk, due to … Continue reading

Posted in Bayes, Bayesian, Berkeley Earth Surface Temperature project, BEST, carbon dioxide, cat1, climate, climate change, climate data, climate education, climate models, convergent cross-mapping, dynamic linear models, ecology, ENSO, environment, Ethan Deyle, evidence, geophysics, George Sughihara, global warming, greenhouse gases, information theoretic statistics, Kalman filter, mathematics, maths, meteorology, model comparison, NOAA, oceanography, prediction, state-space models, statistics, Takens embedding theorem, Techno Utopias, the right to know, theoretical physics, time series, zero carbon | 1 Comment

Cory Lesmeister’s treatment of Simson’s Paradox (at “Fear and Loathing in Data Science”)

(Updated 2016-05-08, to provide reference for plateaus of ML functions in vicinity of MLE.) Simpson’s Paradox is one of those phenomena of data which really give Statistics a substance and a role, beyond the roles it inherits from, say, theoretical … Continue reading

Posted in Akaike Information Criterion, approximate Bayesian computation, Bayes, Bayesian, evidence, Frequentist, games of chance, information theoretic statistics, Kalman filter, likelihood-free, mathematics, maths, maximum likelihood, Monte Carlo Statistical Methods, probabilistic programming, rationality, Rauch-Tung-Striebel, Simpson's Paradox, state-space models, statistical dependence, statistics, stochastics | Leave a comment

“Solar power is contagious. These maps show how it spreads.” (from Vox)

Brad Plumer at Vox writes on take-up patterns of rooftop solar based upon a large dataset from SolarCity. The full article is available at the SolarCity site. Mr Plumer combines it with a report on other studies of solar adoption … Continue reading

Posted in Bloomberg New Energy Finance, BNEF, business, clean disruption, decentralized electric power generation, decentralized energy, demand-side solutions, destructive economic development, diffusion, diffusion processes, disruption, distributed generation, economics, electricity markets, energy, energy utilities, exponential growth, grid defection, investment in wind and solar energy, local generation, Peter Diggle, point pattern analysis, public utility commissions, PUCs, rate of return regulation, regulatory capture, sociology, solar domination, solar energy, solar power, SolarPV.tv, spatial statistics, statistics, stochastics, the energy of the people, the green century, utility company death spiral, zero carbon | Leave a comment

“Lucky d20” (by Tamino, with my reblogging comments)

Originally posted on Open Mind:
What with talk of killer heat waves, droughts, floods, etc. etc., this blog tends to get pretty serious. When it does, we don’t deal with happy prospects, but with the danger of worldwide catastrophe. But…

Posted in Bayes, Bayesian, card decks, card draws, card games, chance, D&D, Dungeons and Dragons, games of chance, mathematics, maths, Monte Carlo Statistical Methods, probability, statistical dependence, statistics, stochastic algorithms, stochastics, Wizards of the Coast | Leave a comment