Category Archives: R

Bayesian blocks via PELT in R

The Bayesian blocks algorithm of Scargle, Jackson, Norris, and Chiang has an enthusiastic user community in astrostatistics, in data mining, and among some in machine learning. It is a dynamic programming algorithm (see VanderPlas referenced below) and, so, exhibits optimality … Continue reading

Posted in American Statistical Association, AMETSOC, anomaly detection, astrophysics, Cauchy distribution, changepoint detection, engineering, geophysics, multivariate statistics, numerical analysis, numerical software, numerics, oceanography, population biology, population dynamics, Python 3, quantitative biology, quantitative ecology, R, Scargle, spatial statistics, square wave approximation, statistics, stepwise approximation, time series, Woods Hole Oceanographic Institution | Leave a comment

Rushing the +2 degree Celsius boundary

I made a comment on Google+ pertaining to a report of a recent NOAA finding. Enjoy. But remember that COP21 boundary is equivalent to 450 ppm CO2.

Posted in adaptation, AMETSOC, Anthropocene, atmosphere, Bill Nye, bridge to nowhere, carbon dioxide, Carbon Tax, Carbon Worshipers, citizenship, civilization, clean disruption, climate, climate disruption, COP21, corporate litigation on damage from fossil fuel emissions, differential equations, disruption, distributed generation, Donald Trump, ecology, El Nina, El Nino, energy, energy reduction, engineering, environment, environmental law, Epcot, explosive methane, forecasting, fossil fuel divestment, fossil fuels, geophysics, global warming, greenhouse gases, greenwashing, Hyper Anthropocene, investment in wind and solar energy, IPCC, local generation, Mark Jacobson, Martyn Plummer, microgrids, Miguel Altieri, philosophy, physical materialism, R, resiliency, Ricky Rood, risk, Sankey diagram | Leave a comment

data.table

R provides a helpful data structure called the “data frame” that gives the user an intuitive way to organize, view, and access data.  Many of the functions that you would us… Source: Intro to The data.table Package

Posted in big data, data science, engineering, numerical analysis, numerical software, numerics, open source scientific software, R, smart data, statistics | Leave a comment

Gavin Simpson updates his temperature analysis

See the very interesting discussion at his blog, From the bottom of the heap. It would be nice to see some information theoretic measures on these results, though.

Posted in AMETSOC, Anthropocene, astrophysics, Berkeley Earth Surface Temperature project, carbon dioxide, changepoint detection, climate, climate change, climate data, climate disruption, climate models, ecology, environment, evidence, Gavin Simpson, Generalize Additive Models, geophysics, global warming, HadCRUT4, hiatus, Hyper Anthropocene, information theoretic statistics, Kalman filter, maths, meteorology, numerical analysis, R, rationality, reasonableness, splines, time series | Leave a comment

R and “big data”

On 2nd November 2015, Wes McKinney, the developer of the highly useful Python pandas module (and other things, including books), wrote an amusing blog post, “The problem with the data science language wars“. I by no means disagree with him. … Continue reading

Posted in Bayes, Bayesian, big data, bigmemory package for R, Jay Emerson, MCMC, numerics, Python 3, R, Yale University Statistics Department | Leave a comment

STUFF IN PROGRESS

It’s a good time to reconnoiter and review the things I have in progress and are planned, both as a teaser, and as a promise. I am currently working the following technical projects, entirely on my personal time outside of … Continue reading

Posted in numerical analysis, planning, R, rationality, reasonableness, state-space models, statistics | 1 Comment

Comprehensive and compact tutorial on Petris’ DLM package in R; with an update about Helske’s KFAS

A blogger named Lalas produced on Quantitative Thoughts a very comprehensive and compact tutorial on the R package dlm by Petris. I use dlm a lot. Unfortunately, Lalas does not give details on how the SVD is used. They do … Continue reading

Posted in Bayes, Bayesian, dynamic linear models, dynamical systems, forecasting, Kalman filter, mathematics, maths, multivariate statistics, numerical software, open source scientific software, prediction, R, Rauch-Tung-Striebel, state-space models, statistics, stochastic algorithms, SVD, time series | Leave a comment

Earth Day, my hope

Posted in carbon dioxide, Carl Sagan, Charles Darwin, citizen science, citizenship, civilization, clean disruption, climate, climate change, climate education, compassion, conservation, Darwin Day, demand-side solutions, ecology, economics, education, efficiency, energy reduction, environment, ethics, forecasting, fossil fuel divestment, geophysics, history, humanism, investing, investment in wind and solar energy, IPCC, mathematics, maths, meteorology, NCAR, NOAA, oceanography, open data, open source scientific software, physics, politics, population biology, Principles of Planetary Climate, privacy, probit regression, R, rationality, Ray Pierrehumbert, reasonableness, reproducible research, risk, science, science education, scientific publishing, Scripps Institution of Oceanography, sociology, the right to know, Unitarian Universalism, UU Humanists, WHOI, wind power | Leave a comment

We are trying. And the bitterest result is to have so-called colleagues align themselves with the Koch brothers

I attended a 350.org meeting tonight. One group A group presenting there called “Fighting Against Natural Gas” applauded themselves for assailing Senator Whitehouse of Rhode Island for his supportive position on natural gas pipelines. Now, I am no friend of … Continue reading

Posted in Anthropocene, astrophysics, Boston Ethical Society, bridge to nowhere, carbon dioxide, carbon dioxide sequestration, Carbon Tax, chemistry, citizenship, climate, climate change, climate education, consumption, decentralized electric power generation, demand-side solutions, ecology, economics, energy reduction, engineering, forecasting, fossil fuel divestment, investment in wind and solar energy, IPCC, JAGS, meteorology, methane, model comparison, NASA, natural gas, NCAR, Neill deGrasse Tyson, oceanography, open data, physics, politics, population biology, Principles of Planetary Climate, Python 3, R, rationality, reasonableness, reproducible research, risk, science, science education, Scripps Institution of Oceanography | 4 Comments

Dynamic Linear Models package, dlmodeler

I’m checking out the dlmodeler package in R for a work project. It is accompanied by textbooks, G. Petris, S. Petrone, P. Campagnoli, Dynamic Linear Models with R, Springer, 2009 and J. Durbin, S. J. Koopman, Time Series Analysis by … Continue reading

Posted in Bayesian, geophysics, mathematics, maths, oceanography, open source scientific software, Python 3, R, science, sea level rise, state-space models, statistics, stochastic algorithms, time series | Leave a comment

Markov Chain Monte Carlo methods and logistic regression

This post could also be subtitled “Residual deviance isn’t the whole story.” My favorite book on logistic regression is by Dr Joseph Hilbe, Logistic Regression Models, CRC Press, 2009, Chapman & Hill. It is a solidly frequentist text, but its … Continue reading

Posted in Bayes, Bayesian, logistic regression, MCMC, notes, R, statistics, stochastic algorithms, stochastic search | 2 Comments

R vs Python: Practical Data Analysis

R vs Python: Practical Data Analysis (Nonlinear Regression).

Posted in Bayes, Bayesian, biology, climate change, ecology, environment, Python 3, R, statistics, Wordpress | Leave a comment

The designers of our climate

Originally posted on …and Then There's Physics:
Okay, I finally succumbed and actually waded through some of the new paper by Monckton, Soon, Legates & Briggs called Why models run hot: results from an irreducibly simple climate model. I…

Posted in astrophysics, bridge to nowhere, carbon dioxide, carbon dioxide capture, carbon dioxide sequestration, Carbon Tax, Carl Sagan, citizenship, civilization, climate, climate change, climate education, differential equations, ecology, economics, engineering, environment, ethics, forecasting, fossil fuel divestment, geoengineering, geophysics, humanism, IPCC, mathematics, mathematics education, maths, meteorology, methane, NASA, NCAR, Neill deGrasse Tyson, NOAA, oceanography, open data, open source scientific software, physics, politics, population biology, Principles of Planetary Climate, probabilistic programming, R, rationality, reasonableness, reproducible research, risk, science, science education, scientific publishing, sociology, solar power, statistics, testing, the right to know | Leave a comment

On nested equivalence classes of climate models, ordered by computational complexity

I’m digging into the internals of ABC, for professional and scientific reasons. I’ve linked a great tutorial elsewhere, and argued that this framework, advanced by Wood, and Wilkinson (Robert), and Wilkinson (Darren), and Hartig and colleagues, and Robert and colleagues, … Continue reading

Posted in approximate Bayesian computation, Bayes, Bayesian, biology, ecology, environment, forecasting, geophysics, IPCC, mathematics, maths, MCMC, meteorology, NCAR, NOAA, oceanography, optimization, population biology, Principles of Planetary Climate, probabilistic programming, R, science, stochastic algorithms, stochastic search | Leave a comment

illustrating particle filters and Bayesian fusion using successive location estimates on the unit circle

Introduction Modern treatments of Bayesian integration to obtain posterior densities often use some form of Markov Chain Monte Carlo (“MCMC”), typically Gibbs sampling. Gibbs works well with many Bayesian hierarchical models. The standard problem-solving situation with these is that a … Continue reading

Posted in Bayes, Bayesian, biology, mathematics, maths, population biology, probabilistic programming, R, statistics, stochastic algorithms | 1 Comment

Bayesian deconvolution of stick lengths

Consider trying to determine the length of a straight stick. Instead of the measurement errors being clustered about zero, suppose the errors are known to be always positive, that is, no measurement ever underestimates the length of the stick. Such … Continue reading

Posted in Bayesian, Gibbs Sampling, JAGS, mathematics, maths, optimization, probabilistic programming, R, statistics, stochastic algorithms, stochastic search | Leave a comment

The dp-means algorithm of Kulis and Jordan in R and Python

dp-means algorithm. Think k-means but with the number of clusters calculated. By John Myles White, in R. (Github link off that page.) By Scott Hendrickson, in Python. (Github link off that page.)

Posted in Bayesian, Gibbs Sampling, JAGS, mathematics, maths, R, statistics, stochastic algorithms, stochastic search | Tagged | Leave a comment

Blind Bayesian recovery of components of residential solid waste tonnage from totals data

This is a sketch of how maths and statistics can do something called blind source separation, meaning to estimate the components of data given only their totals. Here, I use Bayesian techniques for the purpose, sometimes called Bayesian inversion, using … Continue reading

Posted in Bayesian, BUGS, conservation, consumption, engineering, environment, Gibbs Sampling, JAGS, mathematics, maths, MCMC, MSW, politics, probabilistic programming, R, rationality, recycling, statistics, stochastic algorithms, stochastic search | Leave a comment

“The joy and martyrdom of trying to be a Bayesian”

Bayesians have all been there. Some of us don’t depend upon producing publications to assure our pay, so we less have the pressure of pleasing peer reviewers. Nonetheless, it’s all reacting to “What the hell are you doing? I don’t … Continue reading

Posted in Bayesian, BUGS, Gibbs Sampling, JAGS, MCMC, optimization, probabilistic programming, R, rationality, reasonableness, risk, SPSA, statistics, stochastic algorithms, stochastic search | Leave a comment

How fast is JAGS?

How fast is JAGS?.

Posted in BUGS, engineering, Gibbs Sampling, JAGS, mathematics, maths, MCMC, probabilistic programming, R, statistics, stochastic algorithms | Leave a comment

The zero-crossings trick for JAGS: Finding roots stochastically

BUGS has a “zeros trick” (Lund, Jackson, Best, Thomas, Spiegelhalter, 2013, pages 204-206; see also an online illustration) for specifying a new distribution which is not in the standard set. The idea is to couple an invented-for-the-moment Poisson density to … Continue reading

Posted in Bayesian, BUGS, education, forecasting, Gibbs Sampling, JAGS, mathematics, MCMC, probabilistic programming, R, statistics, stochastic search | Tagged , , | 3 Comments

HadCRUT4 version “HadCRUT.4.2.0.0” available as .RData R workspace or image

I’m happy to announce that I have made available the HadCRUT4 observational ensemble data as an .RData image for use with R. These were downloaded from the MetOffice Hadley Observations Web site. Detailed documentation is available on this page, with the … Continue reading

Posted in climate, climate education, ecology, environment, geoengineering, geophysics, Gibbs Sampling, oceanography, physics, R, science, statistics | Tagged , , , | Leave a comment