Dataset containing anonymized energy and temperature data from homes in the Assendorp neighbourhood in Zwolle, the Netherlands

The dataset is currently in review to ensure it meet our standards for anonymization. Here, you can already view the metadata, so you know what you may expect.

• General info
• Subject Recruitment
• Inclusion criteria
• Data
• Status
• License
• Credits

This repository will contain an anonymized dataset comprising time series measurement data about energy and temperatures in residential homes (mostly) in the Assendorp neighbourhood in Zwolle, the Netherlands, obtained during the heating season of 2021-2022 in the Twomes research project.

Note: Git LFS is required to clone big CSV files

Subjects were recruited from the municipality of Zwolle during autumn 2021. Recruitment was primarily targeted at the Assendorp neighbourhood, via 50 Tinten Groen Assendorp, e.g. via a news message. In November 2021, additional subjects were recruited amongst colleagues and students of Windesheim University of Applied Sciences, e.g. via an internal message.

Subjects could volunteer to participate and give informed consent by filling out this generic online recruitment survey (this survey is also available in XML-format). Subjects that satisfied all inclusion criteria were also asked to fill out this online survey to provide their static Bluetooth MAC addressto us (this survey is also available in XML-format) and were asked to forward this latter survey to the other members of the household with a smartphone.

Inclusion criteria were:

• the home address lies in the municipality of Zwolle, the Netherlands;
• the home is equipped with an internet connection and wireless internet (Wi-Fi);
• the home is heated by a gas-fired heating boiler that is controlled by a thermostat and is not predominantly heated via other means;
• a smart energy meter is installed in the home;
• at least one of the occupants has an Android/iOS smartphone.

We documented our Data Management Plan online. The privacy policy (in Dutch) is available online as well in a layered structure: short summary, summary and full version.

In the sections below, the data pre-processing and data formats used in the data files will be described.

TODO: describe

We used the following measurement device types to collect data. Some devices consisted of a main device and one or two satellite devices.

All timestamps were measured in Unix time format, using device clocks regularly synchronized via NTP with the correct UTC time. Setting the local device clock to the proper UTC time via NTP was one of the first steps performed by the measurement devices after they were connected to the internet via the home Wi-Fi network of a subject. Each measurement device synchronized its device clock via NTP every 6 hours. Uploads of measurement data (which could contain more than one measurement) were timestamped both by the measurement device according to the local device clock and by the server. We did not yet check for deviations between the last device timestamp of a measurement upload and the upload timestamp at the server.

Timestamps were converted to a timezone-aware pandas.Timestamp value, in the Europe/Amsterdam timezone. In the csv files we use ISO 8601 format with time offset: YYYY-MM-DDThh:mm:ss±hhmm.

Raw measurements will be available in the folder /raw-measurements/ in three formats:

• twomes_raw_measurements.parquet: a single parquet file with all 23 homes for which we have more than 3 weeks data;
• 8nnnnn_raw_measurements.parquet: 23 parquet files, one for each home;
• 8nnnnn_raw_measurements.zip: 23 zipped csv files, one for each home;

All measurement data is structured according to the table below. By importing the parquet variant using pandas.read_parquet(), you automatically get a DataFrame wih the recommended indices and data types.

Alternatively, you can also read the zipped csv files, but this typically takes much longer. You can use the code below to end up with a DataFrame with the recommended indices and data types:

TODO: insert pandas.read_csv() code here

In the folder /raw-properties/ we will make various measured properties available in an 'unstacked' format with each property in its own column and an appropriate datatype. Similar to measurements, we will make data available in three formats:

• twomes_raw_properties.parquet: a single parquet file with all 23 homes for which we have more than 3 weeks data;
• 8nnnnn_raw_properties.parquet: 23 parquet files, one for each home;
• 8nnnnn_raw_properties.zip: 23 zipped csv files, one for each home;

All property data is structured according to the table below. By importing the parquet variant using pandas.read_parquet(), you automatically get a DataFrame wih the recommended indices and data types.

Alternatively, you can also read the zipped csv files, but this typically takes much longer. You can use the code below to endup with a DataFrame with the recommended indices and data types:

TODO: insert pandas.read_csv() code here

Below is a table that lists all properties that were measured, the data type in the raw-properties DataFrame, the measurement unit, the measurement interval, the source device and sensor that measured it, as well as the property name and value format as retrieved from the Twomes database.

Weather data was collected and geospatially interpolated using HourlyHistoricWeather from the Royal Netherlands Meteorological Institute (KNMI), based on average hourly values.

For all homes, we used the same location for geospatial interpolation of weather data: lat, lon = 52.50655, 6.09961, the center of the Assendorp neighbourhood in Zwolle, the Netherlands. Average values were converted from the source units to the units as indicated in the table below.

Preprocessing steps include:

• if available, use timestamps based on e_timestamp__YYMMDDhhmX or g_timestamp__YYMMDDhhmX for smart meter reading measurements, instead of the timestamp obtained from the twomes-p1-gateway-firmware;
• remove duplicate measurements (including duplicates that arise from the previous step);
• filter out smart meter resets: for each property with _cum in the name, for each time series of a particular home for that property, typically by taking a diff() of series, followed by setting any negative values to zero, then taking the cumsum() of the series;
• calculate energy flow rates for meter readings: for each property with _cum in the name, for each time series of a particular home for that property, take a diff() of the series and assign it to a series with the same name, but without _cum in the name;
• remove absolute outliers, i.e. measurement values smaller than the value in the column Min or larger than the value in the column Max in the table below;
• remove statistic outliers, i.e. measurement values with an absolute z-score higher than the value indicated in the Sigma column in the table below;
• interpolate measurements to intervals of 15 minutes, but to not interpolate between measurements that are 60 minutes apart or more;
• calculate derived properties as a combination of other properties, as indicated in the column Calculation in the table below.

All column values in a preprocessed data frame represent the average during the interval that starts at the timestamp indicated.

Dataset is: collected, anonymization-in-progress

This data is made available under the CC BY 4.0 by the Research group Energy Transition, Windesheim University of Applied Sciences

• Henri ter Hofte · @henriterhofte · Twitter @HeNRGi
• Stichting 50 Tinten Groen Assendorp

Thanks go to those who are the ultimate source of this dataset:

• all anonymous subjects who volunteered to make their measurement data available

We use and gratefully acknowledge the efforts of the makers of the following source code and libraries:

• HourlyHistoricWeather, by @stephanpcpeters, licensed under an MIT-style licence