Ease of pulling weather data online by using Python for Energy Consumption Pattern Analysis and Demand Forecasting

Ease of pulling weather data online by using Python for Energy Consumption Pattern Analysis and Demand Forecasting  

Energy demand forecasting is part of process optimization in terms of daily capacity reservation and programming daily purchase of energy resource, stabilizing the stress on the lines, diversifying resources and lowering their cost with regard to spot and future markets, and planning future investments. In that sense, forecasting becomes crucial part of managing the company to survive in the sector through peak times or times when resources are constrained. 

Especially, during winter times, when the need for heating is high, residential heating peaks. Energy management and projection of daily demand, even hourly, would put much strain on financial and operational sides of any company ranging from wholesaler and distribution ones to power producer plants; such as the risk of falling into spot market, factored network operations charges due to wrong planning of supply, losing money on take-or-pay agreements in the long run and so on. Therefore, a methodology has been built up by the forecasters and statisticians regarding different type of segments including mainly households, industrial users and power producers. 

Since this topic is related to space heating during winter times, residential consumers are of focus in this analysis. In residential heating, energy consumption depends on the factors mainly as outside temperature, the isolation of buildings, energy efficiency of boilers/ACs, price, income, etc. Among all, outside temperature will mostly determine the energy consumed. Whenever the temperature is out of comfort range, meaning 18 ⁰­ C – 22 ⁰­ C, energy is consumed to heat up or cool down the room space. As the actual temperature diverges from one of these threshold temperatures, energy need for households increases. That being said, "Heating degree days", or "HDD", are a measure of how much (in degrees), and for how long (in days), outside air temperature was lower than a specific "base temperature" (or "balance point"). For example, for one day, if the outside temperature was 14 ⁰­ C, then the HDD would be 18 – 14 = 4. In that sense, as HDD increases, the heating need for space, eventually energy consumption, would increase, as well. Specifically, when we talk about natural gas consumption by boilers, in general, energy need per subscriber would look like as in the following graph: 

* Numbers for energy consumption / subscriber are arbitrary

Yet, when we analyze energy consumption data by using Eviews, we would treat data in time series format in seasons; winter period, spring period, summer period. Each period has its own consumer behavior model with regard to energy consumption pattern. In that sense, daily Natural Gas consumption data analysis is done as such; retrieve the temperature data for each day, then convert to HDD, and then match and compare the total energy consumed per subscriber with the relevant HDD data by running regression tool.

On Eviews, energy consumption regression graph would look like the following one:

This graph shows us how total energy consumption per subscriber in one day changes with regard to HDD of that day. Using the formula derived from this regression of energy and temperature data, companies can estimate the energy need by residents with regard to HDD vaue of coming days. 

Regarding the data needed to create time series and then run a regression, Energy companies can track energy consumption and subscriber data, yet tracking temperature data is difficult, and even some meteorological observatory institutions would ask for a fee to provide such a service. On the other hand, jotting down numbers from websites would take time, as well. At this point, a software called as Python would step in to bring digital data to your computer from the web. Python, as a programming language, needs codes to get the orders and serve you as an agent to capture the data. Since I am not a programmer, I would not talk about the coding procedure, yet I will give you the overall picture. First, one should have installed the relevant programs:

-          Install Python 2.7.3

-          Then install Spyder 2.1.11

-          Then install xlwt-0.7.3

*Warning: If you run on 64 bit computer, you may need different versions for these programs. 

Here, “xlwt” is a program to create excel files; as Python captures data with regard to the codes you entered, they will be formatted as excel files. Besides, the codes you entered to python file can be edited through IDLE. Regarding the temperature data, a format has been already placed on web so that Python will recognize the data format and create excel sheets as the process of capturing data starts. At this point, a website called as www.weatherunderground.com steps in. This website is highly popular one among those who are keeping an eye on weather conditions and compatible with Python users. The compatibility issue is a matter of formatting the data on websites as “json” files. On this website, max and min temperature data, wind direction and magnitude data, along with comments and other data are formatted as json files so that Python reads the data and brings them to your computer to format in an excel file. In that sense, for any province or city, one can find meteorological data in json format. Then, it is part of your programming skill to write the codes to capture historical temperature data or 10 days of forecast. Below is part of the code that is written to capture historical data for Muğla, a western city of Turkey. 

Likewise, the website also has daily json files for 10 days of forecast so that energy companies can forecast the related energy need change with regard to temperature forecast in any city. On excel sheet, the forecast data will look like as such: 

As you create your model on Excel, you can easily link your forecasting formula with the name of excel files you create each day by running the software. In conclusion, one leg (temperature data) of tri-pod (energy consumed, subscribers, temperature) on which residential energy consumption pattern analysis is mounted would also be digitally available in an easier format.