About Model Error Calculations

Before you use the model error feature on our site you should understand how the data is calculated. It is important to note that all model error data on this site was calculated by us. The error values are not official values. The model error data you see on our site is only made up of the models that were released by the National Hurricane Center (NHC) into the public model guidance folder ("aid_public" folder) in the Automated Tropical Cyclone Forecast (ATCF) system. This means that some models, including ones frequently used, are not included in this error calculation. Please also keep in mind that every storm is different. One model is not always best or there would not be a need for more than one model. You can learn more about forecast models here at the NHC.

When we process data live, sometimes additional model data is released for prior runs. Our site does not go back and recalculate past data for a previous run. This means that sometimes, model error data will be incomplete since some models, and sometimes entire runs, are missing. The missing data will only be added later if we add the missing data in historical mode after the storm ends.

We offer model data through 174 hours on our site. 168 hours is 7 days. To allow 7 full days of late cycle data, an extra 6 hours is added. This means that early cycle data will be through 7 days and 6 hours when available. Error data beyond that would be less useful due to the larger error and fewer models that offer data beyond 174 hours.

There are two very important things to know about the model error data on our site. The first thing you should know is that early and late cycle models are compared to each other in a method that is non-homogeneous. Comparing early cycle models to late cycle models is not a completely equal comparison since late cycle model data was actually initialized six hours prior at or near the previous storm position. Our error calculations do not point out what models are early cycle and which are late cycle. The second thing you should know is that model error data that is calculated in real time on an active storm is calculated differently on our site from model error data that is calculated for a storm that is added to our site after it was active. The zero hour error interval data for late cycle models is instead associated with its initialization time rather than its release time, meaning the data will appear six hours prior to when it would have had the data been processed by our site live. We will cover these two topics later. (We sometimes recreate all the model data again for a storm after it ends, meaning model error would then appear differently than it had when the storm was active.)

Model error data on our site can be displayed in table or chart form. In table form you have a variety of options. There are two error types. The first is the "Single Run" error type. This represents the error data for a single model run. The second is the "Average (Entire Storm)" error type. This represents the combination of all the "Single Run" data. The error date might not be what you expect and we will now go over the error date for situations where data was processed live on our site and situations where data was processed after the storm was ended in what we call "historical mode".

We will first talk about the "Single Run" error type, though the "Average (Entire Storm)" error type is the combination of all "Single Run" error distance and intensity error data over the life of the storm.

The error date represents the initialization time of early cycle model data. That is also the time that the early cycle and late cycle model data was available to the forecaster. However, late cycle model data was actually initialized six hours prior. The difference between early and late cycle models is not noted in the model error table or chart. It is however important to realize that late cycle models that appear can not be fairly compared to early cycle models which is why we call it a non-homogeneous comparison. That topic is covered by the NHC here. However, please only refer to our product description when it comes to other aspects of how model error is calculated when using our model error data as it may differ from what they describe. While the zero hour error of an early cycle model is compared with the storm position as of that error time, late cycle model data at the zero hour error is compared with the storm position six hours prior, when available. All other hourly error values, for both early and late cycle models, use the storm position as of the selected error date to calculate model error values.

The error date is the date at which early cycle model data is compared to the storm position as of that date. Late cycle model data released at that error date is actually initialized six hours prior which means all late cycle error data is shifted. The late cycle zero hour error data is not being compared to the current storm position and is not the 0 hour error at the date you selected. It is actually compared to the previous storm position six hours ago, when available. It is important to note that the hour interval error can only be directly applied to the date you have selected if it is an early cycle model where the initialization time matches the error date. For late cycle models, the zero hour error is actually from the previous storm position six hours ago compared to the zero hour forecast from the model available in the latest released run, which had an actual initialization six hours prior to the current error date. The six hour error is from the current storm position compared to the six hour forecast from the model available in the latest released run, which again had an actual initialization six hours prior to the current error date. This can be best seen in the upcoming chart.

The chart below shows an example of how model error data is calculated for position error when data is processed by our site in real time. If any data needed for the calculation at a specific hour error interval is not available then that particular position on the chart will have a "-" to denote that the calculation is not available for that position. Some models do not have model data that goes out every six hours. That means that data, when available, may only be available at twelve hour intervals. For those types of models we do not assume what the model forecast will be between the twelve hour intervals by calculating the midpoint. The storm may curve and/or travel at a different speed at times which is why we don't calculate the midpoint.

Example Error Date: September 1st at 18Z

Model Type	0 hr Error	6 hr Error	12 hr Error	18 hr Error
Early Cycle	Distance from the current storm position (at 18Z) to the current model run's 0 hour initialization. (The current model run will have an initialization time at 18Z.)	Distance from the current storm position (at 18Z) to the model run 6 hours ago's (12Z's) 6 hour model forecast position. (The model run 6 hours ago will have an initialization time at 12Z.)	Distance from the current storm position (at 18Z) to the model run 12 hours ago's (6Z's) 12 hour model forecast position. (The model run 12 hours ago will have an initialization time at 6Z.)	Distance from the current storm position (at 18Z) to the model run 18 hours ago's (0Z's) 18 hour model forecast position. (The model run 18 hours ago will have an initialization time at 0Z.)
Late Cycle	Distance from the storm position 6 hours ago (at 12Z) to the current model run's 0 hour initialization. (The current model run will have an initialization time at 12Z, but it was not released until 18Z.)	Distance from the current storm position (at 18Z) to the current model run's 6 hour model forecast position. (The current model run will have an initialization time at 12Z, but it was not released until 18Z.)	Distance from the current storm position (at 18Z) to the model run 6 hours ago's 12 hour model forecast position. (The model run 6 hours ago will have an initialization time at 6Z, but it was not released until 12Z.)	Distance from the current storm position (at 18Z) to the model run 12 hours ago's 18 hour model forecast position. (The model run 12 hours ago will have an initialization time at 0Z, but it was not released until 6Z.)

The table above explains how model error is calculated at each hourly error interval. From the last hour error intervals you should be able to understand how the model error data is calculated through the full 174 hours, when available. Additionally, intensity error data is generated the same way, only rather than storm position and model forecast positions we use storm intensity and model forecast intensity for models that give intensity. The default error distance unit is nautical miles. However, you can also choose statute miles or kilometers in the table. (The graphical chart only displays nautical miles.) The distance between two points is calculated by determining the great circle distance. The great circle distance is the shortest distance between two points on a sphere. An optional feature with the "Single Run" error type is displaying the error bearing. There are two different error bearings that you can choose from. The "Constant (rhumb line)" bearing is the bearing we use for other bearing calculations on our site, such as what direction a storm is located from a city. The bearing is calculated along a rhumb line. A rhumb line is the path between two points if you were to draw a straight line on a map projection such as a Mercator projection. Along this line we can determine the constant bearing between the two points. From that we can then determine the cardinal direction. For example, if the bearing is 180 degrees and the error distance is 75 nautical miles, then that particular model forecast that the storm would be 75 nautical miles south of the actual storm position. The "Initial (great circle)" bearing is the initial bearing that you start off at if you traveled along a great circle path between the two points. On a Mercator map projection a great circle path would appear as a curve, with the curve being more pronounced the further the distance is between the two points The problem with this bearing is that the bearing you start at will change as you travel along the great circle path. Since the bearing along a great circle path varies, we do not provide a cardinal direction like we do for the constant bearing option. You will probably not want to use the initial bearing for anything. We only include it since we do our distance calculations along a great circle path. While there may be little use for the initial bearing, we could not think of any use for calculating the model error distance along a rhumb line. You can learn more here about how our site performs distance calculations.

The "Single Run" error type simply provides a single run's model error data. The "Average (Entire Storm)" error type combines all data from every "Single Run" error type. For each run where there is model data for a specific error hour interval it is added up and divided by the number of instances, which we call cases. You can view how many cases there are by selecting that option.

For models at the zero hour error interval keep in mind that some models are given the start position that the model should be run from and others are not. This makes the zero hour position not as helpful when it comes to an average error. However, it is nice to know how well or poorly a model did at initializing the storm position. If a model consistently has a zero hour position error of zero, it is probably given the initialization position. It is the same for intensity error.

Late cycle model error data for the zero hour error interval that was created by our site after the storm was active will always appear six hours sooner than it would have if data for the storm was generated by our site in real time. All other model error data will appear as you would expect. The reason for this is that our site does not know what models are early cycle and which are late cycle. While we could compile a list of all late cycle models, more could be added at any time. Rather than take the chance of getting it wrong, we choose to process all model data by the valid time, the initialization time, rather than by the release time. For example, if model data was processed by our site live at 18Z we would have the latest early cycle models, initialized at 18Z, and the latest late cycle models, initialized at 12Z, though not released until 18Z. The error date listed on the model error page would be 18Z. When our site processes data for a storm after it was active we only know the initialization time, not the release time. Therefore, if we were to process data for this same example after the storm was active, the 18Z early cycle models would be listed with an 18Z error date. The late cycle models however would actually be associated with 12Z instead however. If you were to view the error date at 12Z you would see that the zero hour error data for the late cycle models are now located there. For the 18Z error date you would actually see the 0Z late cycle models for the zero hour error. This is because all late cycle model data is now associated with the run six hours prior since the data is organized by initialization time in historical mode rather than release time in live mode. Other data, beyond the zero hour error for late cycle models will still appear as you would expect. This is because in the example above the late cycle error data that is associated with its initialization time of 12Z will only compare zero hour forecast data to the storm position at that time. It will not be until 18Z that six hour forecast data from the 12Z late cycle initialization would be compared to the storm position at 18Z. It will not be until 0Z that twelve hour forecast data from the 12Z late cycle initialization would be compared to the storm position at 0Z. And so on. If you compare that to what would happen when displaying six hour, twelve hour, and on, error data, you would find that the data would be the same past the zero hour error data had the storm been processed live. If the storm had been processed live, the 12Z late cycle data would have been processed when it was released, at 18Z. At 18Z, the latest error data for the zero hour error would be compared to the storm position six hours prior at 12Z. The latest error data for error hour six would be compared to the storm position at 18Z. It would not be until six hours later, at 0Z, that the twelve hour forecast data would be compared to the storm position at 0Z. This is exactly how the live system works.

Basically, you probably should avoid looking at zero hour error data when viewing a storm's historical data.

You may wonder why we process data after a storm was active. If we redesign something that changes the file structure on our site or add a new feature that we would like to see available for data we have already processed, we need to reprocess the data. Sometimes additional model data is added or sometimes model data is sometimes even changed if it had an error, after the model data has been processed live on our site. That is why if you were to actually compare the output created live to an output created after the storm was active, some data might be different beyond the zero hour error differences noted above. Make sure to keep in mind that data for a storm that was processed live on our site might be recreated later after the storm was active for these reasons. The other reason we may add data after a storm was active is to have data available for significant historical storms in the past that we are able to get model data from in case anyone wanted to use that data for research.

Intensity error allows you to determine how a model does with forecasting intensity. Some models you will notice appear very bad at forecasting intensity. The reason why is that it is not forecasting peak intensity. If a storm is a hurricane and the model intensity is consistently initialized very low then it may be one of those models. Intensity error is given as positive or negative for the "Single Run" error type. If the value is positive it means the model forecast the intensity to be that much higher, in the intensity unit you have selected, than the actual storm intensity. Meaning, it thought the storm would be stronger than it actually turned out to be. If the value is negative it means the model forecast the intensity to be that much lower, in the intensity unit you have selected, than the actual storm intensity. For the "Average" error type, the intensity error is given as a positive number. For example, if you have two cases (two single runs) and one was where the intensity error was -12 and the other 6, the absolute value is used for the "Average" error type. The "Average" error type would be calculated by this equation (12 + 6) / 2 to get 9. There would be no great way to use a negative value for the "Average" error type which is why the absolute value is used. If there were two cases where you had -6 for one and 6 for the other, the average should be displayed as 6 and not that there was no intensity error.

The default intensity unit is knots. However, you can also choose mph (miles per hour), km/h (kilometers per hour), or m/s (meters per second) in the table. (The graphical chart only displays knots.)

For both error types we calculate the average error for each hour error interval. Every model error for that interval is added up and then divided by the total number of models that had data available for that position or intensity error, depending on which average row you are looking at. For intensity error, the absolute value of each model error is added up. If the model did not have data available we do not include it. (Meaning it had "-".) Model data that had an error of 0 at that interval is included in the average row.

Average Error heat map colors are designed to visually make it easier to see what models are performing better for a storm at specific error intervals and as a whole for the storm. It is not based on any official method.

To get each color for this type of heat map the model error for a particular interval is compared to the error for that hour interval in the average row for that interval. For example, to get a color for the heat map for a position error we divide the model error for a particular interval, 95 nautical miles for example, divided by the total model error for that hourly error interval located in the position average row, 100 nautical miles for example. 95 divided by 100 is 0.95. Since the error is less than the average 0.95 is then subtracted from 1 to get 0.05. That multiplied by 100 gives you 5%. For our heat map color we would assign a very light green color to indicate that the model is between 0.0001 and 9.9999% below the average error for other models at that error interval. Now for another example we will use 105 as the model error for a particular interval and 100 for the average again. 105 divided by 100 is 1.05. Since the error is greater than the average 1 is subtracted from 1.05 to get 0.05. That multiplied by 100 gives you 5%. For our heat map color we would assign a very light red color to indicate that the model is between 0.0001 and 9.9999% above the average error for other models at that error interval. If a model is exactly the same as the average, it will be colored white. You can see a heat map image on the page that helps you to tell what each color represents. The intensity error works in the same way, we just use the absolute value of anything negative.

Error Trend heat map colors are designed to visually make it easier to see how a model is trending from the selected hours ago to the current selected error date. It is not based on any official method.

To get the error trend the error value from the amount of hours ago you select is subtracted from the error value at the error date that is currently selected. For example, if you want to get the trend from 24 hours prior to the selected error date to the selected error date, then select "24 hrs" as the "Trend". For this example the position error value 24 hours prior to the selected error date was 50.0 nautical miles at the 6hr error interval for a particular model. For that same model at the 6hr position error interval the current error value is 100.0 nautical miles. If you have opted to "Show Error Trend Text" you will see "+50.0" to show that since 24 hours prior to the current error date to the selected error date the error is now 50 nautical miles greater than it was.

The preceding example did not specify what error type it was but there is a huge difference between the two that you must understand. If you had that example for the "Single Run" error type that "+50.0" would mean that over the past 24 hours the error trend has literally doubled. (We'll go over that more after we get to the complex error type.) The "Average (Entire Storm)" error type is different and more complex. For this error type a model would have a large error over the past 24 hours for the average at the 6hr error interval to increase by "+50.0". Lets say that there is a case for each run up until and including the selected error date. (18 hours prior, 12 hours prior, 6 hours prior, and the average error for the selected error date) That means an additional 4 cases. Since the storm started, lets say there are now 8 cases for the average error. In this example, lets say we calculated that 50 nautical mile average position error by adding up each of the first 4 cases and then dividing by 4. The formula in the example being ((35 + 40 + 60 + 65) / 4) which equals 50. Over the next 4 cases, the error grew substantially. Those next 4 cases had 115, 135, 185, and 165 nautical mile errors. The formula to get the average error, ((35 + 40 + 60 + 65 + 115 + 135 + 185 + 165) / 8), equals 100. However, the actual average over the last 24 hour period was 150 nautical miles and that number is not shown to you. It is important to note what our product actually displays, which is always using the average error for the entire storm to date in the calculations when you are using the Average error type, not simply the average error over the time period you are looking to see a trend. When you look at a trend for the "Single Run" error type, then you are looking at single values where it is more straightforward since you are not counting many different cases. For that error type, lets repeat the same example with the same initial figures again. 50.0 nautical miles at the 6hr error interval 24 hours prior to the selected error date and 100.0 nautical miles for the same model at the 6hr position error interval for the selected error date. That "+50.0" is a lot more straightforward since 24 hours ago the actual position error was 50 nautical miles and now it is 50 nautical miles greater. It has literally doubled and you can easily tell that.

When there is no previous data for a particular square, there will be no color when using the heat map. There will also be no positive or negative number since a trend could not be calculated. For the Average error type we also do the same thing if there are not additional cases to compare, only we add the text "Same Cases". This means that for the Average error type if there were 8 cases 6 hours ago, and 6 hours was the selected trend, and there was no new case this run then there is nothing new to report. We do not want to even say "0.0" since there was not an additional case that could have possibly resulted in the trend actually being "0.0".

The intensity trend is more complex than the position trend when it comes to the "Single Run" type where intensity values can be negative. For the "Single Run" intensity trend text calculation, negative values are no longer used in the calculation. Our site has handled this differently in the past. The way it works now is that if, for example, 24 hours prior to the selected error date we had a value of -5 knots and for the selected error date we have 5 knots, the trend would actually be "0.0" knots. This is how our site originally did it before I changed it so the trend would say "+10.0" knots for this circumstance. I decided to change it back so that it once again would say "0.0" knots. The reason why is due to how trend text for position error is done. For that we do not have signs. A model might be off to the north, to the south or any direction. We cannot assign a sign to it. If the model is 50 nm off to the north and then off 50nm to the south, the trend would be "0.0" nm. This same principle is once again carried over to the intensity trend text. While it was 5 knots below and then 5 knots above it is still just 5 knots off each time and therefore the trend should be "0.0" knots. My concern previously was that people would not be able to easily realize that there was a change of sorts, since if it had been 5 knots before and 5 knots now, there really would be no change of any kind. However, in the end I feel the "+10.0" knots would be even more confusing. When you enable the additional info that pops up when you hover your mouse over data in the table you can learn more about that "0.0" knots. The color for the previous example would be white since while there was a change of sorts, the error was still 5 knots both times and there is no net change, it is just that at first the model thought the storm would be 5 knots weaker than it was and then it thought it would be 5 knots stronger than it actually was. There is really no good way I can think of to display the intensity color as anything but white for this example. The methodology behind the intensity error trend text and the intensity error trend heat map calculation for the "Single Run" error type may change again in the future based on feedback, though I think at this point I like the way it is.

The percentages for the Average Error heat map work in a similar manner for the trend heat map. When it comes to negative error the absolute value is used. If you have 10 knots 24 hours prior to the selected error date and 9 knots for the selected error date, then the formula used is 9 divided by 10 which equals 0.9. That converts to 10% below the previous error so we assign the error trend heat map color a slight green color. If one or both of those values were negative the absolute value would be used. Whether it was 10 knots above or 10 knots below, the error is 10 knots.

The trend text is in whatever unit you have selected for the section you are looking at. So if you are using nautical miles for position error and knots for intensity error, then the trend text will be in their respective units for each section.

Intensity error can be deceiving when it comes to some models. Some models do not initialize with the maximum wind speed. To hide those models you can enter a wind speed, in the intensity unit you have selected, that the zero hour error must be equal to or less. If a model is way off on its initial position for a strong storm, and is often that way, then you may want to hide it when comparing it to models that initialize with something close to the initial intensity. The GFS ensemble members and some global models are often good examples of models that do not initialize with the maximum wind speed. (They also don't try to forecast the maximum wind speed.) These models are still included for those that know how to use the data.

Models that are hidden in the intensity error section are not included in the Average row calculation. For the "Single Run" error type, the absolute value is used, so if you enter 10 knots, something with 11 or -11 knots of error for the zero hour intensity error would be both hidden.

Just because model data is not available for a position does not mean that error data cannot be calculated. Model error data is always compared to the CARQ position, the center position, located in the model file unless it is not available. In that case, model error data is compared to the best track position. Any models previously available that made a forecast for a time when models would normally come out can have error data generated even if new models do not come out. If 48 hours ago a model forecast the storm to be in a certain position we can still check to see if it was correct by comparing the data to the best track position. (The same applies to intensity.) As soon as best track data is available our site processes model error data on it. If new model data is available later, we then process model data on it and recreate the model error data. In fact, if previous CARQ positions become available for a position that did not previously have CARQ data (this comes in the model file as a negative hour for CARQ denoting how many hours prior the position was for.), we will recreate the model data automatically for those positions as well. The reason why we create the model error data for the best track position first and then the CARQ position is so that if the storm ends we make sure that we have created the error data from any best track position updates which may have been made after model data was last generated for the storm. In order to make sure that the last data is always created, we need to always create the model error data using the best track position first, which does use a lot of system resources to do and is for the vast majority of the time not needed.