' Dr. Wilmut's Secondary Suite Survey Summary Critique - Full Text
Secondary Suites Community Summary needs to be reanalyzed
This report critiques Urban Systems Secondary Suites Community Survey Summary (Survey Summary). It was found that the Survey Summary did not process all the responses, did not report responses in a consistent manner, used a poor display format, did not fully extract the information in the data and draws an unsubstantiated conclusion from the data. The Survey Summary needs to be reanalyzed.
Observation 1
The written text states there were 995 responses. This is in agreement with the data files. However, the number used to construct the pie charts in Question 1 (Q1)and Q2 is 942. That is all the data was not processed. Fifty-three responses were not analyzed.
Conclusion
The data needs be reprocessed using all the data.
Observation 2
Questions 1,2,4,20,23,27 allow for “unsure/prefer not to answer”, while Questions 10,14,16,19 do not allow the” unsure/prefer not to answer choice”. This is inconsistent and makes it almost impossible to compare their corresponding pie charts.
Conclusion
For consistency, when reprocessing the “unsure/prefer not to answer” options, data should be omitted and the displays should be redone.
Observation 3
For this data, most statisticians would not use pie charts. Rather they would use bar graphs as it is more informative to the reader, especially when the categories have a natural progression.
Consider Question 20 which asks “Generally speaking, which of the scenarios listed in Table 2 do you prefer for permitting secondary suites?” Table 2 is a chart which describes the various scenarios where Scenario A-least restrictive to Scenario D-most restrictive with gradation between for Scenarios B and C. The result after omitting the “unsure/prefer not to answer” choice in the question is:
Scenario A B C D
Preference (%) 33.6 11.7 12.9 41.8
The above output format is very similar to how a bar graph would display the data. Statisticians describe the data as “U” shaped (A and D at the extremes contain most of the preferences compared to the middle (B and C). Compare the readability of this information display to the Question 20 pie chart displayed below.
Conclusion
Bar graphs rather than pie charts are a much more informative way to display this data.
Observation 4
Statisticians would explore this “U” shaped scenario distribution of preferences of Question 20 For example, does neighbourhood affect scenario choice? (The neighbourhood information is from Question 1)
A matrix is an array of numbers. Their rows run horizontally and their columns vertically.
The 7 rows (neighbourhoods) by 4 columns (scenarios,) below is an example of a matrix. In statistics it is called a contingency table.
The entries in this matrix are the responses to the various options. Consider for example the number 52 in the third row and fourth column. This means that 52 people in the Estevan neighbourhood chose scenario D as their response.
Scenarios
A B C D Row Totals
# 1 Henderson 38 20 17 56 131
#2 Uplands 8 4 3 11 26
#3 Estevan 48 18 16 52 134
#4 North Oak Bay 30 9 20 45 104
#5 Oak Bay Ave 19 7 6 20 52
#6 South Oak Bay 109 29 33 127 298
# 7 Harling Point 1 1 2 3 7
Column Totals 253 88 97 314 752
Percent 33.6 11.7 12.9 41.8 | 100
A visual study of the table (comparing responses across rows) shows neighbourhood does not affect scenario choice.
Conclusion
The individual neighbourhoods’ scenario preferences closely follow the total community scenario preferences i.e. neighbourhood does not affect scenario preference.
Next, does experience (live in suite, own suite, live on street with suites) affect scenario preference? The experience information is in Question 2.
The three rows (experience) by four columns (scenario) contingency table is:
A B C D Total
Live in 15 4 1 4 24
Own 47 17 19 7 90
On Street 114 44 46 211 415
-----------------------------------------------------
Total 176 65 66 222 529
Percent 33.3 12.3 12.5 42.0 100
of total
Visually one can see that those that live in or own a suite have a high probability of preferring Scenario A, while those respondents that live on a street with suites have a high probability of preferring Scenario D.
A statistical test called the Chi-Squared test for independence was run on this data. It clearly confirmed what was seen visually is correct.
Conclusion
People who live in or own a suite prefer weak legalization rules. Those who live on a street with suites, a much larger number, prefer strong legalization rules.
From these two questions, one can see how informative looking at the relationship between variables can be.
Observation 5
Under the heading Secondary Suite Program Scenarios Urban Systems author states “the relatively large number of respondents who selected Scenario A (i.e. least stringent regulations) understands that if the regulations are too imposing, then it will not encourage compliance and legalization of existing (and even new) secondary suites, and will not help with housing affordability in Oak Bay”. This conclusion presumably is based on the responses to the Scenarios responses (Question 20). But the question does not ask WHY respondents replied as they did. Where is the author’s evidence for this statement?
Conclusion
The quote above should be removed from a revised document unless evidence is given to substantiate it.
Next obvious steps are to examine the relationship between experience and: owner occupied (Q10), off-street parking (Q14); enforcement (Q16); compliance (Q19).
My five observations illustrate serious flaws and omissions in Urban Systems’ Survey Summary. As a benchmark if a student taking a university statistics course submitted such material, I assure the reader they would get a failing grade.
To properly understand the public’s wishes on “HOW” to legalize secondary suites Urban System’s analysis needs to be redone.
Another critical flaw Urban Systems made is that the survey questions do not ask the public’s views on “IF they want to legalize suites” as directed in the Official Community Plan. I have addressed a solution to this flaw in a companion submission “Urban Systems Secondary Suite Process Needs Major Correction”.
Mike Wilmut Ph D
Retired Professor of Mathematics and Statistics
----------------------------------------------------------------------------------------
Oak Bay Watch is a volunteer community association and its members have a variety of professional backgrounds in both the public and private sector.
*******Please help us continue to provide you with information about Community concerns and Council decisions and actions. Oak Bay Watch members also help community groups with their specific development concerns. Donate to Oak Bay Watch - even $5 or $10 dollars provides expenses for door- to- door handouts and helps us maintain our website. Oak Bay Watch is committed to ensuring the Community gets the full range of information on budget, governance and all key development issues – a well-informed opinion cannot be made without this.
(Please use Donate Button at bottom of oakbaywatch.com Home Page)
Keep informed and sign up for our newsletter – bottom of Newsletter Menu Item.
Secondary Suites Community Summary needs to be reanalyzed
This report critiques Urban Systems Secondary Suites Community Survey Summary (Survey Summary). It was found that the Survey Summary did not process all the responses, did not report responses in a consistent manner, used a poor display format, did not fully extract the information in the data and draws an unsubstantiated conclusion from the data. The Survey Summary needs to be reanalyzed.
Observation 1
The written text states there were 995 responses. This is in agreement with the data files. However, the number used to construct the pie charts in Question 1 (Q1)and Q2 is 942. That is all the data was not processed. Fifty-three responses were not analyzed.
Conclusion
The data needs be reprocessed using all the data.
Observation 2
Questions 1,2,4,20,23,27 allow for “unsure/prefer not to answer”, while Questions 10,14,16,19 do not allow the” unsure/prefer not to answer choice”. This is inconsistent and makes it almost impossible to compare their corresponding pie charts.
Conclusion
For consistency, when reprocessing the “unsure/prefer not to answer” options, data should be omitted and the displays should be redone.
Observation 3
For this data, most statisticians would not use pie charts. Rather they would use bar graphs as it is more informative to the reader, especially when the categories have a natural progression.
Consider Question 20 which asks “Generally speaking, which of the scenarios listed in Table 2 do you prefer for permitting secondary suites?” Table 2 is a chart which describes the various scenarios where Scenario A-least restrictive to Scenario D-most restrictive with gradation between for Scenarios B and C. The result after omitting the “unsure/prefer not to answer” choice in the question is:
Scenario A B C D
Preference (%) 33.6 11.7 12.9 41.8
The above output format is very similar to how a bar graph would display the data. Statisticians describe the data as “U” shaped (A and D at the extremes contain most of the preferences compared to the middle (B and C). Compare the readability of this information display to the Question 20 pie chart displayed below.
Conclusion
Bar graphs rather than pie charts are a much more informative way to display this data.
Observation 4
Statisticians would explore this “U” shaped scenario distribution of preferences of Question 20 For example, does neighbourhood affect scenario choice? (The neighbourhood information is from Question 1)
A matrix is an array of numbers. Their rows run horizontally and their columns vertically.
The 7 rows (neighbourhoods) by 4 columns (scenarios,) below is an example of a matrix. In statistics it is called a contingency table.
The entries in this matrix are the responses to the various options. Consider for example the number 52 in the third row and fourth column. This means that 52 people in the Estevan neighbourhood chose scenario D as their response.
Scenarios
A B C D Row Totals
# 1 Henderson 38 20 17 56 131
#2 Uplands 8 4 3 11 26
#3 Estevan 48 18 16 52 134
#4 North Oak Bay 30 9 20 45 104
#5 Oak Bay Ave 19 7 6 20 52
#6 South Oak Bay 109 29 33 127 298
# 7 Harling Point 1 1 2 3 7
Column Totals 253 88 97 314 752
Percent 33.6 11.7 12.9 41.8 | 100
A visual study of the table (comparing responses across rows) shows neighbourhood does not affect scenario choice.
Conclusion
The individual neighbourhoods’ scenario preferences closely follow the total community scenario preferences i.e. neighbourhood does not affect scenario preference.
Next, does experience (live in suite, own suite, live on street with suites) affect scenario preference? The experience information is in Question 2.
The three rows (experience) by four columns (scenario) contingency table is:
A B C D Total
Live in 15 4 1 4 24
Own 47 17 19 7 90
On Street 114 44 46 211 415
-----------------------------------------------------
Total 176 65 66 222 529
Percent 33.3 12.3 12.5 42.0 100
of total
Visually one can see that those that live in or own a suite have a high probability of preferring Scenario A, while those respondents that live on a street with suites have a high probability of preferring Scenario D.
A statistical test called the Chi-Squared test for independence was run on this data. It clearly confirmed what was seen visually is correct.
Conclusion
People who live in or own a suite prefer weak legalization rules. Those who live on a street with suites, a much larger number, prefer strong legalization rules.
From these two questions, one can see how informative looking at the relationship between variables can be.
Observation 5
Under the heading Secondary Suite Program Scenarios Urban Systems author states “the relatively large number of respondents who selected Scenario A (i.e. least stringent regulations) understands that if the regulations are too imposing, then it will not encourage compliance and legalization of existing (and even new) secondary suites, and will not help with housing affordability in Oak Bay”. This conclusion presumably is based on the responses to the Scenarios responses (Question 20). But the question does not ask WHY respondents replied as they did. Where is the author’s evidence for this statement?
Conclusion
The quote above should be removed from a revised document unless evidence is given to substantiate it.
Next obvious steps are to examine the relationship between experience and: owner occupied (Q10), off-street parking (Q14); enforcement (Q16); compliance (Q19).
My five observations illustrate serious flaws and omissions in Urban Systems’ Survey Summary. As a benchmark if a student taking a university statistics course submitted such material, I assure the reader they would get a failing grade.
To properly understand the public’s wishes on “HOW” to legalize secondary suites Urban System’s analysis needs to be redone.
Another critical flaw Urban Systems made is that the survey questions do not ask the public’s views on “IF they want to legalize suites” as directed in the Official Community Plan. I have addressed a solution to this flaw in a companion submission “Urban Systems Secondary Suite Process Needs Major Correction”.
Mike Wilmut Ph D
Retired Professor of Mathematics and Statistics
----------------------------------------------------------------------------------------
Oak Bay Watch is a volunteer community association and its members have a variety of professional backgrounds in both the public and private sector.
*******Please help us continue to provide you with information about Community concerns and Council decisions and actions. Oak Bay Watch members also help community groups with their specific development concerns. Donate to Oak Bay Watch - even $5 or $10 dollars provides expenses for door- to- door handouts and helps us maintain our website. Oak Bay Watch is committed to ensuring the Community gets the full range of information on budget, governance and all key development issues – a well-informed opinion cannot be made without this.
(Please use Donate Button at bottom of oakbaywatch.com Home Page)
Keep informed and sign up for our newsletter – bottom of Newsletter Menu Item.