CCMDB Wiki

Overstay2 colour: Difference between revisions

← Older edit
JMojica (talk | contribs)
CCMDB_legit, admin, approved, Administrators
2,787 edits
JMojica (talk | contribs)
CCMDB_legit, admin, approved, Administrators
2,787 edits
No edit summary
 
(9 intermediate revisions by 2 users not shown)
Line 7: Line 7:
* A  2 x 2 classification  table for incrementing probability levels from 0.001 to 0.200 helps visualize the performance of the predicted results with actual outcomes.  
* A  2 x 2 classification  table for incrementing probability levels from 0.001 to 0.200 helps visualize the performance of the predicted results with actual outcomes.  
* Sensitivity (true positive rate)  is the proportion of actual positives correctly identified by the predicted results. High sensitivity means few false negatives.   
* Sensitivity (true positive rate)  is the proportion of actual positives correctly identified by the predicted results. High sensitivity means few false negatives.   
* Specificity (true negative rate) is the proportion of actual negatives correctly identified by the predicted results.  High specificity means few false positives.   
* Specificity (true negative rate) is the proportion of actual negatives correctly identified by the predicted results.  High specificity means few false positives.  
* Positive Predicted Value (PPV) is the likelihood that a positive result is a true positive.
* Negative Predicted Value (NPV) is the likelihood that a negative result is a true negative.
* False Positive (FP) Rate is the proportion of actual negatives but predicted incorrectly as positives.
* False Negative (FN) Rate is the proportion of actual  positives but predicted incorrectly as negatives.
* Optimal threshold is the probability level which shows equal or close to equal  sensitivity or specificity in the testing data set.  This threshold (also called cut-off) is being used to classify patients.  
* Optimal threshold is the probability level which shows equal or close to equal  sensitivity or specificity in the testing data set.  This threshold (also called cut-off) is being used to classify patients.  
* The optimal threshold by site is below:
* The optimal threshold by site is below:
{| class="wikitable notsortable"
{| class="wikitable notsortable"
  !Site !! Data Set!! Optimal Cut-off !! Sensitivity !! Specificity  
  !Site !! Data Set!! Optimal Cut-off !! Sensitivity !! Specificity || PPV || NPV || FP Rate || FN Rate
|-
|-
| GGH  || Training|| 0.077|| 73.60 || 74.50
| GGH  || Training|| 0.076|| 74.2 || 74.3 || 16.1 || 97.7 || 25.7 || 25.8
|-
|-
| GGH  || Validation|| 0.080|| 74.66 || 74.69
| GGH  || Validation|| 0.079|| 74.4 || 74.5 || 16.4 || 97.7 || 25.5 || 25.6
|-
|-
| HSC  || Training|| ||  ||   
| HSC  || Training|| ||  ||  || || || ||
|-
|-
| HSC  || Validation||  ||  ||   
| HSC  || Validation||  ||  ||  || || || ||
|-
|-
| STB  || Training|| ||  ||   
| STB  || Training|| ||  ||  || || || ||
|-
|-
| STB  || Validation||  ||  ||   
| STB  || Validation||  ||  ||  || || ||
|-
|-
|}
|}


This would have resulted in a assigning
There was a concern that this would have overwhelmed the [[Overstay2 processes on the units to reduce discharge delay]] so we considered a [[#pragmatic threshold]].
'''''xxx% '''''
of patients a "red". This would have overwhelmed the [[Overstay2 processes on the units to reduce discharge delay]] so we needed to determine a [[#pragmatic threshold]].


=== Pragmatic threshold ===
=== Pragmatic threshold ===
We considered whether we should change the threshold from the [[#Optimal threshold]] to reduce the number of patients who are assigned as red,  in addition to modifying the [[Overstay2 processes on the units to reduce discharge delay]] .
{| class="wikitable notsortable"
!Site !! Data Set!! Pragmatic Cut-off !! Sensitivity !! Specificity || PPV || NPV || FP Rate || FN Rate
|-
| GGH  || Training|| 0.094|| 67.2 || 79.0 || 17.5 || 97.3 || 21.0|| 32.8
|-
| HSC  || Training|| ||  ||  || || || ||
|-
| STB  || Training|| ||  ||  || || || ||
|-
|}
{{collapsable | always = I think this can go, just leaving it until confirmed | full=
We used the following process to set a red/yellow threshold that our [[Overstay2 processes on the units to reduce discharge delay]] can function with.  
We used the following process to set a red/yellow threshold that our [[Overstay2 processes on the units to reduce discharge delay]] can function with.  


Line 42: Line 57:
** Use Sensitivity or True Positive Rate which denotes out of delayed discharges, how many are correctly identify as delayed.
** Use Sensitivity or True Positive Rate which denotes out of delayed discharges, how many are correctly identify as delayed.
{| class="wikitable notsortable"
{| class="wikitable notsortable"
  !Site !! Set !!Threshold Cut-off !! Sensitivity !!Threshold Cut-off!! PPV
  !Site !! Data Set!!Treshold Cut-off !! Sensitivity !! Specificity || PPV || NPV
|-
| GGH  || Training|| 0.114|| '''60.3''' || 83.5 || 19.5 || 96.9
|-
| GGH  || Training|| 0.076|| '''75.3''' || 74.0 || 16.2||97.8
|-
| GGH  || Training|| 0.061|| 82.2 || 69.1 || '''15.0''' ||98.3
|-
|-
| GGH  || Training|| 0.114|| 60.3 || 0.061 || 15.00
| GGH  || Training|| 0.090||69.4 || 77.4 || '''17.0''' || 97.2
|-
|-
| GGH  || Training|| 0.076|| 75.3 || 0.090 || 17.00
| GGH  || Validation|| 0.110|| '''60.5''' || 82.3 || 18.9 || 96.7
|-
|-
| GGH  || Validation|| 0.110|| 60.5 || 0.060 || 15.18
| GGH  || Validation|| 0.078|| '''75.2''' || 74.3 || 16.7 || 97.8
|-
|-
| GGH  || Validation|| 0.078|| 75.2 || 0.084 || 17.01
| GGH  || Validation|| 0.060|| 82.3 || 68.5 || '''15.2''' || 98.3
|-
|-
| HSC || Training|| ||   || ||
| GGH || Validation|| 0.084|| 73.0 || 75.6 || '''17.0''' || 97.6
|-
|-
| HSC  || Validation||  ||  || ||
| HSC  || Training||  ||  ||  || ||
|-
|-
| STB || Training|| ||   ||  ||
| HSC || Validation|| || ||  || ||
|-
|-
| STB  || Validation||  ||  ||  ||
| STB  || Training|| ||  ||  || ||
|-
| STB  || Validation||  ||  ||  || ||
|-
|-
|}
|}
Line 66: Line 89:
==== Setting threshold based on these goals ====
==== Setting threshold based on these goals ====
{{DJ | Emailed 2025-06-08: we need to understand the mechanism we used to set the threshold. To be updated when clear. [[User:Ttenbergen|Ttenbergen]] 17:57, 8 June 2025 (CDT)
{{DJ | Emailed 2025-06-08: we need to understand the mechanism we used to set the threshold. To be updated when clear. [[User:Ttenbergen|Ttenbergen]] 17:57, 8 June 2025 (CDT)
}}
}}
}}


== Log ==
== Log ==
* 2025-06-13 - continuous model no better, confirmed using threshold of 0.077 for now
* 2025-06-11 - meeting with DR and JM to discuss, considered continuous rather than binary model, Julie will generate
* 2025-06-08 - email to determine how to set colour in that new context
* 2025-06-08 - email to determine how to set colour in that new context
* 2025-05-27 - Julie finalized an updated [[Overstay2 scoring model for Grace Hospital]]
* 2025-05-27 - Julie finalized an updated [[Overstay2 scoring model for Grace Hospital]]
* 2025-02-?? - we realized we needed separate [[Overstay2 scoring models]] for each site
* 2025-02-24 - we realized we needed separate [[Overstay2 scoring models]] for each site
* 2025-02-21 - [https://q.tenbergen.ca/index.php?title=2025-02-21_JM_DR_Overstay_model_decision_and_cut-off 2025-02-21 JM DR Overstay model decision and cut-off] - We will use Model 8 at cut-off of 0.069  
* 2025-02-21 - [https://q.tenbergen.ca/index.php?title=2025-02-21_JM_DR_Overstay_model_decision_and_cut-off 2025-02-21 JM DR Overstay model decision and cut-off] - We will use Model 8 at cut-off of 0.069  
* 2025-02-20 - [https://q.tenbergen.ca/index.php?title=2025-02-20_DR#eval_the_Overstay2_scoring_model decided] that we want
* 2025-02-20 - [https://q.tenbergen.ca/index.php?title=2025-02-20_DR#eval_the_Overstay2_scoring_model decided] that we want
Retrieved from "https://ccmdb.kuality.ca/index.php?title=Overstay2_colour"