Overstay2 colour: Difference between revisions

(17 intermediate revisions by 2 users not shown)

Line 3:

The colour is based on a probability threshold that is set to balance the number of patients followed under the more elaborate [[Overstay2 processes on the units to reduce discharge delay#red process]] wile minimizing the number of patients who would overstay by more than 10 days that are not captured. It was set separately for each of the [[Overstay2 scoring models]], so '''see those pages for the actual threshold'''s.

== ~~Goals/considerations~~ ==

== Method of setting threshold ==

The ~~thresholds were set~~ to ~~stratify~~ patients ~~into different~~ [[Overstay2 processes on the units to reduce discharge delay]] based on the following considerations:

=== Optimal threshold ===

* A 2 x 2 classification table for incrementing probability levels from 0.001 to 0.200 helps visualize the performance of the predicted results with actual outcomes.

* Sensitivity (true positive rate) is the proportion of actual positives correctly identified by the predicted results. High sensitivity means few false negatives.

* Specificity (true negative rate) is the proportion of actual negatives correctly identified by the predicted results. High specificity means few false positives.

* Positive Predicted Value (PPV) is the likelihood that a positive result is a true positive.

* Negative Predicted Value (NPV) is the likelihood that a negative result is a true negative.

* False Positive (FP) Rate is the proportion of actual negatives but predicted incorrectly as positives.

* False Negative (FN) Rate is the proportion of actual positives but predicted incorrectly as negatives.

* Optimal threshold is the probability level which shows equal or close to equal sensitivity or specificity in the testing data set. This threshold (also called cut-off) is being used to classify patients.

* The optimal threshold by site is below:

{| class="wikitable notsortable"

!Site !! Data Set!! Optimal Cut-off !! Sensitivity !! Specificity || PPV || NPV || FP Rate || FN Rate

|-

| GGH || Training|| 0.076|| 74.2 || 74.3 || 16.1 || 97.7 || 25.7 || 25.8

|-

| GGH || Validation|| 0.079|| 74.4 || 74.5 || 16.4 || 97.7 || 25.5 || 25.6

|-

| HSC || Training|| || || || || || ||

|-

| HSC || Validation|| || || || || || ||

|-

| STB || Training|| || || || || || ||

|-

| STB || Validation|| || || || || ||

|-

|}

There was a concern that this would have overwhelmed the [[Overstay2 processes on the units to reduce discharge delay]] so we considered a [[#pragmatic threshold]].

=== Pragmatic threshold ===

We considered whether we should change the threshold from the [[#Optimal threshold]] to reduce the number of patients who are assigned as red, in addition to modifying the [[Overstay2 processes on the units to reduce discharge delay]] .

{| class="wikitable notsortable"

!Site !! Data Set!! Pragmatic Cut-off !! Sensitivity !! Specificity || PPV || NPV || FP Rate || FN Rate

|-

| GGH || Training|| 0.094|| 67.2 || 79.0 || 17.5 || 97.3 || 21.0|| 32.8

|-

| HSC || Training|| || || || || || ||

|-

| STB || Training|| || || || || || ||

|-

|}

{{collapsable | always = I think this can go, just leaving it until confirmed | full=

We used the following process to set a red/yellow threshold that our [[Overstay2 processes on the units to reduce discharge delay]] can function with.

==== Goals/considerations ====

The thresholds were set to stratify patients based on the following considerations:

* goal: reduce LOS 10-15%

* no more than 15-17% of patients tagged

** Use Positive Predictive Value (PPV) which denotes out of the patients tagged, how many are true positives.

* capture 60-75% of all delayed discharges

** Use Sensitivity or True Positive Rate which denotes out of delayed discharges, how many are correctly identify as delayed.

{| class="wikitable notsortable"

!Site !! Data Set!!Treshold Cut-off !! Sensitivity !! Specificity || PPV || NPV

|-

| GGH || Training|| 0.114|| '''60.3''' || 83.5 || 19.5 || 96.9

|-

| GGH || Training|| 0.076|| '''75.3''' || 74.0 || 16.2||97.8

|-

| GGH || Training|| 0.061|| 82.2 || 69.1 || '''15.0''' ||98.3

|-

| GGH || Training|| 0.090||69.4 || 77.4 || '''17.0''' || 97.2

|-

| GGH || Validation|| 0.110|| '''60.5''' || 82.3 || 18.9 || 96.7

|-

| GGH || Validation|| 0.078|| '''75.2''' || 74.3 || 16.7 || 97.8

|-

| GGH || Validation|| 0.060|| 82.3 || 68.5 || '''15.2''' || 98.3

|-

| GGH || Validation|| 0.084|| 73.0 || 75.6 || '''17.0''' || 97.6

|-

| HSC || Training|| || || || ||

|-

| HSC || Validation|| || || || ||

|-

| STB || Training|| || || || ||

|-

| STB || Validation|| || || || ||

|-

|}

{{DJ | How did those get set, though? If they are arbitrary then any rigor in balancing them is just dazzle. [[User:Ttenbergen|Ttenbergen]] 19:48, 8 June 2025 (CDT) }}

== ~~Method of setting~~ threshold ==

==== Setting threshold based on these goals ====

{{DJ | Emailed 2025-06-08: we need to understand the mechanism we used to set the threshold. To be updated when clear. [[User:Ttenbergen|Ttenbergen]] 17:57, 8 June 2025 (CDT)

}}

== Log ==

* 2025-06-13 - continuous model no better, confirmed using threshold of 0.077 for now

* 2025-06-11 - meeting with DR and JM to discuss, considered continuous rather than binary model, Julie will generate

* 2025-06-08 - email to determine how to set colour in that new context

* 2025-05-27 - Julie finalized an updated [[Overstay2 scoring model for Grace Hospital]]

* 2025-02-?? - ~~Julie made better model~~ for ~~Grace hospital only~~

* 2025-02-24 - we realized we needed separate [[Overstay2 scoring models]] for each site

* 2025-02-21 - [https://q.tenbergen.ca/index.php?title=2025-02-21_JM_DR_Overstay_model_decision_and_cut-off 2025-02-21 JM DR Overstay model decision and cut-off] - We will use Model 8 at cut-off of 0.069

* 2025-02-20 - [https://q.tenbergen.ca/index.php?title=2025-02-20_DR#eval_the_Overstay2_scoring_model decided] that we want

@@ Line 3: / Line 3: @@
 The colour is based on a probability threshold that is set to balance the number of patients followed under the more elaborate [[Overstay2 processes on the units to reduce discharge delay#red process]] wile minimizing the number of patients who would overstay by more than 10 days that are not captured. It was set separately for each of the [[Overstay2 scoring models]], so '''see those pages for the actual threshold'''s.
-== Goals/considerations ==
+== Method of setting threshold ==
-The thresholds were set to stratify patients into different [[Overstay2 processes on the units to reduce discharge delay]] based on the following considerations:
+=== Optimal threshold ===
+* A  2 x 2 classification  table for incrementing probability levels from 0.001 to 0.200 helps visualize the performance of the predicted results with actual outcomes.
+* Sensitivity (true positive rate)  is the proportion of actual positives correctly identified by the predicted results. High sensitivity means few false negatives.
+* Specificity (true negative rate) is the proportion of actual negatives correctly identified by the predicted results.  High specificity means few false positives.
+* Positive Predicted Value (PPV) is the likelihood that a positive result is a true positive.
+* Negative Predicted Value (NPV) is the likelihood that a negative result is a true negative.
+* False Positive (FP) Rate is the proportion of actual  negatives but predicted incorrectly as positives.
+* False Negative (FN) Rate is the proportion of actual  positives but predicted incorrectly as negatives.
+* Optimal threshold is the probability level which shows equal or close to equal  sensitivity or specificity in the testing data set.  This threshold (also called cut-off) is being used to classify patients.
+* The optimal threshold by site is below:
+{| class="wikitable notsortable"
+ !Site !! Data Set!! Optimal Cut-off !! Sensitivity !! Specificity || PPV || NPV || FP Rate || FN Rate
+|-
+| GGH  || Training|| 0.076|| 74.2 || 74.3 || 16.1 || 97.7 || 25.7 || 25.8
+|-
+| GGH  || Validation|| 0.079|| 74.4 || 74.5 || 16.4 || 97.7 || 25.5 || 25.6
+|-
+| HSC  || Training|| ||   ||  || || || ||
+|-
+| HSC  || Validation||  ||  ||  || || || ||
+|-
+| STB  || Training|| ||   ||  || || || ||
+|-
+| STB  || Validation||  ||  ||  || || ||
+|-
+|}
+There was a concern that this would have overwhelmed the [[Overstay2 processes on the units to reduce discharge delay]] so we considered a [[#pragmatic threshold]].
+=== Pragmatic threshold ===
+We considered whether we should change the threshold from the [[#Optimal threshold]] to reduce the number of patients who are assigned as red,  in addition to modifying the [[Overstay2 processes on the units to reduce discharge delay]] .
+{| class="wikitable notsortable"
+!Site !! Data Set!! Pragmatic Cut-off !! Sensitivity !! Specificity || PPV || NPV || FP Rate || FN Rate
+|-
+| GGH  || Training|| 0.094|| 67.2 || 79.0 || 17.5 || 97.3 || 21.0|| 32.8
+|-
+| HSC  || Training|| ||   ||  || || || ||
+|-
+| STB  || Training|| ||   ||  || || || ||
+|-
+|}
+{{collapsable | always = I think this can go, just leaving it until confirmed | full=
+We used the following process to set a red/yellow threshold that our [[Overstay2 processes on the units to reduce discharge delay]] can function with.
+==== Goals/considerations ====
+The thresholds were set to stratify patients based on the following considerations:
 * goal: reduce LOS 10-15%
 * no more than 15-17% of patients tagged
+** Use Positive Predictive Value (PPV) which denotes out of the patients tagged, how many are true positives.
 * capture 60-75% of all delayed discharges
+** Use Sensitivity or True Positive Rate which denotes out of delayed discharges, how many are correctly identify as delayed.
+{| class="wikitable notsortable"
+ !Site !! Data Set!!Treshold Cut-off !! Sensitivity !! Specificity || PPV || NPV
+|-
+| GGH  || Training|| 0.114|| '''60.3''' || 83.5 || 19.5 || 96.9
+|-
+| GGH  || Training|| 0.076|| '''75.3''' || 74.0 || 16.2||97.8
+|-
+| GGH  || Training|| 0.061|| 82.2 || 69.1 || '''15.0''' ||98.3
+|-
+| GGH  || Training|| 0.090||69.4 || 77.4 || '''17.0''' || 97.2
+|-
+| GGH  || Validation|| 0.110|| '''60.5''' || 82.3 || 18.9 || 96.7
+|-
+| GGH  || Validation|| 0.078|| '''75.2''' || 74.3 || 16.7 || 97.8
+|-
+| GGH  || Validation|| 0.060|| 82.3 || 68.5 || '''15.2''' || 98.3
+|-
+| GGH  || Validation|| 0.084|| 73.0 || 75.6 || '''17.0''' || 97.6
+|-
+| HSC  || Training||  ||   ||  || ||
+|-
+| HSC  || Validation||  ||  ||  || ||
+|-
+| STB  || Training|| ||   ||  || ||
+|-
+| STB  || Validation||  ||  ||  || ||
+|-
+|}
+{{DJ | How did those get set, though? If they are arbitrary then any rigor in balancing them is just dazzle. [[User:Ttenbergen|Ttenbergen]] 19:48, 8 June 2025 (CDT) }}
-== Method of setting threshold ==
+==== Setting threshold based on these goals ====
 {{DJ | Emailed 2025-06-08: we need to understand the mechanism we used to set the threshold. To be updated when clear. [[User:Ttenbergen|Ttenbergen]] 17:57, 8 June 2025 (CDT)
+}}
 }}
 == Log ==
+* 2025-06-13 - continuous model no better, confirmed using threshold of 0.077 for now
+* 2025-06-11 - meeting with DR and JM to discuss, considered continuous rather than binary model, Julie will generate
+* 2025-06-08 - email to determine how to set colour in that new context
 * 2025-05-27 - Julie finalized an updated [[Overstay2 scoring model for Grace Hospital]]
-* 2025-02-?? - Julie made better model for Grace hospital only
+* 2025-02-24 - we realized we needed separate [[Overstay2 scoring models]] for each site
 * 2025-02-21 - [https://q.tenbergen.ca/index.php?title=2025-02-21_JM_DR_Overstay_model_decision_and_cut-off 2025-02-21 JM DR Overstay model decision and cut-off] - We will use Model 8 at cut-off of 0.069
 * 2025-02-20 - [https://q.tenbergen.ca/index.php?title=2025-02-20_DR#eval_the_Overstay2_scoring_model decided] that we want