Determining Roe and Metz model parameters for simulating multireader multicase confidence-of-disease rating data based on real-data or conjectured Obuchowski–Rockette parameter estimates

Abstract. Purpose The most frequently used model for simulating multireader multicase (MRMC) data that emulates confidence-of-disease ratings from diagnostic imaging studies has been the Roe and Metz (RM) model, proposed by Roe and Metz in 1997 and later generalized by Hillis (2012), Abbey et al. (2013), and Gallas and Hillis (2014). A problem with these models is that it has been difficult to set model parameters such that the simulated data are similar to MRMC data encountered in practice. To remedy this situation, Hillis (2018) mapped parameters from the RM model to Obuchowski–Rockette (OR) model parameters that describe the distribution of the empirical AUC outcomes computed from the RM model simulated data. We continue that work by providing the reverse mapping, i.e., by deriving an algorithm that expresses RM parameters as functions of the OR empirical AUC distribution parameters. Approach We solve for the corresponding RM parameters in terms of the OR parameters using numerical methods. Results An algorithm is developed that results in, at most, one solution of RM parameter values that correspond to inputted OR parameter values. The algorithm can be implemented using an R software function. Examples are provided that illustrate the use of the algorithm. A simulation study validates the algorithm. Conclusions The resulting algorithm makes it possible to easily determine RM model parameter values such that simulated data emulate a specific real-data study. Thus, MRMC analysis methods can be empirically tested using simulated data similar to that encountered in practice.


Introduction
For the typical diagnostic radiology study, several readers (typically radiologists) assign confidence-of-disease ratings to each case (i.e., subject) based on one or more corresponding radiologic images. The resulting data are called multireader multicase (MRMC) data. These studies are typically used to compare different imaging modalities with respect to reader performance. Often measures of reader performance are functions of the estimated receiver-operatingcharacteristic (ROC) curve, such as the area under the ROC curve (AUC). The Obuchowski *Address all correspondence to Stephen L. Hillis, steve-hillis@uiowa.edu and Rockette method (OR) 1 is a commonly used method of analyzing reader performance outcomes which results in conclusions that generalize to both the reader and case populations.
The most frequently used model for simulating MRMC data that emulate confidence-ofdisease ratings from such studies has been the model first proposed by Roe and Metz 2 and later generalized by Hillis, 3 Abbey, 4 and Gallas and Hillis. 5 We will refer to each of these models as the "Roe and Metz" (RM) model when there is no need to distinguish between them. Numerous studies have used this model for evaluating MRMC analysis and sample size methods. As discussed by Hillis, 6 the RM model generates continuous confidence-of-disease ratings based on an underlying binormal model for each reader-test combination, with the separation between the normal and abnormal rating distributions varying across readers.
Because RM model parameters are expressed in terms of the latent rating data distribution, in contrast to MRMC analysis results that are almost always expressed in terms of parameters that describe the distribution of the reader performance outcomes, it has been difficult to set RM model parameter values such that the simulated data exhibit characteristics that are similar to MRMC data encountered in practice. To remedy this situation, Gallas and Hillis 5 mapped the RM model parameters to variance and covariance parameters that describe the distribution of the empirical AUC outcomes computed from RM simulated data. Similarly, Hillis 6 mapped the RM model parameters to OR parameters that describe the distribution of empirical AUC outcomes computed from RM simulated data. This paper continues that work by developing a numerical algorithm that expresses the RM parameters as functions of the empirical AUC distribution OR parameters. This result makes it easy to determine RM model parameter values such that the simulated data emulate a specific real-data study. The primary uses for the proposed algorithm are testing MRMC analysis methods and computing power estimates, using simulated MRMC data that match real data sets with respect to the empirical AUC distribution OR parameter estimates.
An outline of this paper is as follows. In Sec. 2, we discuss the original Roe and Metz model, the Hillis 3 generalization of it, and the OR model and analysis method. In Sec. 3, we discuss the numerical OR-to-RM algorithm that maps OR parameters to RM parameters, which is derived in Appendix A for the Hillis 3 generalization of the original RM model. In Sec. 4, we illustrate using the OR-to-RM algorithm and the previously derived RM-to-OR algorithm to simulate data emulating a real-data study, along with other examples and remarks concerning the use of the two algorithms. The paper concludes in Secs. 5 and 6.

Original RM model
Let X denote a confidence-of-disease rating assigned by a reader to a case; X is often called a decision variable (DV). The original RM simulation model proposed by Roe and Metz 2 is a mixed four-factor (test, reader, case, and truth) ANOVA model for X with case nested within truth; test, reader, and truth crossed; test and truth treated as fixed factors; and reader and case treated as random factors. Note that we use "test" as a general term that can refer to a diagnostic test, imaging modality, or a treatment. Throughout this paper, we only consider the situation of comparing two tests.
Using the RM notation, the model is given as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 1 1 6 ; 1 6 5 X ijkt ¼ μ t þ τ it þ R jt þ C kt þ ðτRÞ ijt þ ðτCÞ ikt þ ðRCÞ jkt þ ðτRCÞ ijkt þ E ijkt ; (1) where X ijkt denotes the confidence-of-disease rating for test i, reader j, case k of truth state t, and t ¼ −; þ, with "−" indicating a nondiseased case and "+" indicating a diseased case. Here, μ t is the effect of truth state t, τ it is the interaction effect of test i and truth state t, R jt is the interaction effect of reader j and truth state t, C kt is the effect of case k nested within truth state t, the multiple symbols in parentheses denote interactions, and E ijkt is the error term. Thus, X ijkt denotes the confidence-of-disease rating assigned to case k of truth state t by reader j when reading under test i. All effects are random except for μ t and τ it . The random effects are mutually independent and normally distributed with zero means. Roe and Metz denote the corresponding variance components by σ 2 R , σ 2 C , σ 2 τR , σ 2 τC , σ 2 RC , σ 2 τRC , and σ 2 E . They note that σ 2 τRC and σ 2 E cannot be estimated separately for this model with no replications, as re-reading images in radiological studies is uncommon due to the cost, and hence define E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 2 . 1 . 1 ; 1 1 6 ; 6 6 3 σ 2 ε ≡ σ 2 τRC þ σ 2 E : Although not mentioned by Roe and Metz, the omission of test, reader, and test-by-reader effects that do not depend on truth is justified by the invariance of the ROC curve to location shifts; thus inclusion of these terms would not change the ROC curve for a given reader. Note that interactions with truth are denoted only by a t subscript in Eq. (1). Roe and Metz constrain the sum of the error variance and variance components involving case to be equal to one: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 1 1 6 ; 5 5 9 It follows from this constraint 6 that the fixed-reader nondiseased and diseased DV distributions have unit variances (and hence their ROC curves are symmetric about the negative 45 deg diagonal), with the fixed-reader AUCs varying across the reader population. Without loss of generality, Roe and Metz impose the constraints E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 1 1 6 ; 4 7 9 which result in the same DV distributions for both tests 1 and 2. Under this constraint, it can be shown 6 that the mean and median separation of the nondiseased and diseased DV distributions across the reader population is given by μ þ and the median reader-specific AUC is given by where Φ is the cumulative distribution function of the standard normal distribution.

Unequal test DV distributions
Although Roe and Metz only consider simulations for equal test DV distributions for each reader, the model can be easily modified to allow for test DV distributions that differ in their median AUC values by not setting τ 2þ to zero, that is, only the constraints E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 4 ; 1 1 6 ; 3 1 3 are imposed. It follows that the median AUCs for tests 1 and 2 are equal to A ; t e m p : i n t r a l i n k -; e 0 0 5 ; 1 1 6 ; 2 5 5 are the mean and median separations of the nondiseased and diseased DV distributions for tests 1 and 2, respectively, across the reader population. From constraints Eq. (4), it follows that δ 1 ¼ μ þ for test 1 and δ 2 ¼ μ þ þ τ 2þ for test 2. To insure that A ðiÞ z ≥ :5, we assume E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 1 1 6 ; 1 8 5 Note that the RM model that allows for test-dependent AUCs is completely defined by seven parameters: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 7 ; 1 1 6 ; 1 2 9 δ 1 ; δ 2 ; σ 2 R ; σ 2 τR ; σ 2 C ; σ 2 τC ; and σ 2 RC : Note that σ 2 ε can be computed using Eqs. (2) and (7).

Constrained unequal-variance RM model (RMH model)
In practice, estimated binormal-model nondiseased and diseased distribution variances for a  reader-test combination are often different, with diseased subjects typically having more variable  test results. Thus to better emulate real data, Hillis 3 modified the original RM model by allowing  variance components involving cases to depend on truth, with variance components involving  diseased cases set equal to those involving nondiseased cases multiplied by the factor 1∕b 2 , b > 0. Specifically, the model is given by Eq. (1) with variance components (using an obvious notation) denoted by σ 2 CðþÞ , σ 2 τCðþÞ , σ 2 RCðþÞ , and σ 2 εðþÞ , with σ 2 (2), the constraint E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 8 ; 1 1 6 ; 6 0 1 is imposed. It follows that E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 2 . 1 . 3 ; 1 1 6 ; 5 5 5 Constraint Eq. (6) is also imposed. We will refer to this model as the constrained unequalvariance RM model or simply as the RMH model, with the "H" in RMH indicating that it is the generalization of the original RM model proposed by Hillis. 3 Similar to the original RM model, 2 imposing constraint Eq. (3) results in the null model with A Þ, and imposing constraint Eq. (4) results in the nonnull model with E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 2 . 1 . 3 ; 1 1 6 ; 4 4 6 where again A ðiÞ z denotes the median AUC across the reader population for test i, δ i is defined by Eq. (5), and δ i is the mean and median DV separation for test i across readers.
The algorithm discussed in this paper will be for the RMH model, which includes the original RM model 2 as a special case when b is set equal to 1. Note that the RMH model that allows for test-dependent AUCs is completely defined by the eight linearly independent parameters b; δ 1 ; δ 2 ; σ 2 R ; σ 2 τR ; σ 2 Cð−Þ ; σ 2 τCð−Þ , and σ 2 RCð−Þ . We let β RMH denote the vector of these parameters: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 9 ; 1 1 6 ; 3 0 6

Obuchowski-Rockette Model
Obuchowski and Rockette 1 proposed a test × reader factorial ANOVA model for the AUC estimates, but unlike a conventional ANOVA model, the errors are assumed to be correlated to account for correlation due to each reader evaluating the same cases. Their model, which we refer to as the OR model, is given as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 0 ; 1 1 6 ; 1 9 5θ where μ OR is the intercept term, τ i∶OR denotes the fixed effect of test i, R j∶OR denotes the random effect of reader j, ðτRÞ ij∶OR denotes the random test × reader interaction, and ε ij∶OR is the error term. The R j∶OR and ðτRÞ ij∶OR are assumed to be mutually independent and normally distributed with zero means and respective variances σ 2 R∶OR and σ 2 TR∶OR . (OR in the subscripts is to distinguish OR effects and variance components from similarly notated RMH-model quantities.) The ε ij∶OR are assumed to be normally distributed with mean zero and variance σ 2 ε∶OR and are assumed uncorrelated with the R j∶OR and ðτRÞ ij∶OR . Three possible error covariances are assumed: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 1 ; 1 1 6 ; 7 3 5 Covðε ij∶OR ; ε i 0 j 0 ∶OR Þ ¼ 8 > < > : The OR model assumes 7 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 2 ; 1 1 6 ; 6 6 8 Cov 1 ≥ Cov 3 ; Cov 2 ≥ Cov 3 ; Cov 3 ≥ 0: These error variance-covariance parameters are typically estimated by averaging corresponding conditional-on-readers estimates computed using the jackknife, 8-10 bootstrap, 10,11 the method proposed by DeLong et al. 12 (for empirical AUC estimates), or the method proposed by Metz et al. 13 based on the semiparametric binormal ROC model. These four estimation methods are consistent but are not unbiased. An unbiased error covariance estimation method (unbiased method) was recently proposed by Hillis 6,14 for use when empirical AUC is the outcome. This method utilizes the unbiased fixed-reader method discussed by Gallas [Ref. 15,p 362] for estimating the error variance, and extensions of it for estimating the error covariances. This method results in unbiased OR parameter estimates when data are generated from the RMH model. 6 OR analysis using this method is included in the freely available R software package MRMCaov. 16 The ε ij∶OR can be interpreted as AUC measurement error attributable to the random selection of cases and within-reader variability that describes how a fixed reader interprets the same image in different ways on different occasions. The OR model can alternatively be described with population correlations r i ¼ Cov i ∕σ 2 ε∶OR replacing corresponding Cov i . Defining E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 2 . 2 ; 1 1 6 ; 4 5 5 the OR model for two tests, similar to the RMH model, is defined by eight linearly independent parameters: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 3 ; 1 1 6 ; 3 8 1 μ 1∶OR ; μ 2∶OR ; σ 2 R∶OR ; σ 2 TR∶OR ; σ 2 ε∶OR ; Cov 1 ; Cov 2 ; and Cov 3 ; or equivalently, by E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 4 ; 1 1 6 ; 3 3 7 μ 1∶OR ; μ 2∶OR ; σ 2 R∶OR ; σ 2 TR∶OR ; σ 2 ε∶OR ; r 1 ; r 2 and r 3 : We let β OR denote the vector of these parameters: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 5 ; 1 1 6 ; 2 9 3 β OR ¼ ðμ 1∶OR ; μ 2∶OR ; σ 2 R∶OR ; σ 2 TR∶OR ; σ 2 ε∶OR ; r 1 ; r 2 ; r 3 Þ: Note that when the outcome is the empirical AUC that μ 1∶OR and μ 2∶OR are the test 1 and test 2 expected values for the empirical AUC estimates across readers and cases.

OR-to-RMH Algorithm for Estimating RMH Parameter Values When the Goal Is to Emulate a Real-Data MRMC Study
The RMH-to-OR mapping, previously derived by Hillis, 6 and the new OR-to-RMH algorithm that maps OR parameters to RMH parameters and its development are provided in Tables 6 and  10, respectively, in Appendix A.
In this section, we discuss the main points of the OR-to-RMH algorithm when the goal is to emulate data from a real study with the RMH model; i.e., to determine RMH parameter values such that the expected values of the OR parameter estimates from the simulated MRMC samples are described by the β OR vector Eq. (15), estimated from a real study.
The β OR vector Eq. (15) implicitly provides information about the shape of the underlying ROC curve through the value of σ 2 ε∶OR , which is a function of the RMH b parameter in the RMHto-OR mapping. The method used for estimating the RMH b parameter for the OR-to-RMH algorithm is called the b_method. To estimate a β RMH vector Eq. (9) that maps to a particular β OR vector Eq. (15), the algorithm requires use of the option b_method = unspecified, which we assume throughout this section. Two other options for b_method and the situations where they are useful will be discussed in Sec. 3.2.
3.1.1 Overview of OR-to-RMH algorithm Table 6 in Appendix A gives the previously derived analytical RMH-to-OR mapping formulas. 6 Mathematically, we describe this transformation by the function f that maps the RMH parameter vector and the case samples sizes that will be used for the simulations to the resulting OR parameter vector: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 6 ; 1 1 6 ; 5 6 6 fðβ RMH ; n 0 ; n 1 Þ ¼ β OR : This function is analytical and thus does not require a numerical algorithm.
The OR-to-RMH algorithm requires inputted values for β OR ; n 0 ; n 1 , and b_method, where β OR is given by Eq. (15) and n 0 and n 1 are the corresponding real-study nondiseased and diseased case sizes. To derive the OR-to-RMH algorithm, we first assume that there exists an RMH parameter vector β RMH corresponding to β OR such that Eq. (16) is true. We then express the OR parameters in terms of the RMH parameters and solve for the RMH parameters using numerical methods (see Appendix A for details.) It is possible that there are several β RMH vectors satisfying Eq. (16), in which case the corresponding β RMH vectors will differ only in their b values, as discussed in Appendix A. It is also possible that there is no β RMH vector that satisfies Eq. (16). To force the OR-to-RMH algorithm to produce, at most, only one output, the β RMH vector with b closest to 1 with 0.01 ≤ b ≤ 1 is chosen; if no corresponding β RMH vector has 0.01 ≤ b ≤ 1, then the corresponding β RMH vector with b closest to 1 with 1 < b ≤ 4 is chosen. If there are no corresponding β RMH solution vectors with 0.01 ≤ b ≤ 4, the algorithm does not return a solution for β RMH ; see Sec. 3.1.3 for what to do when this happens.
Let g 1 denote the function defined by the OR-to-RMH algorithm, with b_method = unspecified, that maps β OR to a solution for β RMH , denoted by β RMH;solution ; i.e., E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 7 ; 1 1 6 ; 3 3 0 g 1 ðβ OR ; n 0 ; n 1 Þ ¼ β RMH;solution : Ideally, β RMH;solution will be such that the RMH-to-OR mapping will return the original OR parameter, i.e., However, it is possible for the OR-to-RMH algorithm to return a solution such that Eq. (18) holds only approximately, i.e., E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 9 ; 1 1 6 ; 2 1 8 fðβ RMH;solution ; n 0 ; n 1 Þ ≈ β OR : The approximation results because of constraints on the RMH parameters that are imposed by the algorithm, as discussed in Appendix A and given in Eq. (23) in Table 7. For example, if the inputted value of r 3 exceeds that of r 2 then the solution β RMH;solution will be such that r 2 ≥ r 3 in fðβ RMH;solution ; n 0 ; n 1 Þ.
Rationale for the b limits. The lower and upper limits for b of 0.01 and 4 are chosen because b values outside these limits are not realistic for most real data sets. In most situations, a meaningful DV should be an increasing transformation of the likelihood ratio (likelihood of being diseased divided by likelihood of not being diseased). 17   Simulation of data to emulate a real-data study. Figure 2 summarizes how the OR-to-RMH and RMH-to-OR algorithms can be used to simulate data that emulate a real-data study.
The OR-to-RMH algorithm (with b_method = unspecified) is applied to OR estimates (β OR;input ) obtained from a real-data study, resulting in the corresponding RMH model. This model is then used for generating MRMC samples for any specified number of readers and cases, with n Ã 0 and n Ã 1 denoting the case numbers for the simulations and n 0 and n 1 denoting the case  numbers for the original real-data study. The distribution of the empirical AUCs for the simulated data is described by β OR;output . We recommend always checking how closely the simulated data emulate the study data by comparing β OR;input and β OR;output when the simulation model generates samples with the same case sizes as the original study, i.e., with n Ã 0 ¼ n 0 and n Ã 1 ¼ n 1 .

Should the simulated ROC curves resemble the original study ROC curves?
We emphasize that even when simulating data using an RMH model such that β OR;output ¼ β OR∶input in Fig. 3, we do not claim that the resulting empirical ROC curves will be visually similar to those estimated from a real-data study. Rather, we only claim that the expected values of the OR parameter estimates for the simulated data will be the same as those computed from the original real-data study, given by Eq. (13). (Note that Eq. (13) contains the error covariances rather than the error correlations.) However, because of the robustness of the binormal model assumption for fitting ROC curves to real data, [20][21][22] we typically expect there will be some resemblance, although the degree of resemblance will be limited by the RMH model having only eight parameters. In particular, we note that the RMH model requires each reader's ROC curve to have the same b value, which will determine the shape of the ROC curve for a given reader AUC value; this result follows from the one-to-one correspondence between ðb; AUCÞ and ða; bÞ, with a ¼ bΦðAUCÞ , as mentioned in Sec. 3.1.1.

Reasons for neither an exact nor approximate solution
OR-to-RMH algorithm does not work because there is not a solution for b. For given values of the RMH parameters δ 1 ; δ 2 ; σ 2 R , and σ 2 TR (computed in steps 1 to 3 of the OR-to-RMH algorithm in Table 10), the value of b (computed in step 4) determines the value of σ 2 ε;OR : It can happen that the algorithm does not produce a solution for b, either because no solution exists, or the solution is <0.01 or >4.0 that will yield the input value for σ 2 ε∶OR for the values of δ 1 ; δ 2 ; σ 2 R , and σ 2 TR that have been computed by the algorithm in previous steps. When this occurs, one can choose to use one of the other two methods for estimating b, as discussed in Sec. 3.2. OR-to-RMH algorithm does not work because there is not a solution for an RMH parameter other than b. When required, the algorithm imposes the constraints in Table 7(b) by altering somewhat the inputted OR parameter values, which can lead to an approximate solution as given by Eq. (19). However, when other constraints, which are implied by the RMH-to-OR mapping in Table 6, do not hold, the result is a missing value for the particular RMH parameter and for all other RMH parameters requiring it for their computation. For example, from the equations in Tables 8 and 9, it can be shown that there is an upper limit for σ 2 R∶OR , which is a function of the values of the inputted values for μ 1∶OR and μ 2∶OR . Similarly, it can be shown that there are upper limits for σ 2 TR∶OR ; r 1 ; r 2 , and r 3 , which are functions of parameters computed in previous steps. When one of these values exceeds its upper limit, the algorithm does not yield a solution.
This problem is more likely to happen when inputted values for β OR are conjectured than when they are estimates from a real-data study. If this problem occurs, we first recommend that the inputted values be checked for entry errors. If there are none, then we suggest inputting a different (typically smaller) value for the OR parameter corresponding to the RMH parameter, which cannot be estimated. See Appendix A and Table 5  As discussed by Hillis, 6,23 the OR parameters μ 1∶OR ; μ 2∶OR ; σ 2 R∶OR , and σ 2 TR∶OR have meaningful interpretations that do not depend on sample size, and r 1 , r 2 , and r 3 have meaningful interpretations that remain approximately (but not exactly) constant as the sample sizes change. On the other hand, σ 2 ε∶OR varies with the sample sizes. In this section, we discuss two approaches for determining RMH parameters that result in simulated MRMC data for which the empirical AUC distribution matches conjectured values of the parameters in E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 3 . 2 ; 1 1 6 ; 4 1 8β OR ¼ ðμ 1∶OR ; μ 2∶OR ; σ 2 R∶OR ; σ 2 TR∶OR ; r 1 ; r 2 ; r 3 Þ: Note thatβ OR is the same as β OR but without σ 2 ε;OR . The value of σ 2 ε∶OR for the simulated data will be determined by the sample sizes and the RMH parameters.
These approaches are useful when one is primarily interested in simulating data that match an OR correlation and variance component structure and a real-data value of σ 2 ε∶OR is not available. They also are useful when real-data estimates for β OR are available but there is no solution for b using the OR-to-RMH algorithm with b_method = unspecified.

Overview
The two approaches are similar to that described in Sec. 3.1, except that estimation of b does not depend on an inputted value for σ 2 ε∶OR . Instead, b is either (1) explicitly specified using b_method = specified and setting the value of the input variable b_input equal to the desired value for b; or (2) computed so as to result in a median specified mean-to-sigma ratio across readers, using b_method = mean_to_sigma and setting the value of the input variable mean_ sig_input equal to the desired mean-to-sigma ratio.
Use of the OR-to-RMH and RMH-to-OR algorithms to simulate data using these two approaches is summarized in Fig. 3. Figure 3 is similar to Fig. 2 with these differences: (1) No input value for σ 2 ε∶OR is included because the input values are forβ OR instead of for β OR . (2) For the OR-to-RMH algorithm, the g 2 or g 3 function (as defined below) is used in the place of the g 1 function. Note that the outputted OR parameter values include a value for σ 2 ε∶OR .
Approach 1: b_method = specified. With this approach, the value of b is specified.
For example, the parameter values for the original 2 RM model can be determined by setting b_input ¼ 1.
Let g 2 denote the function defined by the OR-to-RMH algorithm, with b_method = specified, that mapsβ OR and an inputted value of b to a solution for β RMH , denoted by β RMH;solution ; i.e., Again, ideally β RMH;solution will be such that fðβ RMH;solution ; n 0 ; n 1 Þ ¼β OR . However, similar to using b_method = unspecified, it is possible for the OR-to-RMH algorithm to return a solution such that fðβ RMH;solution ; n 0 ; n 1 Þ ≈β OR because of constraints on the RMH parameters Eq. (23) in Table 7 that are imposed by the algorithm.
Approach 2: b_method = mean_to_sigma. Recall from Sec. 3.1.3 that when b_method = unspecified is used, the value of b (based on the computed values of the RMH parameters δ 1 ; δ 2 ; σ 2 R and σ 2 TR ) is determined such that σ 2 ε∶OR for the simulated data will match the inputted value for σ 2 ε∶OR . In contrast, when b_method = mean_to_sigma is used, the user specifies a desired median mean-to-sigma value (see discussion of the mean-to-sigma measure below) across readers for the test corresponding to the minimum of the inputted μ 1∶OR and μ 2∶OR values.
Let g 3 denote the function defined by the OR-to-RMH algorithm with b_method = mean_ to_sigma that mapsβ OR and an inputted value of the mean-to-sigma ratio, denoted by q, to a solution for β RMH : E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 2 1 ; 1 1 6 ; 1 4 3 g 3 ðβ OR ; n 0 ; n 1 ; qÞ ¼ β RMH;solution : As was the case for the other two b estimation methods, ideally, fðβ RHM;solution ; n 0 ; n 1 Þ ¼β OR , but it is possible for this relationship to hold only approximately because of constraints on the RMH parameters. Flowchart illustrating the use of the OR-to-RMH and RMH-to-OR algorithms to simulate MRMC data that emulates OR AUCs, reader variance components, and OR correlations, but not σ 2 ε∶OR .

Mean-to-sigma ratio
The mean-to-sigma ratio, denoted by q, is defined as the difference of the latent diseased and nondiseased DV means divided by the difference of their standard deviations. The mean-tosigma ratio was first introduced by Swets, 24 who noticed that it seemed to be approximately constant for a variety of experiments. Some support for this conclusion was provided by later analyses. 22,25,26 For example, Green and Swets 26 note that q ≈ 4 is typical for many studies. As discussed by Hillis and Berbaum, 27 q can be used as a measure of improperness for a binormal ROC curve; specifically, it indicates that the ROC curve crosses the chance line at fpf ¼ ΦðqÞ, where fpf is the false positive fraction. They point out that it follows that an absolute value <2 indicates a noticeably improper binormal curve and an absolute value of infinity indicates a symmetric curve For the RMH model, the mean-to-sigma ratio varies across readers. To avoid simulating data based on visibly improper binormal curves, we suggest that the probability of a reader's true ROC curve being noticeably improper be small for each test, e.g., <0.025. This probability can be computed as a function of the RMH parameters, as discussed in Appendix B.1.

R language Functions
Two functions written in the R statistical software language that perform the OR-to-RMH and RMH-to-OR mappings are available within the freely available MRMCaov R package, 16 which can be downloaded from the Github repository: https://github.com/brian-j-smith/MRMCaov. The function OR_to_RMH transforms OR parameters to RMH parameters using the numerical algorithm described in Table 10, and the function RMH_to_OR performs the analytical RMH-to-OR transformation, described in Table 6.

Approach
In this section, we illustrate the use of the algorithms to simulate data that emulate data provided by Carolyn Van Dyke (VanDyke), 28 which we have used for examples in previous papers, 29,30 with empirical AUC being the reader performance metric. The study compared the relative performance of single spin-echo magnetic resonance imaging (SE MRI) to cinematic presentation of MRI (CINE MRI) for the detection of thoracic aortic dissection. There were n 0 ¼ 69 patients without a dissection and n 1 ¼ 45 patients with an aortic dissection imaged with both SE MRI and CINE MRI; cases were evaluated by five readers using a five-point ordinal confidence-ofdisease scale. Similarly, each RMH simulated sample emulated five readers, each evaluating the same 69 nondiseased and 45 diseased cases. We apply the OR-to-RMH algorithm to the set of parameter estimates ("original" values) obtained from an OR analysis of the data set to obtain corresponding RMH parameters values, simulate 10,000 MRMC samples based on the RMH values and analyze each simulated sample using an OR analysis, using the unbiased error covariance method, with the outcome being the empirical AUC. We set b_method = unspecified for the OR-to-RMH algorithm. Figure 4 shows the computation of the RMH simulation model and the "true values," which we define as the OR parameter values that describe the true distribution of the empirical AUCs computed from the simulated samples; i.e., the true values are the same as the outputted OR parameter values, given by β OR;output . We see that for this data set the outputted values are the same as the inputted values, and hence the original OR estimates exactly describe the true distribution of the simulated empirical AUC estimates. The R code and output for the OR-to-RMH and RMH-to-OR functions used to produce the results in Fig. 4 are provided in Appendix C.1. Table 1 presents the simulation study results. "Unbiased estimates" are the empirical estimates (the means across the simulated sample estimates) for the first eight parameters (μ 1;OR , μ 2;OR , σ 2 R∶OR ; : : : ; Cov 3 ), where OR estimates for each sample were computed using the OR method with the unbiased covariance estimation method discussed in Sec. 2.2. Because the sample estimates for the sample-level correlations r 1 ; r 2 , and r 3 are not unbiased, instead of reporting the empirical estimates we report the quotients resulting from dividing the corresponding empirical covariance estimates by the empirical error variance estimate. For example, the estimate of 0.434 for r 1 is computed by dividing the Cov 1 estimate (0.000343) by the σ 2 ε∶OR estimate (0.000791). Because the resulting estimates are not the means of the sample-level correlations, empirical bias estimates and 95% confidence intervals for the correlations are not included.

Simulation study results
"(Est -true)/true" is defined as (estimatetrue value)/(true value); it describes the deviation of the estimate from the true value and is expressed as a percentage of the true value. For the first eight parameters (i.e., not the correlations), these values can also be interpreted as the empirical estimates of statistical bias expressed as a percentage of the true value. "Within 95% CI?" is "yes" if the empirical 95% confidence interval (not shown) includes the true value, and otherwise is "no." We see that the unbiased estimates for the first eight parameters differ by <1.38% from the true values and that the correlation estimates differ by <0.53%. Moreover, all of the 95% empirical confidence intervals include the true value. Thus, the unbiased estimates agree with the true parameter values and hence provide validation for the OR-to-RMH algorithm.
Plots of the empirical ROC curve for the VanDyke original data and for the first three simulated MRMC samples, based on the RMH model given in Fig. 4, are displayed in Fig. 5. Like the VanDyke study, each simulated sample has five independent readers reading the same set of 69 nondiseased and 45 diseased cases. Although the plots look somewhat different because the VanDyke plots are based on at most five distinct ratings, whereas the simulated-data plots are based on a continuous rating scale, in general the simulated-data ROC curves show a definite resemblance to the VanDyke ROC curves, although this is only our subjective assessment.

DeLong error covariance estimation
For comparison, we also include in Table 1 results using the DeLong et al. 12 (DeLong) error covariance estimation method. Results for μ 1∶OR and μ 2∶OR are omitted since they depend only on the AUC estimation method and hence remain the same. We see from the confidence intervals that DeLong estimates for σ 2 ε , Cov 1 , and Cov 2 are positively biased and the σ 2 TR estimate is negatively biased. Similar results were obtained by Hillis. 6 Although the DeLong method is biased, the estimates are relatively close to the true values, suggesting that results using the DeLong or another resampling error-covariance method, such as the jackknife or bootstrap, will Fig. 4 Flowchart, analogous to Fig. 2, illustrating the use of the OR-to-RMH and RMH-to-OR algorithms to simulate MRMC data that emulate the VanDyke 28 data. Table 1 Simulation study estimates of OR parameters. "unbiased-method estimates" and " DeLong estimates" are the empirical estimates (i.e., means across the 10,000 samples) corresponding to using unbiased and DeLong error covariance estimation methods with the OR method. The correlation estimates are the quotients from dividing the corresponding covariance empirical covariance estimates by the empirical error variance estimate.
"Within 95% CI?" is "yes" if the 95% confidence interval includes the true value and is "no" otherwise. For the DeLong estimates, results for μ 1∶OR and μ 2∶OR are omitted since they are exactly the same as for the unbiased estimates.
typically be similar to those obtained using the unbiased method. This point is illustrated by the example in the next section.

Example of computing power
Suppose our goal is to estimate the power for detecting a difference in test AUCs for a study such as the VanDyke study, assuming that the reader-averaged empirical AUC estimates (0.897 and 0.941) are the true population values. This can be done by simulating similar data (as we did for Table 1) and then estimating power by the proportion of samples where the null hypothesis is rejected. The power estimates from doing this, based on the simulated samples used for Table 1, are 0.106 for the unbiased method and 0.107 using the DeLong method, illustrating how the choice of error covariance method makes almost no difference in our power estimates.  Fig. 4. TPF, true positive fraction (or sensitivity); FPF, false positive fraction (or 1specificity).

Ordinal rating scale
We conducted a simulation study to investigate how close the original OR parameter values might describe the distribution of the empirical AUC for ordinal ratings resulting from binning the continuous ratings generated by the RMH model given in Fig. 4. The simulation study was performed similar to Table 1 study, except that five-category ordinal ratings were created by binning simulated continuous ratings. The binning thresholds corresponded to the empirical cumulative probabilities for ratings 1,. . . ,5 for the VanDyke nondiseased cases, pooled across readers.
Results are presented in Table 2. As expected, the two AUC (μ 1∶OR ; μ 2∶OR ) estimates are less than for the continuous values, but only by a maximum of 1.44%. We also see that the correlations are similar to those for the continuous ratings (maximum deviation is −4.63%), with the relative values of the three even more similar: r 1 ≈ r 2 , as was the case for the continuous ratings, and r 3 is 0.12 lower than the other two, compared to being 0.13 lower for the continuous ratings. The maximum change in the error variance and covariance estimates was 8.07% and there were 6.7% and −7.9% changes in σ 2 R∶OR and σ 2 TR∶OR , respectively, which are in the same "ballpark" as for the continuous ratings.
We conclude that compared with the continuous data, the empirical AUC distribution for the binned data has a similar correlation structure, similar AUC estimates and somewhat similar values for the error variance, error covariances, σ 2 R∶OR and σ 2 TR∶OR . Thus, this example shows that the simulated ordinal data approximately emulate the VanDyke data set. Moreover, one could adjust the RMH parameters to result in a closer emulation using an iterative approach, where each iteration consists of adjustment of original OR values based on results from the previous-iteration simulation study, computation of corresponding RMH values, and a corresponding simulation study. For example, a first iteration might begin by upward adjustment of the μ 1∶OR and μ 2∶OR values.

Changing the numbers of readers and cases
In our examples, thus far we have set the numbers of readers, diseased cases, and nondiseased cases to be the same as those of the VanDyke data set. However, often a researcher will want to investigate the performance of a reader-performance metric for a range of these numbers.
Readers. For a given set of RMH parameter values, changing the number of readers has no effect on the corresponding OR parameters μ 1∶OR , μ 2∶OR , σ 2 R∶OR , σ 2 TR∶OR , σ 2 ε∶OR , Cov 1 , Cov 2 , Cov 3 , r 1 , r 2 , and r 3 , as shown by the omission of the reader number in the RMH-to-OR algorithm formulas in Table 6 in Appendix A.
Cases. For a given set of RMH parameter values, changing the number of cases has no effect on μ 1∶OR , μ 2∶OR , σ 2 R∶OR , or σ 2 TR∶OR , as shown by the omission of the case sample sizes in the corresponding formulas in Table 6. In contrast, σ 2 ε∶OR ; Cov 1 ; Cov 2 , and Cov 3 will be affected. Although the correlations are also affected, changes in the correlations will typically be small [Ref. 6, p 2078].
For example, Table 3(b) shows when the case sizes are doubled (n 0 ¼ 138, n 1 ¼ 90) that σ 2 ε∶OR is reduced by 50%, the correlations are virtually unchanged (maximum of 0.6%), and there is no change in σ 2 R∶OR , σ 2 TR∶OR , q 1 or q 2 . Table 3c shows when the case sample sizes are switched (n 0 ¼ 45, n 1 ¼ 69) that σ 2 ε∶OR is reduced by 19% and there is a small increase in the correlations (maximum increase of 2.3%), with all other values remaining unchanged. These results are computed using the RMH-to-OR formulas in Table 6, thus eliminating the need for simulations.

Null and power simulations
The example in Sec. 4.3.2 showed how power could be easily computed for simulated data that emulate a particular study, assuming the effect size (μ 1∶OR − μ 2∶OR ) is equal to the observed effect size. Other effect sizes can be investigated by adjusting δ 1 and δ 2 in the RMH parameter set accordingly, using the relationship (from Table 6):   (d) Null model 1

45.
In addition, the median mean-to-sigma ratios which implies E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 2 2 ; 1 1 6 ; 6 8 6 where Φ is the cumulative standard normal distribution function. In addition, often the researcher wants to empirically compute the type I error for testing H 0 ∶μ 1∶OR ¼ μ 2∶OR versus H 0 ∶μ 1∶OR ≠ μ 2∶OR . This can be done by creating a null RMH model by setting δ 1 ¼ δ 2 , with the empirical type I error rate given by the proportion of simulated samples where H 0 is rejected. For example, in Table 3(d) we alter the RMH model given in Fig. 4 by setting δ 1 ¼ δ 2 ¼ δ, with the value of δ determined such that the corresponding μ i∶OR values are both equal to μ OR ¼ 0.919, the mean of the two original OR AUC values, 0.897 and 0.941, in Fig. 4. It follows from Eq. (22), with μ OR ¼ ð0.897 þ 0.941Þ∕2, that δ ¼ 2.6452, using the values for b; σ 2 R , and σ 2 TR , given in Fig. 4. In Table 3(e), we similarly determine for a null RMH model the value of δ that correspond to μ OR ¼ 0.75. In both Table 3(d) and 3(e), we see that all of the original OR parameter values are changed, as well as the mean-to-sigma ratios, with Table 3(e) showing much more change. For this reason, we suggest that if the researcher wants to simulate data with error correlations and reader and reader-by-test variance components similar to those from an OR analysis of a realdata study, but with much different AUC values, the OR-to-RMH algorithm with b_option = mean_to_sigma should be used to determine the corresponding β RM vector, as discussed in the next section.
The R code and output for the OR-to-RMH and RMH-to-OR functions used to produce the results in Table 3 are included in Appendix C.2.

Mean-to-sigma ratios and the specified and mean_to_sigma b_options
From Table 3, parts (a)-(c), we see that the mean-to-sigma ratios are q 1 ¼ 4.56 and q 2 ¼ 5.64 for the Fig. 4 RMH model latent distributions, as well as for the models when the case sizes are changed. However, in parts (d) and (e), we see that when the values for the RMH parameters δ 1 and δ 2 are changed, the mean-to-sigma ratios also change.
In Table 3, Pr1 is the probability that a reader's true ROC curve is noticeably improper for test 1. (See Appendix B.1 for how to compute Pr1.) We see that this probability is relatively small (≤0.004) for the first four models and thus is not of concern. In contrast, Pr1 = 0.326 for null model 2, and thus we recommend not using this model for a simulation study. (Note: although Pr2, the analogous probability for test 2, is not included in Table 3, conclusions based on it were the same.) In Table 4, we see for the specified and mean_to_sigma b_methods that the OR parameters corresponding to the resulting RMH models are equal to all of the original OR values except for the error variance and covariances (not shown).
The R code for generating Table 4 is included in Appendix C.3. Table 5 provides examples where the OR-to-RMH algorithm fails to produce a solution. In each example, the OR-to-RMH algorithm is applied to the original parameter estimate values from the VanDyke study, given in Fig. 4, but with one value altered to result in the algorithm not working. For example, in part (a) σ 2 R∶OR is changed from 0.00154 (original value) to 0.154 and the algorithm fails. Using Table 11 in Appendix A, we can identify which input value is causing the problem by checking for the first parameter in the sequence x 1 ; x 2 ; x 3 ; x 4 ; b; x 5 ; x 6 ; x 7 that is missing (NA), where x 1 ; : : : ; x 7 are the alternative RM parameters discussed in Appendix A. Noting that the first parameter with a missing value is x 3 , the rules in Table 11 suggest reducing the value of σ 2 R∶OR : Similarly, in part (b), σ 2 TR∶OR is increased and x 4 is the first parameter with  Used with b_input = 1.

Troubleshooting
Hillis, Smith, and Chen: Determining Roe and Metz model parameters for simulating multireader multicase. . . a missing value; here, Table 11 suggests reducing the value of σ 2 TR∶OR : In part (c), σ 2 ε∶OR is increased and b is the first parameter with a missing value; here, Table 11 suggests either changing (reducing or increasing) the value of σ 2 ε;OR or using b_method = specified or b_method = mean_to_sigma.
The R code for generating the results in Table 5 is provided in Appendix C.4. The values for the x 1 ; : : : ; x 7 parameters are by default not printed unless the option all = T is included in the print function, as illustrated in Appendix C.4. Also note in Appendix C.4 that the OR_to_RM function suggests the remedy, based on Table 11, when the algorithm fails to produce a solution.

Using the algorithm with Gallas parameter estimates
For a real-data MRMC study analyzed by the Gallas method, 15,31 a method has been developed to convert the U-statistic parameters of empirical AUC and variance estimates to RM model Table 5 Troubleshooting examples. For each example, one of the original parameter estimate values from the VanDyke study, as given by β OR;orig in Fig. 4, is replaced by a value that causes the OR-to-RMH algorithm to fail. These examples show how the value responsible for the algorithm failure can be identified from the alternative parameters x 1 ; : : : ; x 7 and b values using the rules given Table 11. All examples use b_method = unspecified. See Appendix C.4 for the R code that produced these results. Note that to print the x 1 ; : : : ; x 7 variables the option all = T must be included in the print function (see Appendix C.4 for examples).
(a) Original value: σ 2 R∶OR ¼ 0.00154. Altered value: σ 2 R∶OR ¼ 0.154. Output from applying OR-to-RMH algorithm to altered β OR∶orig vector is shown below. Noting that x 3 is the first parameter in the sequence x 1 ; x 2 ; x 3 ; x 4 ; b; x 5 ; x 6 ; x 7 that is missing (NA), the rules in Table 11  Output from applying OR-to-RMH algorithm to altered β OR∶orig vector is shown below. Noting that b is the first parameter in the sequence x 1 ; x 2 ; x 3 ; x 4 ; b; x 5 ; x 6 ; x 7 that is missing (NA), the rules in Table 11  parameters. 32 Alternatively, it has been shown by Hillis 14 that the Gallas MRMC method produces the same empirical AUC single test and difference-of-two-tests variance estimates as the OR method, if the constraints given by Eq. (12) are not imposed on the OR estimates. As a result, OR parameter estimates can be computed from the Gallas parameter estimates using formulas provided in Hillis. 14 Hence, RMH model parameters that correspond to real data studies can be derived using the OR-to-RMH algorithm applied to the transformed Gallas parameter estimates.

Discussion
A previous problem with the original RM model and later generalized versions of it was that the RM model parameters were expressed only in terms of the latent binormal rating distributions, as opposed to the more familiar reader performance measure distributions. Thus, it has been difficult to set RM model parameters such that the simulated data were similar to MRMC data encountered in practice. Assuming the constrained unequal-variance RM model, 3 which we have referred to as the RMH model in this paper, Hillis 6 recently remedied this problem by deriving formulas for computing the OR parameter values that describe the distribution of empirical AUC outcomes computed from RMH simulated data. However, that paper did not provide a reverse OR-to-RMH mapping. This paper overcomes that limitation by deriving a numerical OR-to-RMH algorithm that computes RMH parameter values from a specified set of OR parameter values and by providing an R function to implement the algorithm. The OR-to-RMH algorithm and its corresponding R function make it easy to calibrate the RMH model to produce simulated data that emulate specific real data sets with respect to the distribution of the empirical AUC estimates. The original RM model paper 2 presented several simulation structures that were supposed to represent ROC analyses of representative real data sets, which was useful because then researchers could assess the performance of MRMC analysis methods using a commonly accepted set of RM simulation structures. However, there was a mistake in some of the computations of the RM parameters and the model was limited to equal-variance binormal ROC curves, which are not common. 6 The present approach has several limitations that we hope to remedy in future research. It is limited to generating continuous rating data that emulate a set of inputted OR parameter values describing the distribution of the empirical AUC estimates. Although the simulated continuous rating data can be binned, the distribution of the empirical AUC estimates for the binned data will not as closely emulate the inputted OR parameter values. We suggested a method to adjust the parameter values to better fit ordinal discrete ratings through an iterative simulation approach, but this process is time consuming and we hope to develop RMH-to-OR and OR-to-RMH algorithms, similar to the ones in this paper, that are primarily designed for simulation of rating data with a few ordinal values (e.g., 1, 2, 3, 4, or 5).
The present approach is also limited to the empirical AUC as the reader performance measure. We hope to develop an approach that allows for a semiparametric outcome, such as the binormal AUC.
Finally, our algorithm is based on the RMH model, 3 which assumes that the latent distributions are the same for both tests. Thus, another area for future research is to relax this assumption and develop algorithms for a more general RM model, such as the unconstrained unequal variance model, 6 the generalized RM model, 5 or some other generalization of the original RM model.

Conclusions
The main contributions of this paper are the OR-to-RMH algorithm and the corresponding R software OR_to_RMH function; these contributions make it easy to calibrate RMH model parameters to match real-data OR parameter estimates, thus making it easy to simulate rating data that emulate real data sets for testing MRMC analysis methods or for performing power analysis. These contributions will allow researchers to develop sets of RMH simulation structures that are representative of a wide spectrum of MRMC studies, which can then be used to validate MRMC analysis methods. We expect these new RMH simulation structures will replace the original RM model structures, which were not linked to specific real-world data sets and were limited to equal-variance ROC curves, making the representativeness of the structures difficult to evaluate.

Appendix A: Algorithm Details for Mapping OR Model Parameters to RMH model Parameters
In this section, we derive the mapping from OR model parameters to RMH model parameters.
For the mapping, we assume the RMH model because it has the same number of parameters as the OR model. The mapping from a more general RM model, which includes the RMH model as a special case, to the OR model was derived by Hillis. 6 Modifying this more RM general model by constraining the error variance and variance components involving diseased cases to be equal to those involving nondiseased cases multiplied by b 2 ; b > 0, results in the RMH model. Table 6 presents the resulting analytical RMH-to-OR mapping.
To facilitate the derivation of the reverse (OR-to-RM) mapping, an alternative parameterization for the RMH model is presented in Table 7. Table 7(a) expresses the alternative RMH parameters in terms of the RMH parameters, Table 7(b) presents the constraints on these parameters, and Table 7(c) expresses the RMH model parameters in terms of the alternative RMH parameters. Table 8 expresses the OR parameters in terms of the alternative RMH parameters and Table 9 expresses the alternative RMH parameters in terms of the OR parameters.
The proposed algorithm is presented in Table 10. Steps 1 to 6 replace the OR parameters in Table 8 by specified values and then solve for the corresponding alternative RMH parameter values. Note that these steps incorporate the alternative parameter constraints given in Table 7(b). Using Table 7(c) mappings, step 7 computes the final RMH parameter estimates as functions of the estimated alternative RMH parameter values. Table 6 RMH-to-OR mapping: OR parameters expressed in terms of the RMH model parameters for the empirical AUC.  This table is reprinted, adapted, and revised with permission from Hillis [Ref. 6, Table 3]; notation is the same except that ðσ 2 fixedð−Þ þ σ 2 fixedðþÞ Þ has been replaced by 1 þ b −2 ; b > 0, which results in the RMH model. Table 9, it follows that for each of the alternative parameters other than b, there can be only one solution. It then follows from Table 8(a) that there can be only one solution for the RMH parameters other than b. Hence, if there is more than one solution, they differ only in their b values.

From
Sometimes there is not an exact or approximate solution and the OR-to-RMH algorithm returns missing values. When this happens, changing the values of the inputted OR parameters or changing the b_method option will generally result in a solution, as discussed in Sec. 3.1.3. The algorithm solves for the alternative RMH parameters in the following order: x 1 ; x 2 ; x 3 ; x 4 ; b; x 5 ; x 6 , and x 7 . Because the parameters may require estimates of preceding but not subsequent parameters, all parameters following a parameter with no solution are assigned a missing value by the algorithm. Table 11 describes the appropriate correction action that will produce a solution for the OR-to-RMH algorithm, according to which is the first RMH parameter to not have a solution. Table 7 Alternative parameterization for RMH model parameterization.
(a) Alternative RMH parameters expressed in terms of RMH model parameters. V is defined by Alternative RMH model parameters RMH model parameters Constraints on RMH alternative parameters. These follow from the equations in part (a), nonnegativity of the RMH variance components and constraints Eqs. (6) and (8).
E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 2 3 ; 1 6 4 ; 4 7 7 (c) RMH parameters expressed in terms of alternative RMH parameters. Note that in terms of the alternative parameterization, Table 8 RMH-to-OR mapping: OR parameters for the empirical AUC expressed in terms of the alternative parameterization of the RMH model given in Table 7.
Notes: This table results from replacing the RMH model parameters in Table 6 by the alternative RMH model parameters, as defined in Table 7(a). F BVN ð:; :; ρÞ is the standardized bivariate normal distribution function with correlation ρ; Table 9 Alternative RMH parameters expressed in terms of OR parameters.
Step 3. Solve for x 4 , using the values for x 1 and x 2 obtained in step 1: Step 4. Solve for b using one of the following b_method options. The resulting value of b is used for the remaining steps.  Step 5. Compute OR covariance estimates to be used in step 6. (b) If one of the other two methods was used in step 4, then using the computed value of b and the inputted correlationsr 1 ;r 2 andr 3 , compute a new value for the OR error variance, given bỹ Step 6. Solve for x 5 ; x 6; and x 7 , using the following equations and the values for x 1 ; x 2 ; x 4 , b and d Cov i ; i ¼ 1;2; 3, obtained in steps 1, 3, and 5: Step 7. Solve for the estimated RMH parameter values as functions of the estimated alternative RMH parameter values using the mapping given in Table 7c.
Notes:θ 1 andθ 2 denote specified values of the reader-averaged performance empirical AUCs for tests 1 and 2, respectively;σ 2 R∶OR ;σ 2 T R∶OR , andσ 2 ε∶OR denote specified values of the corresponding OR parameters, andr 1 ;r 2 , andr 3 denote specified values for the OR correlations defined by r i ¼ Cov i∶OR ∕σ 2 ε∶OR . These specified values can be computed from real data or conjectured. F BVN ð:; :; ρÞ is the standardized bivariate normal distribution function with correlation ρ. Note that constraints Eq. (23) in Table 7 have been incorporated into the preceding steps. For the RMH model, the mean-to-sigma ratio varies across readers. Letting q ij denote the meanto-sigma ratio for test i and reader j, Hillis 3 shows the RMH model implies that where E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 8 . 1 ; 1 1 6 ; 3 5 5 It follows for test i that the probability that a reader's ROC curve is noticeably improper (i.e., the absolute value of the mean-to-sigma ratio is less than 2, as discussed in Sec. 3.2.2) is given as E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 8 . 1 ; 1 1 6 ; 2 8 7 where E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 8 . , and x 4 from steps 1 to 4 in Table 6, we want to solve for b such that Eðθ 1j Þ ¼ μ 1;OR and medianðq j Þ ¼ q, where q j is the mean-to-sigma ratio for reader j. When the parameter to the left is the first parameter in the column to have a missing value, try the following corrective action: x 1 NA (should be no problem) x 2 NA (should be no problem) Recall that for test 1, the median separation between the latent normal and abnormal distributions for test 1 across readers is equal to δ 1 . It follows that the median mean-to-sigma ratio is given by E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 8 . 2 ; 1 1 6 ; 6 9 9 and hence E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 2 4 ; 1 1 6 ; 6 4 6 From Table 6, we can write E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 8 . 2 ; 1 1 6 ; 6 0 1 Table 7, it follows that E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 2 5 ; 1 1 6 ; 5 4 3 which implies E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 8 . 2 ; 1 1 6 ; 4 0 2 or equivalently E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 2 6 ; 1 1 6 ; 3 3 7 Collecting terms in Eq. (26) Solving for b −1 using the quadratic equation formula yields where E Q -T A R G E T ; t e m p : i n t r a l i n k -; s e c 8 . 2 ; 1 1 6 ; 1 6 4  Table 4 > VanDyke_OR_orig_values <data.frame(n0 = 69, n1 = 45, AUC1 = 0.897, + AUC2 = 0.941, var_R = 0.00154, var_TR = 0.000208, var_error = 0.000788, + corr1 = 0.433, corr2 = 0.430, corr3 = 0.299) > Table4_OR1 <-VanDyke_OR_orig_values[c(1,1,1),] #creates data frame with 3 rows, > # each the same as VanDyke_OR_orig_values > Table4_OR2 <data.frame(b_method=c("unspecified", "mean_to_sigma","specified"), + b_input = c(NA,NA,1), mean_sig_input = c(NA,5.2,NA)) > Table4_OR  *Note that with mean_to_sigma = mean_to_sigma or specified it is not necessary to specify a value for var_error, or the value can be NA > Table4_RMH <-OR_to_RMH(Table4_OR) > print(" Table 4 RMH parameter values") [1] "

Disclosures
No conflicts of interest, financial or otherwise, are declared by the authors.
Stephen L. Hillis is a research professor in the Departments of Radiology and Biostatistics at the University of Iowa. He received his PhD in statistics in 1987 and his MFA degree in music 1978, both from the University of Iowa. He is the author of more than 100 peer-reviewed journal articles and four book chapters. Since 1998, his research has focused on methodology for multireader diagnostic radiologic imaging studies.
Brian J. Smith is a professor in the Department of Biostatistics at the University of Iowa and director of the Biostatistics Core in the Holden Comprehensive Cancer Center. He received his PhD in biostatistics in 2001 from the University of Iowa. His research is cancer focused and includes statistical computing, predictive modeling, and methods for medical imaging studies.
Weijie Chen is a research physicist in the Division of Imaging, Diagnostics, and Software Reliability, Office of Science and Engineering Laboratories, CDRH, US FDA, where he conducts research and regulatory reviews of medical devices. He earned his PhD in Medical Physics in 2007 from the University of Chicago. He has published 36 peer-reviewed journal articles, 31 proceedings papers, two book chapters, two editorials, and one patent. His research interests include performance characterization and assessment methodologies for imaging and AI/ML/ CAD devices.