A Risk Prediction Model for CEA in Patients With ACAS
A Risk Prediction Model for CEA in Patients With ACAS
I rise to congratulate Dr Conrad and his colleagues on an important and timely study that will help us focus on subselecting the asymptomatic group of patients in which the risk-to-benefit ratio is favorable for CEA. Because this kind of a study is mainly a statistical exercise, my comments will revolve primarily around the statistical analysis and its interpretation.
The authors have produced a model showing that the predictors are associated with the outcome. And there appears to be a positive linear relationship between the risk course produced by the model and the likelihood of dying within 5 years. However, this does not answer the question of how useful a model like this would be in actual practice. That is determined by its sensitivity and specificity at various risk score points.
Even more important in this case is the positive predictive value, or the percentage of those predicted to die, who actually do die. In the highest risk score grouping that the author lists, equal or higher than 15 points, the death rate is 66%. If this death rate were used as a reason to deny CEA to patients in this risk-score grouping, approximately one-third of these patients would be denied this treatment, despite many potentially benefitting from the procedure. That seems a little high for use in the real world.
There is probably a high-risk score where the positive predictive value reaches into the 90% range, where we might feel more comfortable denying a treatment, knowing that maybe only 5% to 10% of the time we will be denying care to those who might have benefited. So why did the authors use a maximum of 15 score?
In evaluating whether the model validates a new sample, the practical issue is not whether the ROC curves are significantly different in the derivation and the validation samples, but whether the usable cut point is different.
If we use a risk score cut point of 23 to exclude patients from CEA, because it has a false-positive rate of only 5%, as clinicians we would like to know whether that risk score cut point is reliable.
So Iwould like to know, first,what cut point produces a positive predictive value of 95% in the derivation sample? And second, does the same cut point also produce a positive predictive value of 95% in the validation sample?
And third, you have used death as an outcome, which is a patient-level variable; each patient can only be counted once in the data file with that outcome. With mortality as the outcome, the data file should have 1791 lines, not 2004. If the difference of 213 between these 2 numbers indicates the number of patients who had bilateral procedures, can you confirm that these bilateral procedures were not done during the same operation? If some, or all of them, were, it would seem incorrect to say that the 30-day mortality rate was 0.7%. It should be higher, because death is one outcome variable that cannot be counted twice in 1 patient.
Now, allow me to ask you about a couple of clinical scenarios using your model. Ifwe have a 17-year-old patient with neck radiation and COPD or an 80-year-old patient with coronary artery disease and diabetes, which we see all the time, these 2 scenarios would have a score of 12. So would you deny surgery for these kinds of patients?
Finally, I am glad the authors differentiated between actuarial survival and actual survival and used the terms appropriately. Unfortunately confusing these 2 sometimes occurs in this type of study.
Response From Mark F. Conrad (Boston, MA)
I would like to start by addressing the use of deaths and using the entire cohort. This is something we struggled with when we first started the study. Clearly, if both sides were done on the same day, it would be erroneous to do so. However, we did not perform any carotid endarterectomies bilaterally on the same day. We did have several patients who were staged within 2 months of each other, but the majority of these had an endarterectomy and then an additional one remotely 2 to 3 years later, sometimes longer. And what we've found is, in some of those patients, their risk factors had changed. So you're going to have another dilemma with deciding whether or not to operate on them.
Beyond that, because tracking the outcome of a patient's first CEA ends at 5 years, if a second CEA is performed at some later time, tracking the outcome of this second CEA is a separate 5-year period. There's a chance that the outcome of a patient's first CEA would be positive, and if they died later, it would be attributed to the second CEA. So you're not necessarily having double lives and double deaths with each patient, and that's why we included them. But I agree it could be a flaw of the study.
You astutely point out one of the problemswith trying to predict survival and all-cause mortality after an operation. It's much easier to predict the disease-specific mortality. But we can't take into account things such as previously unidentified cancer, or getting hit by a bus, or falling and breaking a hip. These are things that I, at least, actuarially couldn't figure out. I think it is very difficult to make a scoring system that is usable and accurate.
And that's why the CEA statistics showed we were only 0.74. We were not in the 80s or 90s with the way we could discriminate. Unfortunately, I cannot do a positive predictive value on this model. And the reason is that there were deaths in each of the levels. So even some of the patients with zero as a score died. When looking at patients with a score of 18, which was the largest score in the study, the survival rate was 33%.
We couldn't go further out because we had no patients who scored higher than 18, despite the fact that the maximum possible score is 35.We're considered a pretty aggressive group, but the good thing about that high score of 18 is that it means that we're not doing endarterectomies on 90-year-old patients who are oxygen dependent and on dialysis.
Two of the patients were 12 years old. We approach carotid disease using a risk/benefit ratio. With every patient we examine, we have to decide whether the risk of the operation is commensurate with the benefit they may receive. We know the risks of the operations. Among patients treated by our group, the stroke and death rate was 1.5% in 30 days.
But the benefits are little more difficult to discern. There's been a lot written about preventing stroke, and we certainly have looked at this.Whether we consider the degree of stenosis (probably the biggest driver of risk), the progression of disease, or plaque morphology, the literature doesn't support attempts to predict which patients are going to survive longer on the basis of those risks. We all have our foot-ofthe- bed test, but that's something that's difficult to produce objectively, and objective tests were the goal of this study.
So, would I turn your 2 hypothetical patients down for surgery? It really depends. If the COPD made the patient oxygen dependent and neck radiation was bad and the degree of stenosis was low, I might do that. Clearly, if somebody comes in with a screaming lesion, this system is not something that you're going to use to assess the appropriateness of EAC.
But if the patient has a first-ever ultrasonography, and the systolic velocity is 250 to 290, while the diastolic is normal, arriving at our office at the age of 90 years and oxygen dependent and on dialysis, then we are probably not going to operate on that person. Whereas were the patient 60 years old and had never taken a statin, we would put them on a statin and operate.
As you clearly stated, there's honest controversy regarding CEA for asymptomatic disease. I don't think your study addresses the appropriateness of CEA in asymptomatic patients, but rather investigates whether, if CEA is appropriate, will the patient live long enough to benefit from that operation or not? So I would like your thoughts on my next comment.
I believe that the true value of this study will depend on a careful risk-adjusted assessment of patients with asymptomatic carotid disease. And that assessment should include high-quality imaging of the carotid lesion itself, brain imaging, and whether there are associated infarcts, specific biomarkers of carotid atherosclerosis, and perhaps other testing, such as distal embolic phenomenon with transcranial Doppler in the asymptomatic patient.
Response From Mark F. Conrad (Boston, MA)
I agree that, in the future, we will probably have better imaging that will help us discern which patients are going to be good candidates for CEA. Right now, our best assessment is degree of stenosis, although we do look at plaque morphology and plaque progression. The biomarkers will most probably be something we use in the future, but not something we use now.
The reason for this study was because we found that although carotid disease, which for many years was considered a surgical disease, treated with carotid stenting, was being reconsidered, now that cardiologists and interventionalists got involved. In our institution, there has been a very big push to change the treatment of carotid disease into a multidisciplinary approach, just like a tumor board.
The problem is that all the multiple specialties have advocates for various procedures. We have neurologists in our group who think nobody should get an endarterectomy. As we try to figure out which metrics we can use, I have found that what we're really missing is a long-term follow-up component. I felt that this is something that could complement the other studies that are out there as we make that decision.
Although this is a very nice paper, I'm concerned about your conclusions. The stroke and death rate that you report is 1.5%. The annual stroke risk from an asymptomatic carotid is generally assumed to be 2%. So once you get past a year, in somebody who doesn't die, you have a benefit. Your risk model is a good effort, but the categories you use are broad: coronary artery disease is a broad spectrum, as is neck radiation. Using age as a factor is a significant concern, because there is no evidence that age affects perioperative mortality or morbidity.
If one third of your patients are alive at 5 years in the worst case, and you have a 1.5% mortality rate, I would not feel comfortable denying anyone surgery. This is still a "foot-of-the-bed issue" and your results indicate your selection criterion is working well. What I'm concerned about is that opponents of CEA for asymptomatic patients, like Anne Abbot, will use your data to support their position. Given your results, you should consider looking at a 1- or 2-year survival rate. Once past 1 year's survival, you have benefited these patients. Can you comment on that?
Response From Mark F. Conrad (Boston, MA)
Sure. I agree with you. We can look at the survival rate on a shorter term. However, the problem is that the initial prospective studies have stated that you need 5 years to properly assess the outcome. The ACAS results indicate that 5 years is when the benefit was seen most clearly. That is why we chose that end point.
This was not meant to be an absolute score, such that if you have a patient with a score of 12, an operation is automatically denied. I looked at the results of the study the other way, that if a patient has score a from zero to 11, it's going to be harder for the neurologists to say that the patient shouldn't have an operation.
I hope when you write a paper, and the paper comes into print, that's the focus, because I think that a lot of people are going to take the reverse focus. And I think that you may inadvertently do yourselves an injustice, because you're an excellent group with excellent results, and you shouldn't deny patients surgery.
Response From Mark F. Conrad (Boston, MA)
I appreciate your warning; Dr Cambria and the rest of our team are very aware of the potential for misinterpretation.
Discussants
Anton N. Sidawy (Washington, DC)
I rise to congratulate Dr Conrad and his colleagues on an important and timely study that will help us focus on subselecting the asymptomatic group of patients in which the risk-to-benefit ratio is favorable for CEA. Because this kind of a study is mainly a statistical exercise, my comments will revolve primarily around the statistical analysis and its interpretation.
The authors have produced a model showing that the predictors are associated with the outcome. And there appears to be a positive linear relationship between the risk course produced by the model and the likelihood of dying within 5 years. However, this does not answer the question of how useful a model like this would be in actual practice. That is determined by its sensitivity and specificity at various risk score points.
Even more important in this case is the positive predictive value, or the percentage of those predicted to die, who actually do die. In the highest risk score grouping that the author lists, equal or higher than 15 points, the death rate is 66%. If this death rate were used as a reason to deny CEA to patients in this risk-score grouping, approximately one-third of these patients would be denied this treatment, despite many potentially benefitting from the procedure. That seems a little high for use in the real world.
There is probably a high-risk score where the positive predictive value reaches into the 90% range, where we might feel more comfortable denying a treatment, knowing that maybe only 5% to 10% of the time we will be denying care to those who might have benefited. So why did the authors use a maximum of 15 score?
In evaluating whether the model validates a new sample, the practical issue is not whether the ROC curves are significantly different in the derivation and the validation samples, but whether the usable cut point is different.
If we use a risk score cut point of 23 to exclude patients from CEA, because it has a false-positive rate of only 5%, as clinicians we would like to know whether that risk score cut point is reliable.
So Iwould like to know, first,what cut point produces a positive predictive value of 95% in the derivation sample? And second, does the same cut point also produce a positive predictive value of 95% in the validation sample?
And third, you have used death as an outcome, which is a patient-level variable; each patient can only be counted once in the data file with that outcome. With mortality as the outcome, the data file should have 1791 lines, not 2004. If the difference of 213 between these 2 numbers indicates the number of patients who had bilateral procedures, can you confirm that these bilateral procedures were not done during the same operation? If some, or all of them, were, it would seem incorrect to say that the 30-day mortality rate was 0.7%. It should be higher, because death is one outcome variable that cannot be counted twice in 1 patient.
Now, allow me to ask you about a couple of clinical scenarios using your model. Ifwe have a 17-year-old patient with neck radiation and COPD or an 80-year-old patient with coronary artery disease and diabetes, which we see all the time, these 2 scenarios would have a score of 12. So would you deny surgery for these kinds of patients?
Finally, I am glad the authors differentiated between actuarial survival and actual survival and used the terms appropriately. Unfortunately confusing these 2 sometimes occurs in this type of study.
Response From Mark F. Conrad (Boston, MA)
I would like to start by addressing the use of deaths and using the entire cohort. This is something we struggled with when we first started the study. Clearly, if both sides were done on the same day, it would be erroneous to do so. However, we did not perform any carotid endarterectomies bilaterally on the same day. We did have several patients who were staged within 2 months of each other, but the majority of these had an endarterectomy and then an additional one remotely 2 to 3 years later, sometimes longer. And what we've found is, in some of those patients, their risk factors had changed. So you're going to have another dilemma with deciding whether or not to operate on them.
Beyond that, because tracking the outcome of a patient's first CEA ends at 5 years, if a second CEA is performed at some later time, tracking the outcome of this second CEA is a separate 5-year period. There's a chance that the outcome of a patient's first CEA would be positive, and if they died later, it would be attributed to the second CEA. So you're not necessarily having double lives and double deaths with each patient, and that's why we included them. But I agree it could be a flaw of the study.
You astutely point out one of the problemswith trying to predict survival and all-cause mortality after an operation. It's much easier to predict the disease-specific mortality. But we can't take into account things such as previously unidentified cancer, or getting hit by a bus, or falling and breaking a hip. These are things that I, at least, actuarially couldn't figure out. I think it is very difficult to make a scoring system that is usable and accurate.
And that's why the CEA statistics showed we were only 0.74. We were not in the 80s or 90s with the way we could discriminate. Unfortunately, I cannot do a positive predictive value on this model. And the reason is that there were deaths in each of the levels. So even some of the patients with zero as a score died. When looking at patients with a score of 18, which was the largest score in the study, the survival rate was 33%.
We couldn't go further out because we had no patients who scored higher than 18, despite the fact that the maximum possible score is 35.We're considered a pretty aggressive group, but the good thing about that high score of 18 is that it means that we're not doing endarterectomies on 90-year-old patients who are oxygen dependent and on dialysis.
Two of the patients were 12 years old. We approach carotid disease using a risk/benefit ratio. With every patient we examine, we have to decide whether the risk of the operation is commensurate with the benefit they may receive. We know the risks of the operations. Among patients treated by our group, the stroke and death rate was 1.5% in 30 days.
But the benefits are little more difficult to discern. There's been a lot written about preventing stroke, and we certainly have looked at this.Whether we consider the degree of stenosis (probably the biggest driver of risk), the progression of disease, or plaque morphology, the literature doesn't support attempts to predict which patients are going to survive longer on the basis of those risks. We all have our foot-ofthe- bed test, but that's something that's difficult to produce objectively, and objective tests were the goal of this study.
So, would I turn your 2 hypothetical patients down for surgery? It really depends. If the COPD made the patient oxygen dependent and neck radiation was bad and the degree of stenosis was low, I might do that. Clearly, if somebody comes in with a screaming lesion, this system is not something that you're going to use to assess the appropriateness of EAC.
But if the patient has a first-ever ultrasonography, and the systolic velocity is 250 to 290, while the diastolic is normal, arriving at our office at the age of 90 years and oxygen dependent and on dialysis, then we are probably not going to operate on that person. Whereas were the patient 60 years old and had never taken a statin, we would put them on a statin and operate.
Anthony J. Comerota (Toledo, OH)
As you clearly stated, there's honest controversy regarding CEA for asymptomatic disease. I don't think your study addresses the appropriateness of CEA in asymptomatic patients, but rather investigates whether, if CEA is appropriate, will the patient live long enough to benefit from that operation or not? So I would like your thoughts on my next comment.
I believe that the true value of this study will depend on a careful risk-adjusted assessment of patients with asymptomatic carotid disease. And that assessment should include high-quality imaging of the carotid lesion itself, brain imaging, and whether there are associated infarcts, specific biomarkers of carotid atherosclerosis, and perhaps other testing, such as distal embolic phenomenon with transcranial Doppler in the asymptomatic patient.
Response From Mark F. Conrad (Boston, MA)
I agree that, in the future, we will probably have better imaging that will help us discern which patients are going to be good candidates for CEA. Right now, our best assessment is degree of stenosis, although we do look at plaque morphology and plaque progression. The biomarkers will most probably be something we use in the future, but not something we use now.
The reason for this study was because we found that although carotid disease, which for many years was considered a surgical disease, treated with carotid stenting, was being reconsidered, now that cardiologists and interventionalists got involved. In our institution, there has been a very big push to change the treatment of carotid disease into a multidisciplinary approach, just like a tumor board.
The problem is that all the multiple specialties have advocates for various procedures. We have neurologists in our group who think nobody should get an endarterectomy. As we try to figure out which metrics we can use, I have found that what we're really missing is a long-term follow-up component. I felt that this is something that could complement the other studies that are out there as we make that decision.
John J. Ricotta (Washington, DC)
Although this is a very nice paper, I'm concerned about your conclusions. The stroke and death rate that you report is 1.5%. The annual stroke risk from an asymptomatic carotid is generally assumed to be 2%. So once you get past a year, in somebody who doesn't die, you have a benefit. Your risk model is a good effort, but the categories you use are broad: coronary artery disease is a broad spectrum, as is neck radiation. Using age as a factor is a significant concern, because there is no evidence that age affects perioperative mortality or morbidity.
If one third of your patients are alive at 5 years in the worst case, and you have a 1.5% mortality rate, I would not feel comfortable denying anyone surgery. This is still a "foot-of-the-bed issue" and your results indicate your selection criterion is working well. What I'm concerned about is that opponents of CEA for asymptomatic patients, like Anne Abbot, will use your data to support their position. Given your results, you should consider looking at a 1- or 2-year survival rate. Once past 1 year's survival, you have benefited these patients. Can you comment on that?
Response From Mark F. Conrad (Boston, MA)
Sure. I agree with you. We can look at the survival rate on a shorter term. However, the problem is that the initial prospective studies have stated that you need 5 years to properly assess the outcome. The ACAS results indicate that 5 years is when the benefit was seen most clearly. That is why we chose that end point.
This was not meant to be an absolute score, such that if you have a patient with a score of 12, an operation is automatically denied. I looked at the results of the study the other way, that if a patient has score a from zero to 11, it's going to be harder for the neurologists to say that the patient shouldn't have an operation.
John J. Ricotta (Washington, DC)
I hope when you write a paper, and the paper comes into print, that's the focus, because I think that a lot of people are going to take the reverse focus. And I think that you may inadvertently do yourselves an injustice, because you're an excellent group with excellent results, and you shouldn't deny patients surgery.
Response From Mark F. Conrad (Boston, MA)
I appreciate your warning; Dr Cambria and the rest of our team are very aware of the potential for misinterpretation.