On The Rising Tide Podcast, scientists from the independent, nonprofit Center for Genomic Interpretation discuss with leading experts the need to further raise the bar on accuracy and quality in clinical genetics, genomics, and precision medicine. Only through improving accuracy and quality can the promise of precision medicine be realized. CGI’s ELEVATEGENETICS services are made available to health insurers and stakeholders, to help them identify the most accurate and clinically useful genetic and genomic tests, and to help them steer clear of low quality tests that have the potential to lead to patient harm and wasted expenditure.
Podcast Introduction 00:00 to 08:17
Interview begins at 08:18
MRAZ: Imagine your mom has breast cancer and she’s positive for a pathogenic gene variant. She calls it her BRCA mutation and tells you it caused her breast cancer. Your doctor wants to test you for the exact BRCA mutation your mom has, but it’s hard to get a copy of her genetic test report, so your doctor just orders you a full genetic panel for hereditary breast cancer. Results come in, and you meet with your doctor who explains, “You have a variant, but it must be a different variant than your mom’s because yours is a variant of uncertain significance. Most variants like these are eventually updated to be benign, so that’s good news. Nothing to act on.” You snap a picture of your test report to refer to later.
Six months go by, and your mom finally shows you her positive genetic test report. She’s already had a bilateral mastectomy because of her BRCA mutation. You pull up the photo of your own genetic test result. You feel your stomach drop. It’s the same genetic variant, but the reports don’t match. One result says “positive” and one says “variant of uncertain significance.” You also notice the genetic tests were done at two different labs, so how can these results be different? Which result is correct?
Today we will be diving into genetic variant classification. What might cause two labs to report the same genetic variant differently? To talk with us today is Dr. Bryan Gall, PhD, MLS(ASCP) CM, a Clinical Variant Curation Team Supervisor working in the clinical genomic testing space, who specializes in clinical variant interpretation.
GALL: Variant interpretation has been the Achilles heel of genetic testing. I also like to think of it as the bottleneck of genetic testing.
MRAZ: Dr. Gall touches on who is invested in the accuracy of variant interpretation.
GALL: Not only does a quality variant interpretation support good patient care, it helps companies that actually provide this testing, it helps the companies that provide the supplies, and furthermore, it helps the people who are actually paying for it all, the insurers and that’s why people in the insurance industry may ultimately be some of the drivers towards increased quality care in the genetic industry.
MRAZ: Though the ACMG guidelines provide a foundation for variant interpretation, the art of variant interpretation is not fully objective, and Dr. Gall discusses how discordance can arise.
GALL: It does all seem very regimented based on that rubric, but the fact of the matter is, a lot of times in science there is that degree of subjectivity where certain rules or certain papers or case studies don’t necessarily fall into that perfect little box. And because of that we do see a lot of, I shouldn’t say a lot, but a good amount of subjectivity in our industry. Some of the key factors that actually cause this discrepancy between labs are some issues where like, I had mentioned the ACMG guidelines are now over five years old, so a lot of labs that have been using those ACMG guidelines have kind of modified them individually, and they’re not necessarily well published how they are modifying those guidelines that could very well swing the interpretation of a variant from non-pathogenic to pathogenic of some sort.
MRAZ: Genetics has increasingly become a booming industry. Dr. Gall addresses quality trends that may come into play for variant interpretation competencies as the industry has expanded.
GALL: A lot of your more well-known labs are typically going to have some sort of internal competency evaluation. But there are a lot of newer labs out there that we just don’t know what level of quality control they’re doing as far as the interpretation analysis. A lot of times these individual labs are sort of black boxes, where you don’t necessarily know what one genetic testing company is using as their modified guidelines, and they may not want to disclose that information to other labs, because it gives them a competitive advantage. And a lot of times too, there’s just not much crosstalk between labs. And that’s really why groups like the expert panels are really really positive in my opinion, because not only does it allow you to come up with a centralized interpretation for a variant, but it also kind of forces people from different industries and different companies to kind of work together towards creating this.
MRAZ: Dr. Gall discusses how AI has been a game changer for variant interpretation, but still is considered a scientific tool versus a competitor in the eyes of variant interpretation scientists, due to some of the limitations of AI.
GALL: AI is a huge boost to what we do. You can train an AI to actually look for your variant of interest throughout the literature, and it can just search all the text and say, “Oh. Hey, I found this– I found your information right here. This paper is probably useful for you, here you go.” This saves us hours of looking through the internet for, basically Googling, or the scientific version of Google, which is PubMed, these variants for hours and hours and only coming out with a few papers. This ultimately allows variant interpretation scientists to have more time to actually dig into the literature and kind of interpret those data in the context of each individual variant. So it gives us more time to actually look into the data that AI can’t necessarily interpret at this point. So artificial intelligence is really good at pulling out key phrases, but we haven’t really gotten it to the point where it’s great at understanding what those key phrases mean. To date, they haven’t really perfected the ability to identify, let’s say, a paper that says, “Hey there’s this person affected with this disease, and they’re a homozygote for this variant.” It’s something that, as of now, we still need the human input to actually determine whether or not those type of situations exist. So as of now, that’s really what’s holding back artificial intelligence, is kind of that lack of a consensus, and the difficulty of interpreting scientific literature.
MRAZ: Dr. Gall emphasizes that quality care extends into variant interpretation practices and should be an approachable topic for all involved. He shares how we can all raise the tide when it comes to genetic variant interpretation practices.
GALL: We still need that human input, and you need a quality oversight in how you’re doing your variant interpretation. And that’s for patient safety. That’s also for your company’s safety, in all honesty, because you know one missed diagnosis can dramatically affect a patient’s life and can result in pretty significant lawsuits. A lot of ways that we can raise the tide together is as simple as asking questions. As a patient, don’t be afraid to ask questions of your doctor or your genetic counselor. If you’re a physician, ask questions of the test provider. As scientists, we’re all really really interested in talking about what we do, I mean it’s really why we invested so much time. So, take advantage of that. Ask questions. On the business side of things, if you’re an executive or some sort of shareholder in a genetics company, really try to talk to the scientists who are on the ground kind of developing these tests. I really think that we need as businessmen and scientists to kind of come together and collaborate more than is normal.
GARLAPOW: I’m Dr. Megan Garlapow with the Center for Genomic Interpretation and you’re listening to the Rising Tide Podcast, where we learn from experts about improving the accuracy and quality of precision medicine and clinical genetics and genomics. Please note that this podcast does not provide medical advice and is not intended to be a substitute for professional medical advice, diagnosis, or treatment. Always seek the advice of your physician or other qualified healthcare provider with any questions you may have regarding a medical condition. Additionally, comments of the Rising Tides’ guests are their own and do not necessarily reflect the position of the Center for Genomic Interpretation.
Today I am joined by Dr. Bryan Gall, a clinical variant curator with Natera and laboratory consultant, he has a wide range of experiences that we will discuss in more detail later. Please note that Dr. Gall’s views are his own and do not necessarily represent the views of his employer. Additionally, some of what we discuss is speculative in nature and/or describes anecdotal evidence.
In this episode of the Rising Tide Podcast we discuss, among other topics, the emerging role of artificial intelligence, also known as machine learning, in the field of variant interpretation. Variant interpretation occurs when a variant classification scientist, such as Dr. Gall, assesses available research and data to determine whether a variant is likely pathogenic or pathogenic, a variant of unknown significance, called a VUS, or likely benign or benign. Variant interpretation is hard. You’ve heard me describe it previously on this podcast as the Achilles heel of clinical genetics and genomics. Recently, there has been an effort made by major players in the field to sell prospective startup genetic laboratories on the idea that artificial intelligence is a cost-effective way to replace human interpretation of genetic variants, which are commonly referred to as mutations outside of the field. This would be a wonderful step forward for the industry which has long been bottlenecked by access to trained scientists who interpret the consequences of genetic variants, but it might be too good to be true for right now. Bryan, thank you for joining me today.
GALL: Oh you’re welcome, it’s great to be here!
GARLAPOW: Yay. Can you expand on what a variant interpretation scientist does, and what role they play in genetic testing?
GALL: Oh yeah, absolutely. So our role is kind of sitting in between the detection type sciences that you would normally expect. So a lot of times in genetic science, we are given just a lot of data, and it’s kind of difficult to understand what those data actually mean. So my job is to kind of sit between what comes off of an instrument and what the doctor, physician, genetic counselor will actually recommend for a patient. So we almost act as translators, would be the best way to describe it. We kind of translate these raw data into clinical interpretations. So we identify which one of these many genetic variants that are identified are actually relevant to you as a patient, or you as the physician, as something that would be important in pathogenesis or might be something that would cause a disease in a patient. It’s also important to note that we do not, we’re typically scientists who do not have clinical training, so a lot of what we do does not necessarily suggest clinical treatment options. But, we want to give physicians, physicians and genetic counselors, all of the information that they need to make a proper clinical diagnosis or clinical treatment plan for the patients. So we’re kind of that in-between between the raw data and the clinical action.
GARLAPOW: I think that’s really important to note that we are not diagnosticians. Well, I’m not a variant classification scientist anyways, but variant classification scientists are not diagnosticians, but they are part of the diagnostic journey, quite potentially. But what you just described, that sounds really hard. That sounds really time consuming. How much time do you spend curating a single variant, and how many variants are generally seen in a single patient? Are there any differences in the time and effort across types of variants? That is, is it more challenging to classify say a VUS or a likely pathogenic or pathogenic or likely benign or benign variant? Or are they all sort of equivalent? Does it vary from patient sample to patient sample? What’s been your experience there?
GALL: Now this is really a tough question to answer because it does, it’s so difficult to give a broad answer to this question. I do want to kind of go back real quickly and point out something that you said kind of in the introduction, that variant interpretation has been the Achilles heel of genetic testing. I also like to think of it as the bottleneck of genetic testing, where there are some clinical cases that we get that are just really really easy, is the best way to describe them. You know you might get a patient that comes through on a panel, and they have, let’s say 25 variants. None of them have any clinical evidence to them, there’s nothing that you really have to dig into all that deeply. And those you can go through fairly quickly. I mean they can take on the scale of one to five minutes to go through those type of variants, and that’s usually with some sort of computer assistance that we’ll probably discuss in a little bit more detail later. But then there are some of these cases that you know they might only have one variant, but that variant just has 50 papers published on it, it’s really really well studied and as a variant interpretation scientist, you really have to go through each and every one of those papers to identify exactly what risk our patient has at any given time. And this is something that can be very time consuming, as anybody who’s gone through basic scientific literature can tell you, a lot of these are very dense, and they’re very difficult to get through quickly. So that’s why it really requires somebody with that background information to kind of get through these papers, and that’s why it could take a little bit longer. Now some of the other things that tend to affect the duration or how much time it takes for us to get through a case, are things like the size of a panel. So for example, a lot of these next-generation sequencing panels are hundreds of genes, to say that, and I’ve actually had the pleasure of kind of going through that expansion in my own work as a curation scientist. When I joined the team, our panel was only 27 genes, so we’d get lucky if we’d have maybe 10 variants or 20 variants in a patient. Most of those would be fairly easy, and you can get through them pretty quickly. But we had recently expanded our panel out to 274 genes, so we had a full log-fold expansion in the number of genes that we were curating at any time. And now we can have patients going through that, we see as many as 50 variants identified, even more, so that can take a lot more time to go through. A lot of this can be, and I guess that kind of goes into expanding even further, where a lot of labs are starting to introduce whole exome panels, so they’re looking at, and that basically means that they’re looking at every coding sequence in our DNA, so every single gene, they’re amplifying and detecting variants in every single coding exon. So you can imagine the rapid expansion and the number of variants that are going to be in a single case in that situation. And even further on, there are some labs that are really starting to dig into whole genome sequencing, and that is a very large task to say the least. So, and I guess another thing that really affects the duration, or how long it takes for us to curate a case, kind of goes into something we call database maturity. So a lot of labs that have been around for a while tend to have a history of all of the variants that they have seen in the past, and in that history, a variant interpreter or interpretation scientists, like myself, can kind of go back and look through historical curate or historical work by other scientists, like myself, and we can essentially use that as a building block towards our new curation or our new interpretation of a variant. And that really expedites the process as well, where you’re primarily doing a review of papers that have already been shown to have some sort of clinical evidence in them, and it, like I said, it makes it a lot easier. Plus, the other thing too, is a lot of labs will keep a historical curation for roughly six months, so that basically means that if you see this variant once in the patient, and somebody goes through and interprets it, that interpretation stays constant for six months, and essentially any time that variant is present in another patient, it’ll almost get auto called by whatever system you’re using. So that can also increase the rate that you’re going through patients. So unfortunately, I don’t necessarily have a great answer for you as to how long it typically takes, but a single variant can go anywhere between a minute long to get through, and up to two, three, four hours, if you get a really really complex variant.
GARLAPOW: Okay. So a huge range in amount of time per variant, a big range and amount of variants per patient, within labs, across labs, based on how these panels are being run, if they’re run as panels, whole genomes, whole exomes, etc. So it sounds really super super interesting for labs, for variant curation scientists, but who are the stakeholders here, Bryan? Who should care about variant interpretation, how it’s executed, whether it’s driven by processes that prioritize quality and accuracy, and why are these people the stakeholders?
GALL: Okay that’s a really good question, and first and foremost the primary stakeholder here is always going to be the patient, because it is the patient that is going to bear the brunt of any inaccuracies that occur throughout the process. So ultimately they’re the ones that have the greatest risk to reward ratio here. And like I said, so if you have a, let’s say some sort of mistake during variant interpretation, and you say that a variant is not pathogenic, so that’ll usually fall in kind of in the more technical terms, as a variant of uncertain significance, likely benign, or benign, and generally that is a variant that a clinician isn’t going to take an action on. Let’s say there’s a mistake there, and a interpretation scientist like myself calls it a VUS, when it’s actually pathogenic. Now that can drastically affect the type of patient care that somebody gets, and can extend the time that it takes for a patient to actually get a diagnosis as well. Or, even worse yet, can result in some sort of negative treatment or negative health-related response due to the treatment. Some of the other key stakeholders in this are generally the genetic testing companies. So for instance, the company that I work for, there is a lot involved in like reputation of quality that goes into the testing, as well as whether or not the the company puts a lot of effort into, or not – I shouldn’t even say effort, but kind of that reputation of having good quality patient care. And obviously that also goes into bottom line of business, where you want to give patients the best possible care, but you also have to make money, or at least break even, a lot of times. So that’s an important stakeholder as well. You’ve also got the instrument companies that create either instruments or any reagents that we use in the actual sample preparation, or the running of any of our sequencers. And the main reason for this, that they are stakeholders, is because they want to make sure that variant interpretation quality is still high, because you don’t necessarily want patients to have a negative interpreter– or negative opinion of genetic testing, so if you have high quality genetic interpretation from a scientist, it’s going to make the patients be more confident in this technology, and therefore companies that supply these type of things are going to be selling more, obviously, so they have a huge stake in variant interpretation as well. Another key one that’s often not thought of would be insurers. So as an insurer, you really want to make sure that your patient is getting high quality diagnosis, because as an insurer, if your patient gets a diagnosis, they can go on a treatment regimen, and generally if they get that diagnosis quickly and accurately, then the cost of care is going to be lower for the insurer. So, not only does a quality variant interpretation support good patient care, it helps companies that actually provide this testing, it helps the companies that provide the supplies, and furthermore, it helps the people who are actually paying for it all, insurers, and that’s why people or people in the insurance agency or insurance industry may ultimately be some of the drivers towards increased quality care in in the genetic industry, which is something that would be very interesting to follow up on in the near distant future.
GARLAPOW: Oh, I totally agree. It’s so interesting to me that so much of the time reimbursement from health insurers from payers is not tied to quality or accuracy in clinical genetics and genomics, and I think there really needs to be a major title change there. But so what you’re describing– you and I in route to our doctorate degrees, we both went to journal clubs, and we read you know the same papers as our colleagues, and we had very different homounderstandings of these papers, and what you’re describing here, it sounds like there’s a lot of subjectivity in determining what variants are pathogenic, what variants are not pathogenic, and you know this affects patients, the care that they receive, how they understand their health care, their family history of health, all of that. What rubric or set of guidelines does the industry follow when making these interpretations? Because there has to be something that’s at least trying to standardize this and remove some of that subjectivity.
GALL: Yeah, absolutely. So we do have kind of a guidance system, as you might call it, called the American College of Medical Genetics guidelines, which were published in 2015. They’re typically referred to as the ACMG guidelines, because every scientific industry really likes its acronyms, so we always call it the ACMG guidelines. And they kind of break down what types of evidence are pushed, or are qualifying for either a pathogenic interpretation or benign interpretation. Kind of one example of this would be, if we have, let’s say you have a patient that presents with a variant, and you want to know whether or not that variant could be causing a disease. Now, if you look into the literature, you can find that there are like, let’s say for example, they have a homozygote for that variant that your patient presented with, and they don’t have any other variants detected in that gene. We can be fairly confident that those evidence are actually or the– that variant because it’s in a homozygous state is actually resulting in the patient’s phenotype. So that is what we would consider with strong evidence, and that’s just one example of some of the guidelines that we follow. Ultimately, you kind of add up, using ACMG guidelines, you kind of add up all those evidence from strong, moderate, and supporting level and see whether it adds up, following their rubric, to be pathogenic, likely pathogenic, a VUS, likely benign, or benign. So it does all seem very regimented in that situation or based on that rubric, but the fact of the matter is, a lot of times in science there is that degree of subjectivity where certain rules don’t necessarily– or certain papers or case studies don’t necessarily fall into that perfect little box, and because of that we do see a lot of, I shouldn’t say a lot, but a good amount of subjectivity in our industry. And there are some resources where you can kind of quantify that. For example, there is a resource called variantexplorer.org, and there’s another one called ClinVar Miner, and these will actually, if you dig in through those sources, you can actually see the number of variants that are submitted into, kind of this centralized data repository that have conflicting interpretations. So I think I’m jumping ahead of myself a little bit here, but there is this very important tool that we use in the industry and it’s called ClinVar, where essentially any lab in the industry or any academic lab can submit variants that they’ve identified and they have performed an interpretation on. And each one of those labs will determine if that variant is going to be, what the classification for that variant will be, and you can get a look through that site to determine how confident various labs are. So, for example, let’s say, I’m just kind of pulling a lab out, we’ll say Invitae might call a lab pathogenic, whereas the next lab over, let’s say GeneDX, might call that likely pathogenic. So you can see where there’s a little bit of play there between those two interpretations. Generally, it– those type of discrepancies aren’t huge, it’s when you have one lab that’s calling a variant pathogenic or likely pathogenic, and then you have another lab that’s actually calling it non-pathogenic, so VUS or below, which unfortunately leads to that subjectivity. But this is kind of drifting away from your original question, where we do have those ACMG guidelines, and there are also some third-party organizations that are creating more detailed guidelines. So ClinGen has a group called SVI that kind of updates the ACMG guidelines as we go through, because as I did mention, the ACMG guidelines were submitted or were generated in 2015, so there have been some changes in the industry since then. And there are some of these third parties that are trying to update those guidelines to increase the– I should say to decrease the subjectivity in variant interpretation. And I will say that there have been improvements in the industry since 2015, to decrease that type of subjectivity.
GARLAPOW: Okay. And notably, the ACMG guidelines are being updated currently, expected to be released this year , the updated version. In a prior episode, I got to chat with Madhuri Hegde, Perkin Elmer’s Chief Scientific Officer, about the upcoming updates and other topics in this area. But so it sounds like even with the guidelines, there can still be discordance across laboratories in variant interpretation and classification. I’m a nerd. I love robots, so from my understanding artificial intelligence requires an input of rules to train it to perform a specific task. The ACMG guidelines seem like a good set of rules that can be used to train artificial intelligence, and we have a lot of data from years of genetic testing. What’s holding artificial intelligence from being implemented in clinical genetics and genomics? What could go wrong?
GALL: So I have to preface this answer by saying that my programming skills are not quite up to the level of programming artificial intelligence. But that being said, from my basic understanding of programming, this type of artificial intelligence is commonly referred to as machine learning. So essentially what this process involves is feeding machines a known data set where you know that if you have these inputs it should send out this export. So for example, let’s say that we know that all of these rules are hit by the ACMG guidelines, and that should pump out the curation for the variant or the interpretation for the variant. And that all sounds great on paper, but the real problem here comes in to the fact that we don’t necessarily have that truth set of data to send to an artificial intelligence so that it can learn on its own how to do these interpretations. And that kind of goes back to what I was saying about ClinVar and the discrepant interpretations that are submitted there, where some of the efforts to train artificial intelligence to date have used primarily curations where all of the labs are in complete concordance. And so a lot of times those are the easiest variants to curate because all the evidence is pointing in one direction. So that makes AI, at least from my understanding, fairly decent at curating or interpreting variants that have very well established interpretations for them. Where this starts to become a little bit gray is when you have these variants that you know some labs call them pathogenic others say that it’s not entirely clear what this variant is actually doing to create– basically doing in the context of a disease. So in those particular type of situations it becomes more difficult to train a computer or artificial intelligence because it doesn’t necessarily know what the right answer is and that makes it very very tricky. And the other situation that kind of comes up in those type of situations is the challenge of actually reading and interpreting scientific literature. So the one thing that’s keeping me employed now is the fact that it’s difficult for an artificial intelligence or machine learning algorithm to look at a scientific paper and pull out keywords or pull out key phrases that would indicate what the paper is trying to say. So to-date they haven’t really perfected the ability to identify let’s say a paper that says, hey there’s this person affected with this disease and they’re a homozygote for this variant. It’s something that as of now we still need the human input to to actually determine whether or not those type of situations exist. So as of now, that’s really what’s holding back artificial intelligence is kind of that lack of a consensus and the difficulty of interpreting scientific literature.
GARLAPOW: That’s so interesting. And we’ve discussed on this podcast before how variant classification isn’t just a science, it’s also an art. And I think that really captures it right there. But you say that there is not much of a consensus in the industry for clinical consequences of some variants. Why? Why is that? What is the source of this discordance?
GALL: Yeah and this also is kind of is a complex question to ask– or answer and it’s a very good question as well. So as I had mentioned earlier there are some discrepancies. There are some papers that we can link to in the podcast that kind of show a little bit more evidence on how common this is throughout the industry. But some of the key factors that actually affect the– or cause this discrepancy between labs are some issues where like I mentioned the ACMG guidelines are now over five years old so a lot of labs that have been using those ACMG guidelines have kind of modified them individually, and they’re not necessarily well published how they are modifying those guidelines. So for example, a lab might treat a patient who has a mild phenotype as a decreased level of evidence. Whereas another lab might treat that same patient with a mild phenotype as a higher level of evidence. So the difference in the confidence of that particular patient will cause a variant to either be curated as pathogenic or it might even drop it all the way down to a variant of uncertain significance. So that’s one situation there and like I mentioned before there are those kind of third party groups that are working on kind of modifying the ACMG guidelines, which I think is going to be a lot of what the new ACMG guidelines  are going to adapt. So that’s kind of helped decrease the discordance some. Some other issues that are causing increased discordance in my opinion are internal databases amongst labs. So as an industry, there isn’t much crosstalk amongst individual labs. So for example, I was mentioning earlier that a lot of well-established labs have this database of all of the previous curations that they have. On top of that, they also have information on whether or not these patients actually presented with the disease. So you can see that over decade– or years and years and years of doing genetic testing these labs might have information that’s not published in the literature so that it’s not available to labs that are, well any of the other labs in the industry that could very well swing the interpretation of a variant from non-pathogenic to pathogenic of some sort. I would also say that another area of discrepancy is that there really isn’t, that much of the oversight of variant interpretation kind of falls in a lab-dependent manner, where a lot of it falls on independent laboratory directors who have to establish quality guidelines. Unlike many of the other industries or — many of the other health testing which is typically covered by agencies like CLIA or CAP that ensure consistency amongst labs genetic testing at this scale of next generation sequencing they haven’t really been able to establish a competency testing system for variant interpretation to date. So I know that CAP is really starting to kind of create something along those lines, but it’s yet to reach full adoption in the industry. So some of it might just be differences in each independent lab. Now a lot of your more well-known labs are typically going to have some sort of internal competency evaluation, but there are a lot of newer labs out there that we just don’t know what level of quality control they’re doing as far as the interpretation analysis. Another kind of issue with– that’s causing discrepancy is, there’s differences in the techniques that individual labs use to actually find their evidence. So some of this discrepancy is just down to something as simple as digging through the literature, which might sound very easy, but when you think about a genetic variant, over the years we’ve had a number of different names for these variants. We’ve called them changes in nomenclature. So you might be looking for a variant that’s called one thing now in 2021 but it might have been called something very different back in the 1990s. So some labs know this and other labs might not have that information and might not be able to look under all of those search terms and that can make it fairly difficult. I guess some of the other things actually come down to, as you were talking about how individuals have different interpretations of the literature. So whereas reviewer one might see something and say, “Oh that’s a quality functional assay, and this supports the pathogenesis of this variant,” reviewer two might say, “Yeah but there are some flaws in this experimental design that make me not trust this.” And this is often why most labs will have two reviewers on any variant, to kind of provide for conflict as best, as bad as that sounds, but conflict between individuals in variant interpretation is actually very healthy, because you want to come to a consensus with all of the information that you have from as many parties as you can, and a lot of times that that boils down to two independent people. But again, that really depends a lot too on the training of your individual variant interpretation scientists.
GARLAPOW: Okay. That’s very interesting Bryan. So has there been much of an effort to rectify these discrepancies? What’s that looked like, if there has been an effort in rectification?
GALL: Yeah and a lot of this is going to be kind of covering things that I’ve already covered but…
GALL: The first one here is going to be your second reviewer. As simple as that sounds, it’s really important to have a second set of eyes on a lot of these variants. Because, well first off, if you’re a patient, you really want to get as many eyes on your case as possible, so that you know that nothing was missed. So second reviewers are really good at catching any errors in variant interpretation as long as they’re thorough. Another one, as I was mentioning previously, is ClinVar. So a variant interpretation scientists can actually go back and say, “Oh look at all these labs that are calling this likely pathogenic. I think that this variant is a variant of uncertain significance. I might have missed some evidence. Maybe I should go back and look at this in more detail.” So it provides an extra tool to scientists like myself to kind of determine whether or not we found everything that we need to make an accurate interpretation. There’s also this great organization through ClinGen that are setting up expert panels, where there are a number of scientists who are experts in whatever disease or phenotype you’re looking at, and they actually go through the literature and they all come together and reach a consensus on what the interpretation of that variant should be. And they’ve made a lot of progress. But the real problem is there’s just so many variants to look at that it is physically impossible for anybody to go through and cover all of them. But like I said they’re making great progress and I really tip my hat to that group. Also, as I mentioned CAP is really starting to get in the infancy of competency evaluation for variant interpretation. And I think that, you know, in the next couple of years we’re going to start seeing more and more oversight in the industry and that’s going to decrease the level of discrepancy between labs. And finally there’s also a few smaller groups out there, most of whom are nonprofits who are trying to increase the quality of genetic testing throughout the industry. And they’re also working on this discrepancy issue because this really is a key bottleneck and kind of Achilles heel as you had mentioned in genetic testing.
GARLAPOW: And just to put in a plug for my nonprofit organization, Center for Genomic Interpretation is definitely one of those organizations. So you’re saying that a lot of these discrepancies are the result of understanding the literature and lab specific modifications to the ACMG guidelines. The lab specific guidelines, should those be easy to rectify? What’s happening there?
GALL: Well, no. To put it simply. So a lot of times, just like any other industry the genetic testing industry is fairly competitive. So a lot of times these individual labs are sort of black boxes, where you don’t necessarily know what one genetic testing company is using as their modified guidelines. And they may not want to disclose that information to other labs because it gives them a competitive advantage. And a lot of times too there’s just not much crosstalk between labs. And that’s really why groups like the expert panels are really really positive in my opinion, because not only does it allow you to come up with a centralized interpretation for a variant, but it also kind of forces people from different industries and different companies to kind of work together towards creating this. And I say that as somebody who’s worked with these expert panels, where it allows you to work with people from different companies, and you kind of get a little bit insight as to what other organizations are doing. So that is something that can help, but like I said there’s just not enough clarity to know for sure where our discrepancies lie, lab to lab. And a lot of times too, I don’t want to make it sound completely bad that these are companies, because a lot of times having individual companies are really really great for innovation, so it really pushes the industry forward by having that level of competition between labs. But there are some places where I really think that we could, and in my opinion should, collaborate a little bit more on issues of patient care and patient safety. And as I mentioned before, also some of those guidelines in ACMG may not actually be in a lab’s SOP, but it’s something that’s kind of, through lab culture and through years of interpreting different variants, they’ve kind of drifted away from the rules, without actually putting them in standard operating procedures. But, again, you know I’m really hoping that as the new ACMG guidelines come out, hopefully next year, that should start to bring all of the labs a little bit closer together as to their process for interpretation. And you know, maybe talk to me in a year or two from now and see how the new guidelines have actually changed that, and whether or not it actually has decreased the– or has increased our ability to rectify differences.
GARLAPOW: With how quickly this field evolves it might be in like three months that we bring you back on.
GALL: Well I’m here if you need me.
GARLAPOW: Yeah. So what are some of the issues associated with literature review, and why couldn’t an unbiased artificial intelligence minimize those issues?
GALL: Yeah, and I’m really glad you used the word “unbiased,” because that is one of the major potentials for artificial intelligence, is kind of allowing an unbiased opinion, because no matter how much we as scientists like to say we’re unbiased, we see a variant, and we see a patient and we might develop some sort of preconceived notion that based on what we’re seeing in this patient that variant must have some sort of role in their disease. But that being said, there are also the limitations of artificial intelligence, which I’ve mentioned previously are the ability to understand literature and to come out with a interpretation of what is being stated in the literature. So artificial intelligence is really good at pulling out key phrases, but we haven’t really gotten it to the point where it’s great at understanding what those key phrases mean. So for one instance would be, let’s say functional studies. So there’s a lot of different functional studies that can be used to determine whether or not a variant is causing some sort of damaging effect to a gene or protein. But the AI may not be able to determine where some of the shortcomings of that experiment are and how significant that experiment is to the general physiology of what that gene is doing in the context of the whole human body. So some of the other places where I’ve seen it struggle, personally, is determining what we call the zygosity of a patient. Now the zygosity is essentially a term used to determine how many copies of your genetic variant are present. So for those of you who’ve been through genetics classes, if you have one copy you’re heterozygote, if you have two copies of that same variant you would be a homozygote, and generally if you have just one copy of that variant your heterozygote. And even if that variant is deleterious, you still have the normal copy of the variant on your other allele. In this case since we get one chromosome from your dad, one chromosome from your mom, if your dad’s chromosome has some sort of variant in it that’s causing a decrease in function your mother’s chromosome can still give you enough of that protein or gene to maintain normal human functions. So that’s something that’s really important for AI to be able to pick out because there’s a huge difference between a patient being heterozygous– heterozygous for a variant and homozygous for a variant, and that even becomes more complicated when you go into what if the person has different variants? So let’s say they have one pathogenic variant on the father’s chromosome and they have a completely different one on the mother’s chromosome. How does the computer, how’s AI able to tell are these variants actually on different chromosomes? Are they actually in the same patient? It becomes much more tricky because there’s not necessarily a standard way of reporting this in the literature. Some papers might describe a person’s variants in a sentence, which is really hard for an AI to pick out. Others might put it in a table. And a lot of times, for anybody who has read scientific literature, it’s not always clear how these researchers chose the format that they did for that particular situation because it’s not always clear even to the people who are in the industry, it can be very confusing to interpret this type of information. So it becomes kind of challenging for somebody to train artificial intelligence to be able to interpret that and that’s really where I think the current Achilles heel for artificial intelligence and variant interpretation is to date.
GARLAPOW: Okay. So, you’ve brought up a number of the limitations of artificial intelligence, AI, in clinical genetics and genomics, variant interpretation, and curation, but what are some of the areas where artificial intelligence could benefit variant interpretation scientists?
GALL: Yeah and that’s a great question too, because up until now it sounds like I’m just trying to save my job by knocking AI everywhere, but the truth of the matter…
GARLAPOW: You don’t want the robots to kick you out?
GALL: Yeah, I would prefer to keep my job long enough to retire, at the very least. But that being said, AI is a huge boost to what we do. My lab in particular uses artificial intelligence to identify the information that we need to come to an interpretation of a variant. So that’s one place that AI is really really good at. So that, AI is capable of looking through like population genetics, because a lot of that are in databases that are very standardized, and it’s easy to train artificial intelligence to interpret those data. So you can tell what the population frequency of a variant is, and that often goes into whether we think a variant is benign or pathogenic or anywhere in between. Some of the other things that it’s really good at is actually pulling literature that is important to the variant. So I mentioned that AI is pretty good at picking out key phrases. So you can train an AI to actually look for your variant of interest throughout the literature, and it can just search all the text and say, “Oh, hey, I found this, I found your information right here, this paper is probably useful for you, here you go.” So this saves us hours of looking through the internet for, basically Googling, or the scientific version of Google which is PubMed, these variants for hours and hours and only coming out with a few papers. So that’s something that AI is just awesome at right now. So, and like I said most of those, there are also a couple of situations where some of the criteria that we use to interpret a variant are in databases like ClinVar, for instance. Artificial intelligence is very good at digging through ClinVar and saying, “Hey, your variant’s been seen before, and these labs reported it as this, this, this, and this. I hope this helps you. Here’s that information.” And yes, it really really does help us a lot and we’re really appreciative of that. And this ultimately allows variant interpretation scientists to have more time to actually dig into the literature and kind of interpret those data in the context of each individual variant. So it gives us more time to actually look into the data that AI can’t necessarily interpret at this point. So ultimately this is a huge, huge help for us. There’s also a new, well I shouldn’t even say new, but it’s becoming more and more adopted, where artificial intelligence is used more and more frequently in actual detection of variants. And this is something that is outside of my area of expertise, but what they’re actually doing is using artificial intelligence to take the raw data that comes off of our sequencers, which is very very difficult to interpret and very very complicated and is teaching it to be more and more accurate. And it’s getting to the point now where the data that are interpreted by artificial intelligence are actually more accurate than any of the algorithms that humans are creating. And that’s becoming a bigger and bigger boon to the industry because it allows us to detect more actual variants and reduces the risk of seeing things like false positives or false negatives in the data.
GARLAPOW: Very cool. So a number of the criteria in the ACMG guidelines do have the capacity to be automated, very nice. How does this lend itself to a hybrid human-artificial intelligence approach to variant curation? Would that reduce the time it takes to perform variant interpretation? How would quality and accuracy be affected?
GALL: Yeah I think I might have jumped the gun on this question a little bit…
GARLAPOW: You did, but let’s dive into it again.
GALL: But you know this is really something that I want to go into a little bit more detail, because I kind of want to reiterate, because the artificial intelligence and machine learning has really really increased the ability for variant interpretation scientists to have the amount of time that is needed, or at least more time, to fully understand these variants and what they mean to the patient. And I really want to highlight that too because, ultimately any time that artificial intelligence can afford us, is more time that we’re allowed to have to actually provide the best information to the physicians or genetic counselors who are ultimately going to decide on patient care. So that ultimately, in my opinion, is something that allows us as variant interpretation scientists to provide better care to the patient. And that, in my opinion, is something that is really important in this industry. But as I said, there are still those areas that can’t quite be automated to date.
GARLAPOW: Yeah, not yet. Hopefully in the future, but right now, no.
GALL: Well, hopefully in about 25 or 30 years when I’m able to retire.
GARLAPOW: Right? So you’ve brought up a number of interesting possibilities here. How could you envision artificial intelligence developing to a point to perform variant interpretations, steal your job, independent of human input? Alternatively, how do you see the role of the variant classification scientist changing in the next 10 years?
GALL: Oh, this is great. So this kind of makes me think of, you had mentioned being a nerd at the beginning of the podcast. I’m also a nerd. This kind of reminds me of all of those scary sci-fi movies, where you’re talking about the singularity and artificial intelligence and blah blah blah. I won’t go into that much detail here, but one of the key things that’s often highlighted when you’re talking about artificial intelligence, or any sort of computing power is Moore’s law. So basically, over time we’re seeing an exponential growth of computer capabilities, and in theory, if that continues on its exponential growth, eventually we are going to get to the point where artificial intelligence can ultimately do this. I don’t have enough information on my side or enough knowledge of artificial intelligence to say when that breakpoint is going to be, but I do foresee it in the distant future, becoming a point where artificial intelligence could perform the majority of what we do as variant interpretation scientists. I still believe that for a very very long time, we’re going to need some human input into the system. But, let’s see what else. So you had mentioned, what was coming up in the next 10 years? So I believe that in the next 10 years we’re not necessarily going to get to the point where artificial intelligence can be 100 percent independent of human input. I believe that we’re going to see more and more collaboration through the industry, through organizations like ClinVar and ClinGen’s expert panel, where more and more labs are going to start submitting their data into publicly available databases. A lot of labs really really have started doing this more frequently. And you know, as more and more labs start putting their information out there, it could make it easier and easier to train AI. I don’t necessarily know that for sure. But some of the other areas where I see our industry going in the next couple years, is a lot more oversight in the industry. So this is probably going to come from governmental organizations like CAP and CLIA, as well as, there’s kind of this emerging role for insurers in the industry, where the insurers don’t want to be spending their money on genetic testing that may not be high quality. So they’re going to start pushing– I believe they’re going to start pushing for an increased level of quality and collaboration throughout the industry. Let’s see– some of the other things that I kind of foresee in the future kind of go along with that increased oversight, is that we’re going to start seeing a shrinking in the genomics market, where a lot of these kind of smaller laboratories that may not be able to meet some of the quality metrics that are going to start showing up as more and more oversight come, are going to start phasing out. And there’s going to be, and the spirit of competition, there’s going to be a number of labs that do prevail, and those are ultimately going to be the ones that were proactive in their approach to increasing the quality of their genetic testing. And a lot of that comes in as you were mentioning in the variant interpretation step, because that’s kind of the one where there’s the most subjectivity at this point. So a lot of this oversight and quality control is going to come in the stage of variant interpretation sciences, at least in my opinion. And you know, it may sound like that’s not necessarily a good thing to have fewer laboratories in the industry, but I would argue that’s actually a better thing, because with fewer companies in the industry, you’re going to see those companies actually start competing to increase quality, as opposed to increasing, or sorry, increasing quality so that they can maintain competitive advantage within this oversight, and that’s going to help out patients,that’s going to help out clinicians, that’s going to help out basically all of the stakeholders, in my opinion. But, you know, like I said, maybe have me back in 10 years, and we’ll see how many of these actually came true.
GARLAPOW: Deal, put it on your calendar.
GALL: Alright, I’ll mark it.
GARLAPOW: So let’s say I am a small startup lab, and I have a shoestring budget, and I’m struggling, and I can’t access all the papers I need to because they’re behind paywalls, and I can’t pay for all of that! And you know, I can’t find a Dr. Bryan Gall, or maybe I can find Dr. Bryan Gall and lure him on board, but I need three of you, and I can only get the one of you. What are the solutions available for me? How can I be successful? Can I be successful?
GALL: So this is a really tough situation you’re putting me in. Yeah so it becomes more difficult I believe as this quality metric becomes more and more important, where these startup labs are not going to have as many solutions as they did early on in the genetics boom that we’ve had. So as I’ve been mentioning, we still need that human input and you need a quality oversight in how you’re doing your variant interpretation. And that’s for patient safety, that’s also for your company’s safety, in all honesty, because you know one missed diagnosis can dramatically affect a patient’s life, and can result in pretty significant lawsuits, which is something that obviously as a small company you do not want. So my suggestion would be to actually look into some less permanent options.There’s a number of people in the industry who are doing consulting type roles, or there are variant scientists that if you have a fairly low throughput, you know you’re early on in your lab work, you might be able to hire somebody on as a consultant that might take you know X number of variants a day, and can help you get to the point where you’re established in a lab and you can actually hire on additional employees as a full-time variant interpretation scientist, or multiple full-time very interpretation scientists. You could also go through rounds of funding, which I know is– kind of sounds inconsiderate, but again this is something that it’s not like a new widget for a GameBoy or – wow, that dated me big time!
–or some sort of video game system, this is something that is really going to affect a person’s life and you don’t want to cut corners when doing that. So if that means another round of funding, that’s just something that you’re going to have to go through with. And that’s really kind of the hallmark of this whole talk here is, honestly cutting corners, for the patient can be in the most extreme situations, can be the difference between life and death and so that’s something that’s very important for everybody who’s involved in this industry to really take to heart and consider as you’re going through this. I also did forget to mention, there are some companies that actually provide software that pull data off of our sequencers, and they have integrated consulting agencies, where they actually hire on the variant interpretation scientists, and as long as you use their software, you have access to their scientists. So there are a number of options available. And kind of on that plan as well, you can use their scientists until the point you get comfortable enough to bring it in-house and actually use your own scientist to perform that action. One of the caveats to that is, you want to bring on a lab director who really knows what they’re doing, and is capable of really critically evaluating what those consulting variant scientists are doing. Because they may be working on a different SOP than you are, or you might have to look into the quality of what they’re doing. So it’s just something to consider as a startup, and that’s a really important point that you ask. And I wish that I had more answers, better answers at this point, but yeah, at this point those are kind of the best options in my opinion.
GARLAPOW: I’d like to reiterate something you said that struck me as particularly profound. Cutting corners could be the difference between life and death for a patient. It could be that extreme. And that’s why this is so important.
GALL: And that’s really key to like, for instance, the work that I do is in carrier screening. Which is essentially when you screen the genetics of a prospective mother and prospective father, and you see if they have any risk of when they form a baby of developing a baby with some sort of genetic abnormality. A lot of these abnormalities we look at can be treated, and the patient can live a relatively normal life. But there are some of them that are just unfortunately inconsistent with life, and it forces the parents to make a very very tough decision so that’s kind of a real world example of what I meant between the difference between life and death.
GARLAPOW: Okay. That’s a great example, really profound too. So you know that this is the Rising Tide Podcast, and a rising tide lifts all boats. It improves quality. Bryan, how can we raise the tide?
GALL: And I’m glad you asked this question because my answer to the last one was kind of a downer. So let me bring the mood up a little bit here. And really a lot of this comes to — a lot of ways that we can raise the tide together is as simple as asking questions. And that sounds really really simple, but you know as a patient don’t be afraid to ask questions of your doctor or your genetic counselor. If they don’t know what the answers are, it’s something that you might want to look into a second opinion, or look into research on your own. Try to understand what information you’re receiving from your medical professionals. You know, as a variant scientist like myself or any of you other variant scientists that are out there, really treat every sample that you see as if it were your mom or your dad or a really close friend, somebody that’s really important to you, because ultimately every sample is represents somebody who’s really important to somebody else. And that kind of goes for everybody, you know lab workers, lab directors, anybody that’s involved in the genomics industry. You know, if you’re a physician, ask questions of the test provider. You know, we’re here for a reason we – and I obviously don’t have much contact with physicians, but if you reach out to my company and ask questions of them, they’re going to know exactly who to direct you to to give you the right answer so that you can provide the best care for your patient. As scientists we’re all really really interested in talking about what we do. I mean it’s really why we invested so much time, so take advantage of that. Ask questions! And finally, you know on the business side of things, if you’re an executive or some sort of shareholder in a genetics company, really try to talk to the scientists who are on the ground kind of developing these tests, because even though you really understand the business side of things a lot better than any of us do, this is kind of a rare industry where both science and business need to work together to kind of come up with the best solution. And again like I was saying, this is something that is very important to individual patients as well. So I really think that we need as businessmen and scientists to kind of come together and collaborate more than is normal. And I do start to see that more throughout the industry. Like for example, a lot of the executive leadership team in my company actually has scientific experience and I’ve actually personally had a couple of them come to me and talk to me about the science of what we do. And I really think that’s important to have buy-in from both sides of the industry.
GARLAPOW: Absolutely. Absolutely. Dr. Bryan Gall, it’s always a pleasure to get to speak with you. Thank you so much for joining me today.
GALL: Absolutely! It’s been a pleasure. Thank you for inviting me.
GARLAPOW: Of course.
If you loved this episode of the Rising Tide Podcast, please support us on Patreon and select us on Amazon Smile. The Center for Genomic Interpretation can help your organization identify high quality clinical genetics and genomics partners or can help a laboratory identify areas for quality improvement. Through ELEVATEGENETICS, we engage with laboratories, encourage genetic and genomic test validity, and assess test efficacy. Find us at clarifygenetics.org and genomicinterpretation.org.
For our listeners following along on YouTube, remember to hit subscribe and to activate alerts for when we post new episodes. You don’t want to miss any of these important conversations with leading experts. Together we can raise the tide.
Narrated by Kathryn Mraz, MS, CGC and Dr. Megan Garlapow
Produced and Edited by Kathryn Mraz and Brynlee Buhler