#1: User Research

In order to understand how to improve we glaucoma detection via machine learning, we interviewed five ophthalmologists and optometrists -- the target user of our product.

Our objectives for the interviews were to understand:

• Current state detection processes: Each physician has her or his own personal "algorithm" for how to determine progression. We wanted to learn from these experience-based processes to inform our model development, but also identify potential points for improvement.
• Pain points: Key frustrations opthalmologists and optometrists face today when trying to determine progression of glaucoma-suspect patients.
• Concept feedback: Gathering unprompted and prompted reactions to paper prototypes of product concepts

Key learnings from the interviews included:

• Test variation is a key challenge: Visual field tests have a high degree of variance due to patient focus, technician skills and also a learning effect. All of these complicate the ability to get a clear picture of progression
• Primarily manual, paper-based process: Determining progression today is a very manual process. Clinicians typically line up print-outs from visual field machines and manually look for patterns. Electronic Medical Record (EMR) systems are starting to make in-roads but are primarily digitization of existing paper than real leaps forward in any analytics
• Clinicians want the raw data: Due to test variability and lack of gold standard metrics, clinicians grown accustomed to looking at the raw data to sanity check the results of existing algorithms. Making raw data easily available is important for securing a clinician's trust
• Advanced algorithm have limited adoption today: While clinicians are aware of progression algorithms from recent, large research studies (e.g., AGIS, CIGTS), they are not well understood and rarely used clincially as they require extensive, manual calculations. Instead, clinicians tend to stick with the methods they learned during their training
• Greyscale images of eye are important for patient education: Clincians regularly use greyscale images to help patients understand their diagnosis and progression. Patient education is important ensure treatment adherence and attendance at follow-up visits

The next steps for user research would include testing our web-based prototype with clincians and expanding our research to include patients to test for success in enabling patient education.

#2: Data Filtering

The initial dataset had 177,172 patients with 831,240 visual fields (VF) from 5 different institutions. We focused our analysis on the most commonly used methods, the SITA (see above) on 24-2 field measurement (which measures 24 degrees temporally and 30 degrees nasally and tests 54 points spaced 2 degrees apart). The stimulus size was III (code for a particular standard size) and was white on a white background. These criteria were selected for us by our opthalomologist advisor as current best clinical practice.

From the raw data, we removed tests with high level of false positive (greater than 20%) and we kept only the VFs where there were at least 5 studies conducted on the same eye. The criteria for grouping studies was [patient-id, eye (OD or OS)]. OD is the medical term for the right eye and OS for the left eye. Thus each eye’s (left or right) studies were considered separate. After this filtering process, we ended up with a final dataset of 90,713 visual fields of 13,156 eyes included in the study.

#3: Data Exploration

Glaucoma is known to be a disease that is more prevalent in older patients and our data confirms this. As the histogram below shows, the prevalence of Glaucoma increases with age with the average age of the patient being 67 years. The fall off in the prevalence of glaucoma beyond the peak reflects possibly two trends - patients reaching the end of their life span before Glaucoma occurs and saturation of Glaucoma incidence in the Glaucoma susceptible population.

The typical HFA test can be tedious for most subjects and this may result in errors (false positives or negatives) in patients. This situation can be exacerbated in older patients due to their age. The average duration of the test is more than 6 minutes, but higher rate of errors may prolong the test.

As expected, there is a correlation between the duration of the test and the age of the patients.

The results provided by the HFA include false positive and false negative errors as well as both global indices and point wise measures. Some of them are:

• Mean Deviation (MD) a global index of the deviation away from age matched controls. Patients without glaucoma have MD values close to zero. Patients having Mean Deviation more negative than -10 dB are classed as having advanced Glaucoma. As we can see from the relative picture, the majority of the patients had negative MD indicating Glaucoma vision loss.
• Raw Sensitivity, (point wise metric) values of each tested point in decibels.
• Total Deviation, (point wise metric) this is the deviation of each point in the visual field compared to control subjects for that specific age set. Patients without glaucoma have values close to zero.
• Pattern Deviation, is the total deviation after reducing deviations uniformly across the visual field. The rationale is that this accounts for global deficits in VFs caused by other reasons such as cataract, leaving the typical localized Glaucoma related deficit in place.

Spatial correlation

The picture below, on the left side, shows the structure of the retinal nerve fibre bundles that are visible after a digital enhancement of a picture taken with a blue filter. In the right side, the same structure is manually highlighted in red. The retinal nerves transmit the visual information from the retina to the brain.

Overlapping the structure of the retinal nerves with the visual fields matrix, suggests a complex spatial correlation between the points.

The following plots display the spatial correlation obtained by analyzing the correlation matrixes of the raw sensitivities, the total deviations and the pattern deviations. The 2 lines without data correspond to the position of the blind spot.

Correlation matrix of the total deviations

Correlation matrix of the pattern deviations

Visual field deficits in Glaucoma are caused by damage to the nerve fibers that conduct nerve impulses to the brain. The above described complex spatial correlations between VF points result in patterns of visual field loss that cannot be summarized by a single metric. This was the rationale for our decision to use point-wise data rather than a global index as the input to our classifiers. A global index typically loses most of the spatial information encoded in the sensitivity of VF points.

#4: Data Normalization

Different patients have different number of visual fields, but for our machine learning models, we need to provide a fixed length input. Moreover, there was a lot of variation in the temporal range of the VF recordings. For one patient the fields may be 10 years apart, for another it may be 2 years apart. So we had to normalize the time dimension in our training dataset and to do that we explored a few different solutions:

• We divided the visual fields for each patient into two groups (first half and second half) and calculated the difference in pointwise means between the two groups. We then divide the delta (in pointwise means) by the time spanned by the visual fields.
• Like the previous approach but without dividing by time
• Instead of taking all the visual fields, we took the first and the last two visual fields.

The second approach gave us the best results with our classifiers.

#5: Label Generation

In order to apply a classifier to the dataset, we had to assign a label to every single eye of the patients who met our inclusion criteria (described previously). We used two labels: (1) Stable: vision in a specific eye of the patient is stable or not decreasing at a rate that is measurable (2) Progressing: the eye of the patient is measurably getting worse. Note that it is quite possible that the left eye of a patients has severe glaucoma while the right eye is disease free (or vice versa). Hence each eye (left or right) was considered separately as a single sample.

We used 6 methods to determine the same metric - the progression of visual field loss (or lack thereof):

• AGIS Score - Advanced Glaucoma Intervention Study
• CIGTS Score - Collaborative Initial Glaucoma Treatment Study
• Mean Deviation - (provided by the HFA)
• VFI index - calculated using an open source R package
• PLR - Pointwise Linear Regression
• PoPLR - Permutation of Pointwise Linear Regression

The first 4 algorithms evaluate the severity of vision loss in a patient (how bad is it ?). The last two algorithms measure “progression” of visual loss (is it getting worse with time.) Since the quantity we are actually interested in is progression i.e. is there a measurable rate of loss - for the severity measures we compute rate of severity increase across multiple VF studies to get the progression.

None of these indices (Except MD) are readily available in the data to use. For the rest, we either implemented the algorithms ourselves (AGIS, CIGTS) or we used open source R packages (rest of the indices). The open source package we used was the R package “visualFields” developed by Ivan Marin-Franch. One important finding from generating the labels is that the 6 different approaches result in 6 different verdicts. The following table is an extract from the table we developed for all our 10,000 or so eyes.

For machine learning, we needed a singe label for each eye. So we decided to use a voting system. The winning vote (progressing or stable) became the training label for our classifier.

After the labeling we analyzed the data using a Principal Component Analysis and it’s clear from the following set of pictures that there are two distinct regions (one for stable and one for progressing) and then a big overlap between the two clusters. The first chart is created plotting the stable patients first, the second plotting the progressing first.

Chart plotting the stable patients first

Chart plotting the progressing patients first

The following chart presents the final result with the “boundary” patients i.e. patients who are neither measurably stable or measurably progressing.

From the figures it’s clear that the "boundary" patients are those in the intersection between the 2 sets. It’s very complex to study the data using unsupervised clustering. In fact both group of patients are losing their sight, but the ones that are classified as “stable” are losing it at a imprerciptible slower speed compared to the “progressing” ones. Basically it’s human perception that divides a patient into progressing or stable. In reality there exists a continuum between the two states and their overlap.

#6: Model Development

We applied different classifiers to our dataset and in particular:

• Logistic Regression
• Random Forest
• Extreme Gradient Boosting
• Support Vector Classifier
• Vanilla Neural Network
• Convolutional Neural Network

Our goal in modelling was not to come up with a perfect classifier for the proxy label we used (majority vote label). Our hypothesis instead was that the proxy label reflected a strong "true" signal corrupted by noise caused by our imperfect proxy label. We tuned our hyperparameters to deliberately underfit to the noisy label. The tuning was carried out by assuming that within each category of stable and progressing, there exists (given sufficient eyes) a normal distribution of eyes. We felt that 10,000 eyes was a sufficiently large population of eyes to give us a reasonably accurate normal distribution within each category. We then started fitting models to the training set using the simplest and least expressive model possible (for a random forest, the hyperparameter we used was maximum depth of the trees in the forest). We then increased the max-depth hyperparameter until we had a reasonably accurate normal distribution of eyes in each of the stable and progressing categories. In other words we chose the simplest model that resulted in a normal distribution of eyes in each class. By underfitting in this fashion, our hypothesis was that we would be picking out the hypothesized strong signal in the data (without being affected by the noise introduced by our proxy labels.)

In the rest of this section we discuss the challenges we faced in developing some of models, particularly convolutional neural networks.

The structure of the Neural Network is described in the following table. To avoid overfitting we added 4 dropout layers with decreasing dropout probabilities, from 0.5 to 0.2.

We trained the Neural Network for 20,000 epochs since we found that more training didn’t improve the accuracy of the classifications. For our model we used Keras with a Tensorflow backend and we noticed that the model loss on the validation set was smaller that the one one the training set. We believe we have a reasonable explanation for why this is so (the following is adapted from the Keras documentation):

• 1. During the calculation of the validation set accuracy, regularization mechanisms such as Dropout are turned off
• 2. The training loss is the average of the losses over an epoch. Typically a model is worse at the start of an epoch compared to the end of an epoch. A model whose accuracy is the average accuracy across the epoch will typically fare worse than the (validation) accuracy measured at the end of the epoch (unless there is strong overfitting.)

For the Convolutional Neural Network we had to rectify the shape of the matrix of the visual fields. We manipulated the data in the way described in the following picture, adding zero valued padding where needed:

The following table describe the structure of the classifier, that we trained for 300 epochs. More training didn't improve the capability of the network to correctly classify the patients. Also in this case we added some dropout layers to avoid overfitting. In particular the dropout probabilities were 0.2 for both layers.