HK1231361B - Retina prosthesis - Google Patents

Description

retinal prosthesis

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Application Nos. 61/308,681 (filed February 26, 2010), 61/359,188 (filed June 28, 2010), 61/378,793 (filed August 31, 2010), and 61/382,280 (filed September 13, 2010).

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with U.S. government support under Grant No. GM0779 awarded by the National Institutes of Health (NIH) and Grant No. FEY019454A awarded by the National Eye Institute. The U.S. government has certain rights in this invention.

Technical Field

The present invention relates to methods and devices for restoring or improving vision, and for treating blindness or visual impairment. In particular, the present invention relates to methods and devices for restoring or improving vision using a set of encoders and high-resolution sensors targeted to retinal cells, wherein the encoders are capable of producing normal or near-normal retinal output.

Background Art

Retinal prostheses are used in patients with retinal degenerative diseases such as age-related macular degeneration (AMD) and retinitis pigmentosa (RP), which affect two million people in the United States (Friedman et al., 2004; Chader et al., 2009) and 25 million people worldwide (Chopdar et al., 2003). Both diseases involve the degeneration of the retinal input cells: cones in AMD and rods in RP.

The goal of prostheses is to bypass degenerating tissue and stimulate surviving cells so that visual information can reach the brain again. The main targets of prostheses are retinal ganglion cells and retinal bipolar cells (Loewenstein et al., 2004; Gerding et al., 2007; Winter et al., 2007; Lagali et al., 2008; Chader et al., 2009; Zrenner et al., 2009; Thyagarajan et al., 2010).

Currently, the main approach to retinal prosthesis involves implanting an electrode array into the patient's retina near bipolar or ganglion cells (Gerding et al., 2007; Winter et al., 2007; Chader et al., 2009; Zrenner et al., 2009). The patient is then fitted with a camera/signal processing device that captures images and converts them into electrical signals; these signals are then transmitted to the electrodes, which stimulate the cells (reviewed in (Chader et al., 2009)). Although patients can see some light, the performance of these devices is still very limited: patients can, for example, see dots and edges (Nanduri et al., 2008; Chader et al., 2009), which can provide some assistance for navigation and detecting rough contours, but still does not approach normal vision. (With respect to navigation, patients are able to detect light sources such as doorways, windows, and lamps. With respect to shape detection, patients are able to distinguish objects or letters if their visual field spans ∼7 degrees (Zrenner et al., 2009); this corresponds to a visual acuity of approximately 20/1400 (legal blindness is defined as 20/200 in most places).

Improvements in electrode-based retinal prostheses have primarily focused on enhancing their resolution; the primary focus has been on reducing the size of the electrodes and increasing their density within the array (Chader et al., 2009). Current electrode diameters range from 50 to 450 micrometers (Kelly et al., 2009; Zrenner et al., 2009; Ahuja et al., 2010), which is 10 to 100 times the size of a retinal cell. Despite this increase in resolution, current technology still cannot match the resolution of a normal retina because stimulating individual cells with electrodes is currently impractical and presents significant technical challenges: finer electrodes require more current, which can lead to tissue burns (see, for example, the title and program of a recent retinal prosthesis conference: "The Eye and the Chip 2010: 2010 Special Emphasis on Retinal Stimulation Safety for Neuro-Prosthetic Devices").

Optogenetics has been employed as an alternative to stimulating cells with electrodes. Optogenetic approaches involve expressing a protein, such as channelrhodopsin-2 (ChR2) or one of its derivatives, in ganglion cells or bipolar cells. ChR2 is light-sensitive; upon activation by light, cells expressing ChR2 experience a voltage change, which allows the cells to transmit electrical signals (Bi et al., 2006; Lagali et al., 2008; Zhang et al., 2009; Tomita et al., 2010). This approach has the potential to provide higher resolution, meaning that, in theory, cells can be stimulated individually. Although animal studies have demonstrated that high resolution is possible, several recent publications in this field have yet to report achieving near-normal or even partially normal vision (Bi et al., 2006; Lagali et al., 2008; Zhang et al., 2009; Thyagarajan et al., 2010; Tomita et al., 2010).

Current cutting-edge approaches focus less on driving the stimulus (electrodes or channelrhodopsin) in a manner that most closely resembles the endogenous signaling from the retina to the brain. Endogenous retinal signaling is complex. When a normal retina receives an image, it performs a series of operations on it—extracting information from the image and converting it into a code that the brain can read.

Current electrode-based devices employ much simpler signal processing than the retina; for example, they simply convert the light intensity at each point in the image into a pulse rate in a linear manner (Loewenstein et al., 2004; Fried et al., 2006; Kibbel et al., 2009; Ahuja et al., 2010). Because of this, the retinal output produced by these devices differs significantly from normal retinal output; the brain expects one signal to be coded, but actually receives another.

Current optogenetic approaches are similarly limited. Improvements have primarily focused on exploiting the properties of channelrhodopsin (e.g., enhancing its light sensitivity and altering its dynamics), while little effort has been devoted to mimicking endogenous retinal signal processing (Bi et al., 2006; Lagali et al., 2008; Zhang et al., 2009; Thyagarajan et al., 2010; Tomita et al., 2010).

Therefore, there is a need to develop a retinal prosthesis that can convert visual input into normal retinal output that is easily understood by the brain. Such a prosthesis also needs to provide high-resolution signals, ideally with the ability to target individual retinal cells, such as retinal ganglion cells. The present invention relates to such a prosthesis, which combines an encoding step that produces normal or near-normal retinal output with a high-resolution sensor, to provide normal or near-normal vision to blind individuals.

SUMMARY OF THE INVENTION

In one aspect, a method is disclosed, comprising: receiving raw image data corresponding to a series of raw images; and processing the raw image data using an encoder to generate encoded data, wherein the encoder is characterized by input/output transformations that substantially simulate input/output transformations of one or more retinal cells of a mammalian retina.

In certain embodiments, the encoder is characterized by input/output conversion that substantially simulates the input/output conversion of one or more retinal cells of a mammalian retina, and the input range includes natural scene images, including spatiotemporally transformed images.

In some embodiments, processing the raw image data using an encoder to generate encoded data includes processing the raw image data to generate a plurality of values, X, converting the plurality of X values into a plurality of responses, λm, representing respective responses of retinal cells m in the retina, and generating the encoded data based on the responses. In some embodiments, the responses correspond to a firing rate of a retinal cell. In some embodiments, the responses correspond to a function of the firing rate of the retinal cell. In some embodiments, the responses correspond to output pulses of a retinal cell. In some embodiments, the responses correspond to retinal cell generated potentials, i.e., outputs of image convolution with a spatiotemporal filter.

In some embodiments, processing the raw image data using an encoder to generate encoded data includes: receiving images from the raw image data, and for each image, rescaling brightness and contrast to generate a rescaled image stream; receiving a set of N rescaled images from the rescaled image stream, and performing a spatiotemporal transformation on the N image sets to generate a set of retinal responses, each value in the set corresponding to each of the retinal cells; and generating the encoded data based on the retinal responses.

In some embodiments, the response comprises retinal cell firing rate. In some embodiments, N is at least 5, at least about 20, at least about 100 or more, such as in the range of 1-1000 or any range therein.

In certain embodiments, performing the spatiotemporal transformation comprises: convolving the N rescaled images with a spatiotemporal kernel to generate one or more N space-time transformed images; and applying a nonlinear function to the space-time transformed images to generate the response set.

In certain embodiments, performing the spatiotemporal transformation comprises: convolving the N rescaled images with a spatial kernel to generate N spatially transformed images; convolving the N spatially transformed images with a temporal kernel to generate a spatially transformed output; and applying a nonlinear function to the temporally transformed output to generate the response set.

In certain embodiments, the encoder is characterized by a set of parameters, and wherein the parameter values are determined using response data obtained from an experiment in which a mammalian retina is exposed to white noise and natural environmental stimuli.

In some embodiments, the encoder is configured to generate a Pearson correlation coefficient between a test input stimulus and a corresponding reconstructed stimulus reconstructed from encoded data generated by the encoder in response to the test input stimulus of at least about 0.35, 0.65, at least about 0.95, or more, such as in the range of 0.35-1.0, or any range therein. In some embodiments, the test input stimulus comprises a series of natural scenes.

In another embodiment, an apparatus is disclosed, comprising: at least one memory storage device configured to store raw image data; and at least one processor operably connected to the memory and programmed to perform one or more methods described herein.

In certain embodiments, a non-transitory computer-readable medium having computer-executable instructions for performing the steps of one or more methods described herein.

The present invention provides methods and systems for restoring or improving vision. Vision is restored or improved by a method comprising: receiving a stimulus, converting the stimulus into a set of codes via a set of encoders, converting the codes into signals via an interface, and then activating a plurality of retinal cells via a high-resolution sensor driven by the signals from the interface. The activation of the plurality of retinal cells results in retinal ganglion cells responding to a wider range of stimuli that are substantially similar to the responses of retinal ganglion cells from a normal retina to the same stimuli.

The method for restoring or improving vision may have the following characteristic performance: (i) the score correct on a forced-choice visual discrimination task performed with the code is at least about 95%, 65% or 35% of the score correct on a forced-choice visual discrimination task performed with retinal ganglion cell responses from a normal retina; or (ii) the Pearson correlation coefficient between a test stimulus and a stimulus reconstructed from the code in the presence of the test stimulus is at least about 0.95, 0.65 or 0.35.

Alternatively, the method for restoring or improving vision may have the following characteristic performance: (i) the fraction correct on a forced-choice visual discrimination task performed with retinal ganglion cell responses from an activated retina is at least about 95%, 65%, or 35% of the fraction correct on a forced-choice visual discrimination task performed with retinal ganglion cell responses from a normal retina, or (ii) a Pearson correlation coefficient of at least about 0.95, 0.65, or 0.35 between a test stimulus and a stimulus reconstructed from retinal ganglion cell responses from an activated retina in the presence of the test stimulus.

The encoding step may include the following steps: (i) preprocessing the stimulus into a plurality of values, namely X; (ii) converting the plurality of X values into a plurality of firing frequencies, namely λ _m , where m is a retinal ganglion cell in the retina; and (iii) generating a code, the code representing a spike potential from the firing frequencies. The encoding step may include the step of modifying the code by a burst elimination step. The code may be non-transitory stored during the burst elimination step. The burst elimination step may include the following steps: (i) setting the duration of a segment to be tested and a standard number of pulses for the duration segment; (ii) counting the number of pulses in the segment; and (iii) if the number of pulses exceeds the standard number, replacing the segment with a segment having an approximately maximum inter-pulse time.

The encoder may have parameters whose values are determined by response data obtained from the retina during exposure to white noise and natural scene stimuli.

The code can be converted into an output via an interface, wherein the output is a plurality of visible light pulses. The sensor can be a visible light responsive element, such as a protein. The protein can be channelrhodopsin-1, channelrhodopsin-2, a light-operated ionotropic glutamate receptor (LiGluR), ChETA, SFO (step function opsin), OptoXR (photosensitive G protein-coupled receptor), channelrhodopsin-1, channelrhodopsin-2, ChIEF, NpHr, eNpHR, or any combination thereof.

A viral vector can be used to introduce the gene encoding the protein into the cell. The viral vector can be a recombinant adeno-associated virus. The gene can be selectively expressed in at least one retinal ganglion cell. In one embodiment, a dual-vector cre-lox system can be used to selectively express the gene, wherein the dual-vector expression type overlaps only in the selected cell type. In this embodiment, the dual vector is: (a) a first vector comprising an inverted gene expressing a light-sensitive protein, wherein the inverted gene is flanked by loxP sites in reverse, and the inverted gene is regulated by a second gene promoter, and the second gene is expressed at least in the selected cell type; and (b) a second vector comprising a Cre recombinase, wherein the recombinase is regulated by a third gene promoter, and the third gene is expressed at least in the selected cell type and other non-overlapping cell classifications.

The apparatus for implementing the method of restoring or improving vision can be used to treat subjects with retinal degenerative diseases, such as macular degeneration or retinitis pigmentosa. The treated subjects can achieve at least about 95%, 65%, or 35% of normal visual acuity as measured by EVA or ETDRS protocols. Alternatively, two or more factors are altered in the treated subjects as measured by pattern visual evoked potentials (VEPs) or sweep visual evoked potentials (Sweep VEPs).

The present invention also provides a method for activating multiple retinal cells, comprising receiving a stimulus, converting the stimulus into a set of codes via a set of encoders, converting the codes into signals via an interface, and then activating the multiple retinal cells via a high-resolution sensor, wherein the high-resolution sensor is driven by the signals from the interface. Activation of the multiple retinal cells results in retinal ganglion cells responding to a wide range of stimuli that are substantially similar to the responses of retinal ganglion cells from a normal retina to the same stimuli.

Alternatively, the method of activating multiple retinal cells may have the following characteristic performance: (i) the fraction correct on a forced-choice visual discrimination task performed with the code is at least about 95%, 65% or 35% of the fraction correct on a forced-choice visual discrimination task performed with retinal ganglion cell responses from a normal retina, or wherein the Pearson correlation coefficient between a test stimulus and a stimulus reconstructed from the code in the presence of the test stimulus is at least about 0.95, 0.65 or 0.35.

The methods and systems of the present invention also provide an apparatus for restoring or improving vision in a subject in need thereof, wherein the apparatus comprises: (i) a device for receiving a stimulus; (ii) a processing device comprising: (a) a non-transitory computer-readable medium storing a set of encoders for generating the stimulus into a set of codes, (b) at least one processor, and (c) a non-transitory computer-readable medium storing the codes; (iii) an interface for converting the codes into an output; and (iv) a high-resolution sensor for activating a plurality of retinal cells. The apparatus for restoring or improving vision may be characterized by activation of the plurality of retinal cells such that the retinal ganglion cells respond to a wide range of stimuli that are substantially similar to the responses of retinal ganglion cells from a normal retina to the same stimuli. Alternatively, the apparatus may be characterized by: (i) a score correct on a forced-choice visual discrimination task performed with the codes being at least about 95%, 65%, or 35% of the score correct on a forced-choice visual discrimination task performed with the responses of retinal ganglion cells from a normal retina; or (ii) a Pearson correlation coefficient between a test stimulus and a stimulus reconstructed from the code in the presence of the test stimulus being at least about 0.95, 0.65, or 0.35. After treatment with the device for restoring or improving vision, the subject is able to achieve at least about 35% of normal visual acuity as measured by the EVS or ETDRS protocol. Alternatively, two or more factors are altered in the treated subject as measured by pattern VEP or scanning VEP. The device for restoring or improving vision can be used to treat a subject suffering from a retinal degenerative disease, such as macular degeneration or retinitis pigmentosa.

The methods and systems of the present invention also provide a non-transitory computer-readable medium having computer-executable instructions. The computer-executable instructions are a set of instructions for converting at least one stimulus into a non-transitory code, wherein the code is capable of activating a plurality of retinal cells via a high-resolution sensor. The system is characterized in that, when tested, the fraction correct on a forced-choice visual discrimination task performed with the code is at least about 35% of the fraction correct on a forced-choice visual discrimination task performed with retinal ganglion cell responses from a normal retina, or the Pearson correlation coefficient between the test stimulus and the stimulus reconstructed from the code in the presence of the test stimulus is at least about 0.35. The set of instructions has parameters, and the values of the parameters can be determined from response data obtained from the retina when the retina is exposed to white noise and natural scene stimuli.

The methods and systems of the present invention also provide a non-transitory computer-readable medium having computer-executable instructions having a signal corresponding to a stimulus, the computer-executable instructions being operable to control at least one sensor that activates at least one cell in the damaged retina to produce a response that is substantially similar to the response of a corresponding ganglion cell in a normal retina to the stimulus. The signal can be a set of codes, wherein when tested for performance, a fraction correct on a forced-choice visual discrimination task performed with the code is at least about 35% of the fraction correct on a forced-choice visual discrimination task performed with responses of retinal ganglion cells from a normal retina, or a Pearson correlation coefficient between a test stimulus and a stimulus reconstructed from the code in the presence of the test stimulus is at least about 0.35.

The methods and systems of the present invention also provide a method for generating representative stimuli for the retina using an encoder. The method includes the following steps: (i) preprocessing the stimulus into a plurality of values, namely, X; (ii) converting the plurality of X values into a plurality of firing rates, namely, λ _m ; and (iii) converting the firing rates, λ _m , into a code. In this case, the performance of the method can be tested as follows: (i) the fraction of correct performance in a discrimination task performed using the encoder output is within 35% of the fraction of correct performance in a discrimination task performed using a normal retina; (ii) the Pearson correlation coefficient between the reconstructed stimulus reconstructed from the encoder output and the original stimulus is at least about 0.35, or the performance of the encoder output in error type detection is at most about 0.04. The conversion step may include spatiotemporally converting the plurality of X values into a plurality of firing rates λ _m , where λ _m (m represents each retinal ganglion cell in the retina) is a function of L _m , _which is a linear filter corresponding to the spatiotemporal kernel from the mth retinal ganglion cell, and N _m is a function describing the nonlinearity of the mth retinal ganglion cell. There may be multiple encoders e _m , where e _m is the encoder for the mth ganglion cell. The code may have a discrete plurality of bits forming a bit stream. Alternatively, the code may be a continuous wave.

The stimulus can be electromagnetic radiation. For example, the electromagnetic radiation can be visible light. The code can be converted into an output, which can be a plurality of visible light pulses, via an interface. Activation of a plurality of cells in the retina using the plurality of visible light pulses can generate at least one initial spike train, wherein at least a portion of the cells in the retina have at least one sensor that is a visible light-responsive element.

The method of generating representative stimulation using an encoder for the retina can further include activating a plurality of cells in the retina, the activation driven by a plurality of visible light pulses to generate at least one initial spike train, wherein at least a portion of the cells in the retina have at least one sensor comprising at least one visible light responsive element. The cells can be retinal ganglion cells. The visible light responsive element can be a synthetic photosensitive isomerized azobenzene-regulated K+ (SPARK), a depolarizing SPARK (D-SPARK), or any combination thereof. The visible light responsive element can be a protein such as channelrhodopsin-1, channelrhodopsin-2, LiGluR, ChETA, step-function opsin (SFO), photosensitive G protein-coupled receptor (OptoXR), channelrhodopsin-1, channelrhodopsin-2, ChIEF, NpHr, eNpHR, or any combination thereof. The proteins, genes encoding the proteins, and viral vectors are as described above. The stimulation can vary in a spatiotemporal manner or can be static. The present invention also provides a method for setting a set of parameters of an encoder, comprising the following steps: (a) recording electrical signal data, and storing the data. The data comprises the number of action potentials of retinal ganglion cells from the retina during exposure to white noise and natural scene stimulation; (b) calculating the negative correlation between the number of action potentials of the ganglion cells and the calculated stimulus intensity to determine an initial set of linear filter _Lm values; (c) processing _Lm as a product of a spatial function and a temporal function, wherein the spatial function is parameterized as a weight grid, the temporal function is parameterized as a sum of weighted time-based functions, and Nm is converted to a linear filter. _m is assumed to be an exponential function to ensure that there are no local maxima; (d) the similarity of the parameter set for a given stimulus is calculated and the ganglion cell responses are recorded; (e) the initial optimal parameters of the spatial function, the temporal function and the exponential nonlinearity are determined by maximizing the similarity of the parameters; (f) the exponential nonlinearity is replaced by a cubic spline; (g) the parameters of the spline are optimized to achieve maximum similarity; (h) the parameters of the spatial and temporal functions are optimized to achieve maximum similarity while keeping the result of step (g) constant; (i) step (g) is repeated while keeping the result of step (h) constant and step (h) is repeated; and (j) step (i) is repeated until the change in similarity between the two steps is less than an arbitrarily selected decimal. The aforementioned method can be implemented in a non-transitory computer-readable medium having computer-executable instructions to determine the values of multiple parameters for converting at least one first stimulus into a non-transitory code, and to determine parameters for a linear filter L _m , a spatial function, and a temporal function, wherein the parameters are determined by the following steps, which include: (a) recording electrical signal data, and storing the data, wherein the data includes the number of action potentials of retinal ganglion cells from the retina during exposure to white noise and natural scene stimuli; (b) calculating the negative correlation between the number of action potentials of the retinal ganglion cells and the intensity of each stimulus to determine an initial set of linear filter L _m values; (c) establishing a parameter set for the spatial function; (d) establishing a parameter set for the temporal function; (e) calculating the similarity of the parameter sets of the spatial function and the temporal function for a given stimulus, and recording the responses from the retinal ganglion cells; and (f) determining the optimal parameter set for the spatial function, the temporal function, and the nonlinearity by maximizing the parameter similarity.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of one embodiment of a prosthetic approach. On the far left is the stimulus, followed by the image—the captured stimulus. The captured stimulus is then processed by a set of encoders, which in turn drive the interface device. The interface device then triggers light pulses in retinal ganglion cells transfected with the light-sensitive protein channelrhodopsin-2 (ChR2). The resulting spike patterns are similar to those produced by a healthy retina.

Figure 2 is a schematic diagram of one embodiment of the device. On the outside of each lens area of the glasses are cameras; signals from the cameras are fed to a processing device, which in this embodiment is located on the arms of the glasses. The processing device controls a light array located on the inside of each lens area.

Figure 3 shows that the information content of the encoder (model cell) closely matches that of the corresponding real cell. For this analysis, we used three stimulus sets: a drifting grating that varied in temporal frequency, a drifting grating that varied in spatial frequency, and a natural scene. For each cell, we calculated the mutual information between the model cell's response and the stimulus and plotted it against the mutual information between the real cell's response and the stimulus (for the three stimulus sets, n = 106, 118, and 103, respectively; each stimulus had 5 bits of entropy; and bin sizes ranged from 250 to 31 ms).

Figure 4 (Figures 4-1, 4-2, and 4-3) shows that the distribution of encoders (model cells) after stimulation approximately matches the distribution of the corresponding real cells. A. For each cell, we plotted a matrix pair. The matrix on the left is the response of the model cell after stimulation (the average of all responses); the matrix on the right is the response of the corresponding real cell after stimulation. The histogram next to the matrix pair is a measure of the distance between them. In short, for each row, we calculated the mean squared error (MSE) between the model and the real cell after stimulation and normalized it. The normalization is to divide the model by the MSE between the real cell and the random shuffle after stimulation. A value of 0 means that the two rows are identical. A value of 1 means that the difference between them is the difference between two randomly shuffled rows. (Due to limited data, some cells have values higher than 1 by chance). The vertical light gray line represents the median of the histogram. B. Histogram of the median values of all cells in the data set, and histogram of the K-L dispersion of all cells in the data set (for stimulation n of 106, 118, and 103, respectively).

Figure 5 shows that the encoders (model cells) make the same predictions as real cells. Top left, the model shows that under scotopic conditions, ON cells are better at discriminating low temporal frequencies than OFF cells, while OFF cells are better at discriminating high temporal frequencies than ON cells. Bottom left, the results for the real cells are the same. Top, scotopic and photopic conditions are examined, and the model shows that these differences in behavior only exist under scotopic conditions: under photopic conditions, both types of cells perform almost equally well. Bottom, scotopic and photopic conditions are examined, and the results for the real cells are the same. Top, repeated examination of both conditions, the model shows that under scotopic conditions, both ON and OFF cells perform well only over a narrow range of frequencies, while under photopic conditions, they perform well over a wide range. Bottom, repeated examination of both conditions, the predictions also hold for the real cells. Predictions were made using increasing numbers of cells until the prompt performance reached saturation. Error bars are SEM.

Figure 6 shows encoders (model cells) predicting shifts in visual motion. Left: The model predicts a shift toward higher temporal frequencies when animals move from scotopic to photopic conditions. Right: Based on the predictions, the animals' behavioral performance shifted toward higher temporal frequencies (n = 5 animals). This prediction was stable from 1 cell to saturation (20 cells).

Figure 7 shows that the responses produced by ganglion cells in the retinal prosthesis closely matched those produced by normal retina, while the responses produced by ganglion cells using the standard optogenetic method (i.e., using ChR2 as a sensor) did not match those produced by normal retinal cells. Images of natural scenes were presented to the retinas of three groups of mice: normal mice, retinas of blind mice treated with retinal prostheses (i.e., blind retinas expressing ChR2 in ganglion cells and stimulated with encoder-processed images), and retinas of blind mice treated with the standard optogenetic method (i.e., blind retinas expressing ChR2 in ganglion cells but stimulated with images not processed by encoders). Spike trains were then recorded from the ganglion cells in each group.

Figure 8 shows that the performance of the retinal prosthesis on a visual discrimination task closely matches that of the normal retina, while that of the standard optogenetic method does not match. A. The confusion matrix generated when the test set was obtained from a normal WT retina. The matrix for a single ganglion cell is on the left, and the matrix for a cell group (20 cells) is on the right. The correct score for the group is 80%. B. The confusion matrix generated when the test set was obtained from the encoder (note that the encoder was constructed from the input/output relationship of the WT retina used in Figure A), with a correct score of 79%. C. The confusion matrix generated when the test set was generated from a blind retina, where an encoder + sensor (ChR2) was used to drive the ganglion cells. The correct score was 64%. D. The confusion matrix generated when the test set was generated from a blind retina, where the ganglion cells were driven using a standard optogenetic method (i.e., only ChR2, no encoder), with a correct score of 7%.

Figure 9 shows that the reconstructed image from the retinal prosthesis response closely matches the original image, while the reconstructed image from the standard optogenetic method response does not match the original image. Although the reconstructed image is not necessary for the brain, reconstruction is still a convenient method to compare methods and give an approximate level of vision restoration that each method may provide. A. Original image. B. Reconstructed image from the encoder response. C. Reconstructed image from the encoder + sensor (ChR2) response. D. Reconstructed image from the standard optogenetic method response (ChR2 only, as described above). Note that Figure B is a critical figure because it shows the output of the encoder, which can be combined with different types of sensors. Reconstruction was performed in 10X 10 or 7X 7 square blocks on the processing cluster of the present invention. As described herein, maximum similarity was used, that is, for each block, we found the array of gray values that maximized the probability of the observed response (high-dimensional search based on Paninski et al. 2007).

Figure 10 shows tracking with a retinal prosthesis. A. Baseline drift (no stimulus present). As described herein, the eye position of blind animals exhibits drift similar to that observed in blind humans. B. Response to a drifting grating presented using standard optogenetics (i.e., presented in its actual form on the screen). C. Response to a drifting grating presented using a retinal prosthesis (i.e., presented in its encoded form on the screen). When the image is converted into the code used by ganglion cells, the animal can track it. Top row, raw eye position trajectories, representative examples. Middle row, smoothed components (saccades and removal of artifactual movement, see raw trajectories above). Bottom row, average trajectories across all trials (n = 15, 14, and 15 trials in total).

Figure 11 shows a schematic diagram of the device. A camera (top) captures the stimulus from the field of view. The signal from the camera is fed into a processing device, i.e., an encoder. The encoder performs a series of steps, represented in the figure in module form: preprocessing, spatiotemporal conversion, and spike generation. The output of the spike generation step is stored non-temporarily in preparation for conversion into a format suitable for the sensor, which includes a pulse elimination step. The output is then converted into a format suitable for the sensor in the interface, which then passes the converted signal to the sensor. The arrows represent the flow of signals from a specific area of the field of view, which passes through the encoder module and the interface device to the sensor in the retinal cells. Overlapping loops indicate that the encoder carries information from overlapping areas of the field of view, which represent the image in a manner similar to that of a normal retina.

Figure 12 illustrates an exemplary encoder converting an image into light pulses. A shows an exemplary image. B shows a preprocessed image and indicates the position of an exemplary encoder that produces the outputs shown in C-E. C shows the output of the spatiotemporal conversion step. D shows the output of the spike generation step. E shows the light pulse corresponding to the output of the spike generation step.

Figure 13 shows that the responses of monkey retina to natural images generated by the encoder closely match those of normal monkey retina. Images of natural scenes were presented to normal monkey retina and virtual retina. The top row shows the spike trains from a normal monkey ganglion cell; the bottom row shows the results for the corresponding model cell (i.e., its encoder).

FIG14 shows that the performance of the monkey encoder on a visual discrimination task (the same task as in FIG8 ) closely matches that of normal monkey ganglion cells. A. Confusion matrix generated when the test set was obtained from a normal monkey retina. The matrix for a single ganglion cell is on the left, and the matrix for a population of 10 cells is on the right. The fraction correct for the population was 83%. B. Confusion matrix generated when the test set was obtained from the encoder derived from monkey ganglion cells. The fraction correct was 77%. All analyses were performed as in Example 8, FIG8 . Thus, the fraction correct using the encoder responses was 92.8% of the fraction correct using the normal monkey ganglion cell responses.

Figure 15 shows the ganglion cell responses generated by the encoder + sensor, which are then encoded with high fidelity output. The encoder output is converted into a train of light pulses, which are presented in the retina of a double transgenic mouse that is blind and expresses ChR2 in its ganglion cells. A. Light pulses and corresponding ganglion cell outputs. For each pair in each row, the top row shows the number of light pulses and the bottom row shows the number of action potentials generated by ganglion cells expressing ChR2. Each dot represents an occurrence of a light pulse or a ganglion cell action potential. B. Magnified view of the circled area in (A), which shows a one-to-one correspondence between light pulses and action potentials. Because the action potential follows the light pulse, the encoder has high fidelity.

Detailed Description of the Invention

The present invention provides a method and device for restoring or improving vision, increasing visual acuity, or treating blindness or visual impairment, or activating retinal cells. The method includes capturing stimuli, encoding stimuli, converting the codes into sensor instructions at an interface, and transmitting the instructions to retinal cells. The device includes a method for capturing stimuli, a processing device that executes a set of codes, an interface, and a set of sensors, wherein each sensor targets a single cell or a small number of cells; the set of sensors refers to high-resolution sensors. In one embodiment, each encoder performs a preprocessing step, a spatiotemporal conversion step, and an output generation step. The present method can be used for retinal prostheses that can produce representative responses to a wide range of stimuli, including artificial and natural stimuli.

The present invention and apparatus can process any type of stimulus. For example, the stimulus can include visible light, but can also include other types of electromagnetic radiation, such as infrared, ultraviolet, or other wavelengths within the electromagnetic spectrum. The stimulus can be a single image or multiple images; furthermore, the image can be static or can change in a spatiotemporal manner. Simple shapes such as charts, or relatively complex stimuli such as natural scenes can be used. Further, the image can be a grayscale image or a color image, or a combination of grayscale and color images. In one embodiment, the stimulus can include white noise ("WN") and/or natural stimuli ("NS") such as images of natural scenes, or a combination of the two.

The stimulus is transformed or converted into an alternative to the output of the normal retina, i.e., an output form that the brain can easily understand and use to represent images. The time scale on which the transformation occurs is roughly the same as the time scale performed by a normal or near-normal retina, i.e., the initial response of the retinal ganglion cells to the stimulus occurs within a time interval of about 5-300 ms. The methods and apparatus of the present invention can help patients or diseased mammals to restore near-normal or normal vision, or can improve vision, including grayscale vision or color vision, wherein the patients or diseased mammals have any type of retinal degenerative disease and their retinal ganglion cells (which may also be referred to herein as "ganglion cells") are still intact. Non-limiting examples of retinal degenerative diseases include retinitis pigmentosa, age-related macular degeneration, Usher's syndrome, Stargardt's macular degeneration, Leber's congenital amaurosis and Barbette-Biedl syndrome, retinal detachment, and retinal vascular occlusion.

Diseases with complications of retinal degeneration include: snowflake retinal degeneration; choroidal neovascularization with adult-onset foveal dystrophy; crystalline retinitis pigmentosa; and diabetic retinopathy. Some diseases with retinal degeneration as a symptom include: aceruloplasminemia; adrenoleukodystrophy; Alstrom disease; Alstrom syndrome; asphyxiating thoracic dysplasia; Bonneman-Meinecke-Reich syndrome. syndromw); Bonnemann-Meinecke-Reich syndrome; CDG type 1A syndrome; dominant chorioretinopathy-microcephaly; choriohypopituitarism; congenital disorder of glycosylation type 1A; congenital disorder of glycosylation type Ia; cystinosis; oligotrichosis, syndactyly, and retinal degeneration; Jeune syndrome; mucolipidosis IV; mucolipidosis type 4; mucopolysaccharidosis; muscle-oculo-cerebral syndrome; neonatal ALD; olivopontocerebellar atrophy type 3; osteopetrosis, autosomal recessive 4; retinopathy pigmentosa; pseudoadrenoleukotrophy; retinoschisis, X-linked; retinoschisis 1, X-linked, juvenile; Santavuli disease; spastic paraplegia 15, autosomal recessive; and Werner syndrome.

The methods and devices of the present invention can be used to treat any mammalian subject in which a portion of the retinal ganglion cells, a portion of the optic nerve originating from the retinal ganglion cells, and a portion of other functional central visual system processing functions remain intact. The range of retinal ganglion cell loss that can be treated by the methods and devices of the present invention can include only a portion of all retinal ganglion cells, or can include all retinal ganglion cells present in the retina.

Retinal prostheses, like normal retinas, are image processors—they extract basic information from the stimulus they receive and reformat it into a type of action potential that the brain can understand. The type of action potential produced by a normal retina is referred to as the retinal code or ganglion cell code. Retinal prostheses convert visual stimuli into the same code or a similar alternative code, so that a damaged or degenerated retina can produce normal or near-normal outputs. Because the retinal prosthesis uses the same or similar code as a normal retina, the firing pattern of ganglion cells in the damaged or degenerated retina, i.e., the type of action potential, is the same or substantially similar to that produced by normal ganglion cells. The visual recognition abilities of subjects treated with the present device will closely match those of normal or near-normal subjects.

The methods and devices of the present invention can reproduce normal or near-normal ganglion cell output in response to a wide range of stimuli, including both artificial and natural stimulation, as tested using a variety of different criteria described below. In this retinal prosthesis approach, the methods of the present invention utilize encoding steps, interface steps, and sensing steps. The methods and devices of the present invention can drive activation of different retinal cell types, including, but not limited to, retinal ganglion cells and retinal bipolar cells.

In one embodiment, the prosthesis targets retinal ganglion cells. In this embodiment, the encoding step converts visual stimuli into a code used by the ganglion cells, or a code very similar to it, and the sensors, through an interface, drive the ganglion cells to fire according to the instructions of the code. The result is a normal or near-normal output from the damaged or degenerated retina, i.e., a normal or near-normal firing pattern. In another embodiment, the prosthesis targets retinal bipolar cells (i.e., the sensors are targeted to retinal bipolar cells, also referred to herein as "bipolar cells"). In this case, the encoding step occurs at an earlier stage, converting the visual stimulus into a code that drives the bipolar cells, which in turn drives the ganglion cells to produce a normal output. Other codes are also possible. In both examples, the prosthesis comprises a set of encoders and a set of sensors that interact with each other: the encoders drive the sensors. As described below, the encoders drive the sensors through an interface. The result is a method that enables retinal output cells to produce normal or near-normal firing patterns and transmit normal or near-normal visual signals to the brain.

Because different retinal cell types exist, different encoders can be used. The differences can correspond to specific cell types or cell locations within the retina. When more than one encoder is present in a retinal prosthesis, the encoders can be controlled in parallel, either independently or through at least one or more couplers.

As described above, in one embodiment, the retinal prosthesis targets retinal ganglion cells. In the embodiment, the retinal ganglion cells of a subject (e.g., a blind patient) are first modified by gene therapy to express a sensor, such as a light-sensitive protein (e.g., ChR2). The subject then wears glasses that carry a camera, a processing device that executes a set of encoders (one or more), and an interface for generating light pulses. The camera captures the image (stimulus) and transmits it through the encoder set. The encoder performs a series of operations on the stimulus and converts it into an encoded output, a type of electrical pulse (also called a stream), which is equivalent to the type of action potential (or stream) generated by a normal ganglion cell in response to the same stimulus. The stream of electrical pulses is then converted into a stream of light pulses to drive cells expressing ChR2 on the subject's retina. The schematic diagram of Figure 1 shows the following steps: the stimulus (image) is converted into a stream of electrical pulses, which is then converted into a stream of light pulses, which then drives the sensors in the retinal cells. Figure 2 is an embodiment of a device provided to the patient (an external device that interacts with a sensor operating in the body).

Alternatively, instead of receiving gene therapy, the patient can implant electrodes in the patient's retina at a location closest to the ganglion cells or bipolar cells to provide them with the sensor, namely ChR2. In this case, the patient then wears glasses that carry a camera and a processing device that executes a set of encoders. The electrical pulses or bit streams are stored in memory and converted into signals that instruct the electrodes to fire electrical pulses, which ultimately drive the ganglion cells to fire.

The methods and devices herein can be used with mammals, such as humans. Mammals include, but are not limited to, rodents (e.g., guinea pigs, hamsters, rats, mice), primates, marsupials (e.g., kangaroos, wombats), monotremes (e.g., platypus), murines (e.g., mice), lagomorphs (e.g., rabbits), canines (e.g., dogs), felines (e.g., cats), equines (e.g., horses), porcines (e.g., pigs), ovines (e.g., sheep), bovines (e.g., cows), apes (e.g., monkeys or apes), monkeys (e.g., marmosets, baboons), and great apes (e.g., gorillas, chimpanzees, orangutans, gibbons).

The method and apparatus of the present invention may also be used with robots or other types of mechanical devices in applications requiring visual information or optical image processing.

The algorithms and/or parameters of the encoders can vary from patient to patient and can be adjusted over time based on age or disease progression. Furthermore, as described herein, a single patient can be equipped with multiple encoders in a single prosthesis, where the encoders can be altered based on their spatial location on the retina or other factors, such as cell type. The present invention allows for convenient and safe algorithm changes within a patient's body. Adjustments to the algorithm can be made by one of ordinary skill in the art.

A description of the encoder (or encoding step) and the sensor (or sensing step) follows.

encoder

The encoder is a model of the input/output of retinal cells (e.g., ganglion cells or bipolar cells). It provides a stimulus/response relationship. The encoder operates according to an algorithm; as described herein, the algorithm can be executed by a processing device with dedicated circuitry and/or using computer-readable media.

In one embodiment, the encoders are input/output models for ganglion cells. These encoders include algorithms that convert stimuli into electrical signal types that are identical or substantially similar to those produced by normal ganglion cells in response to the same stimuli. A retinal prosthesis can utilize multiple encoders, which can be assembled in parallel, for example, as shown in FIG11 , where different segments of stimulation (in other words, different regions of the visual field) pass through separate encoders, which in turn control different specific sensors. In this embodiment, each encoder can have parameters tailored to its target sensor, which can be, for example, based on the location and/or type of retinal cells, cells simulated by the encoders, or cells driven by the encoder outputs. The term "code" can refer to the type of electrical pulses that correspond to the type of action potentials (also known as spike trains) generated by the retina in response to stimulation. The term "code" can refer to a bit stream corresponding to the type of spike train. Each bit can correspond to the activity of a neuron (e.g., 1 means the neuron fired; 0 means the neuron did not fire). The code can also be a continuous wave. The present invention can encompass any type of waveform, including non-periodic waveforms and periodic waveforms, including but not limited to sine waves, square waves, triangle waves, or sawtooth waves.

The following flow chart generally summarizes one embodiment of the operation of the encoder.

Preprocessing steps

This is a rescaling step that can be performed in a preprocessor module of a processing device, which maps the real-world image, I, into a quantity, X, that is within the operating range of the spatiotemporal transformation. Note that I and X are both time-varying, i.e., I(j, t) represents the intensity of the real image at each location j and time t, and X(j, t) represents the corresponding output of the preprocessing step. The preprocessing step can perform the following mapping: I(j, t) is mapped to X(j, t) by X(j, t) = a + bI(j, t), where a and b are constants selected to map the range of real-world image intensities to the operating range of the spatiotemporal transformation.

Rescaling can also be performed using variable history to determine the quantities a and b, and user-controlled switches can be used to set the values of these quantities under different conditions (e.g., different lighting or different constants).

For a grayscale image, for each position j and time t, I(j, t) and X(j, t) each have only one value.

For color images, the same strategy is used, but it applies separately to the red, green, and blue color channels. In one embodiment, for each position j and time t, the intensity I(j, t) has three values (I ₁ , I ₂ , I ₃ ), where these three values I ₁ , I ₂ , I ₃ represent the intensities of red, green, and blue, respectively. Each intensity value is then rescaled to its corresponding X value (X ₁ , X ₂ , X ₃ ) using the above transformation.

Space-time conversion steps

In one embodiment, the conversion is achieved using a linear-nonlinear cascade (reviewed in Chichilnisky EJ 2001; Simoncelli et al. 2004), where the firing rate λ _m , of each ganglion cell m , is given by:

λ _m (t; X) = N _m ((X*L _m ) (j, t) (1)

Where * represents spatiotemporal convolution, _Lm is a linear filter corresponding to the spatiotemporal kernel of the mth cell, and _Nm is a function describing the nonlinearity of the mth cell, and, as described in the previous section, X is the output of the preprocessing step, j is the pixel position, and t is time. The firing rate _λm is converted into a code used to drive the interface (discussed below). This spatiotemporal conversion step can be performed by the spatiotemporal conversion module of the processing device.

_Lm is parameterized using the product of a spatial function and a temporal function. For example, in one embodiment, the spatial function consists of weights for each pixel in a grid (e.g., a digitized image from a camera), although other alternatives, such as the sum of orthogonal basis functions on the grid, may also be used. In the described embodiment, the grid consists of a 10 x 10 pixel array, with a total visual space of 26 x 26 degrees (where each pixel is 2.6 x 2.6 degrees in visual space), but other alternatives may also be used. For example, because the area of visual space corresponding to retinal ganglion cells varies with spatial location on the retina and species, the total array size may vary (e.g., from or approximately 0.1 x 0.1 degrees to 30 x 30 degrees, corresponding to a visual space of or approximately 0.01 x 0.01 degrees to 3 x 3 degrees for each pixel in a 10 x 10 pixel array). It will be understood that the angular ranges and sizes of the pixel arrays are merely illustrative of a particular embodiment, and other pixel array angular ranges or sizes are also encompassed by the present invention. For any chosen array size, the number of pixels in the array can also vary depending on the shape of the region in visual space represented by the cell (e.g., arrays of or about 1 x 1 to 25 x 25 pixels). Similarly, the temporal function consists of the sum of weights for several time bins, which in turn are raised cosine functions in the logarithmic time of other time bins (Nirenberg et al. 2010; Pillow JW et al. 2008). Other alternatives can also be used, such as sums of orthogonal basis functions.

In the described embodiment, the time sample span is 18 time bins of 67 ms each, for a total duration of 1.2 sec, but alternatives may be used. For example, because different ganglion cells have different phasic properties, the duration span in bins and the number of bins required to represent the cell dynamics may vary (e.g., a duration of or approximately from 0.5 to 2.0 sec and a number of bins of or approximately from 5 to 20). The phasic properties may also vary from species to species, but such variations are still within the above ranges.

Equation 1 can also be modified to include terms that modify the encoder output based on the past history (ie, the spike trains that cell m has produced) and the past history of the outputs of other ganglion cells (Nirenberg et al. 2010; Pillow JW et al. 2008).

For the two sets of parameters of L (spatial and temporal), the choice of resolution (pixel size, block size) and span (number of pixels, number of time blocks) is determined by two factors: the need to obtain an approximate surrogate for the retinal code and the need to keep the number of parameters small enough to be determined by a practical optimization procedure (see below). For example, if the number of parameters is too small or the resolution is too low, the surrogate value will not be accurate enough. If the number of parameters is too large, the optimization procedure will overfit and the conversion result will not be obtained (Equation 1). Using an appropriate set of basis functions is a strategy that can reduce the number of parameters and thus avoid overfitting, i.e., a "dimensionality reduction" strategy. For example, the temporal function (covering 18 time blocks, each of 67 ms) can be parameterized by the sum of 10 weights and the basis functions; see the "Example 1, Method of Constructing an Encoder" section and (Nirenberg et al., 2010; Pillow JW et al. 2008).

The nonlinearity _Nm is parameterized using a cubic spline function, but other parameterization methods such as piecewise linear functions, higher-order spline functions, Taylor series, and quotients of Taylor series may also be used. In one embodiment, the nonlinearity _Nm is parameterized using a cubic spline function with 7 knots. The number of knots is selected to accurately capture the nonlinear shape while avoiding overfitting (see the above discussion on overfitting). At least two knots are required to control the end point, and therefore the number of knots can range from about 2 to at least about 12. The spacing of the knots should cover the range of values given by the linear filter output of the model.

For the spatiotemporal transformation step, in addition to the linear-nonlinear (LN) cascade described above, alternative mappings are also included within the scope of the present invention. Alternative mappings include, but are not limited to, combinations of artificial neural networks and other filters, such as linear-nonlinear-linear (LNL) cascades. In addition, the spatiotemporal transformation can incorporate feedback from the spike generation stage (see below) to provide historical correlations and interneuronal relationships, as described in (Pillow JW et al. 2008; Nichols et al., 2010). For example, this can be achieved by convolving additional filter functions with the output of the spike generator and passing the results of these convolutions through the nonlinear verification in Formula 1.

Other models can also be used in the spatiotemporal transformation step. Non-limiting examples of models include the following: the model described in Pillow JW et al. 2008; dynamic gain control; neural networks; models expressed as ordinary differential equations, where the system of equations is an ordinary algebraic formula of integration, differentiation, and approximation to discrete time steps, whose form and coefficients are determined by experimental data; models expressed as the result of a sequence of steps consisting of linear projection (convolution of the input with the spatiotemporal kernel) and nonlinear distortion (transformation of the resulting scalar signal by a parameterized nonlinear function), whose form and coefficients are determined by experimental data; models where the spatiotemporal kernel is the sum of a small number of terms, Each term is the product of a function of a spatial variable and a function of a spatial variable and a function of a time variable, which is determined by experimental data; the spatial and/or temporal function is represented by a model of a linear combination of a set of basis functions, the size of the basis function set is smaller than the number of spatial or temporal samples, and its weight is determined by experimental data; the nonlinear function is a model consisting of one or several segments, all of which are polynomials, whose intercept points and/or coefficients are determined by experimental data, and the model is a combination of the outputs of the above models, which may be recursively combined through calculation steps such as addition, subtraction, multiplication, division, square root, exponentiation and super functions (for example, exponentiation, sine and cosine).

Spike generation steps

In the spike generation step, the ganglion cell firing frequency is converted into a pulse pattern (also called a stream), which is equivalent to a ganglion cell spike train. This step can be performed by the output generation module of the processing device.

In one embodiment, for each cell m, a non-homogeneous Poisson process with an instantaneous discharge frequency λ _m is established. In one embodiment, time intervals (blocks) of length Δt are used. For each neuron, the output of the spatiotemporal conversion given in the above formula 1, i.e., λ _m (t; X), is multiplied by Δt to obtain the discharge probability. A random number uniformly distributed between 0 and 1 is selected. If the number is lower than the discharge probability, a spike potential is generated at the beginning of the time interval. In one embodiment, Δt is 0.67ms, but other block widths can be used. The number of Δt is selected by a standard method for generating a Poisson process, that is, the block width is selected so that the product of the block width and the maximum discharge frequency is much lower than 1. The block size is selected as a compromise between computer efficiency and the high temporal resolution and wide dynamic range allowed. A person of ordinary skill in the art can make a choice without having to conduct too much experimentation. That is, a smaller block size will result in an increase in computational time, while a larger block size will blur the resolution of the spike potential type.

For the spike generation step, alternative methods may also be used, including but not limited to, the inhomogeneous gamma process, the integrate-and-discharge process, and the Hodgkin-Huxley spike generator (Izhikevich EM 2007; Izhikevich EM 2010).

The output of the encoder, a stream of pulses, is ultimately converted into a form suitable for driving a sensor, such as an electrode, a ChR2 protein, or other light-sensitive element. One potential problem is that the output of a given encoder may include a pulse train, in which several pulses occur in rapid succession (a spike "train," or a spike train, or a train of pulses, or a pulse train). If a particular type of sensor (e.g., ChR2) cannot recognize such a pulse train, the performance of the prosthesis may be slightly reduced.

The method of the present invention addresses the aforementioned issues and is referred to as a burst elimination step, correction step, or modification step. If the encoder output contains burst sequences, a replacement method is employed in which the very short intervals between spikes (or pulses) are minimized. This can be accomplished by generating a Poisson-variant code. To achieve this in a manner that matches the real-time requirements of the prosthesis, the following operations can be performed: Each brief segment of the spike generator output is examined as it is generated (i.e., the rescaled output, spatiotemporal conversion, and spike generation steps). Segments containing a number of pulses greater than or equal to a predetermined standard number, _Nseg , are replaced by segments having a number of pulses equal to _Nseg and approximately equal spacing. In one embodiment involving ChR2, for a segment with a duration of Tseg = 33 ms, the standard number of pulses, _Nseg , used for replacement is 3. _Tseg can be selected from or approximately between 3 ms and 66 ms, and _Nseg can be selected from or approximately between 2 and 20. As an alternative to this step, a burst elimination step can remove any pulses that appear within a T _win window of the previous pulse to ensure that no pulses greater than a standard number N _win appear within that window. In this case, T _win can be selected in the same manner as T seg described above, and N _win can be selected in the same manner as N _seg described above. The values of T _seg , N _seg , T _win , and N _win are selected to suit the dynamics of the particular sensor being used.

As mentioned above, problems with spike trains can cause encoder performance degradation. This problem appears to be relatively rare; for example, in the 12,000 1-second long spike train used to generate the baby face shown in Figure 9, approximately 1% of the spike trains required a spike correction step.

It is important to note that the variability in the spike potentials produced by the encoders of the present invention is lower than that of normal retina, particularly due to the lower noise. Therefore, the encoders can carry more information about the stimulus than real cells.

Determination of space-time conversion parameter values

As described in the previous section, in one embodiment, the spatiotemporal transformation is achieved by a linear-nonlinear (LN) cascade as given in Formula 1. This section describes a method for determining the parameters _Lm and _Nm in the formula. First, two stimuli are presented to the normal biological retina: white noise (WN) and natural scene images (NS). The encoder used to generate the data shown in Figures 3-9, 12, 13 and 14 presents the stimuli for 10 minutes each and continuously records the responses of the ganglion cells to both of them; the data set contains the responses to the two stimuli. The presentation can last for at least about 5 minutes, at least about 10 minutes, at least about 15 minutes or at least about 20 minutes each, and other time intervals can also be used. A person of ordinary skill in the art can determine the length of the detection time without excessive experimentation. The values of the parameters _Lm and _Nm are then selected to maximize the logarithmic similarity of the peak potential sequence observed in the rate function given in Formula 1, where the logarithmic similarity, i.e., Z, is given by the following formula

where all terms are defined as above, except that τ _m (i) is the time of the i-th spike in the m-th cell responding to stimulus X. It is noteworthy that in Equation 2, Z depends indirectly on L _m and N _m , since these quantities are related to λ _m calculated by Equation 1. To maximize the logarithmic similarity, the following steps can be followed. First, the nonlinearity N _m is assumed to be exponential, since in this case the logarithmic similarity, i.e., Z, has no local maxima (Paninski et al. 2007). After optimizing the linear filter and the exponential nonlinearity (e.g., by coordinate-raising), the nonlinearity is replaced by a spline curve. The final model parameters are then determined by alternating stages of logarithmic maximum similarity, where (i) the spline parameters and (ii) the filter parameters are maximized until a maximum value is reached.

This approach can also be used to extend Equation 1 to include history dependencies and correlations between ganglion cells, as described in (Pillow JW et al. 2008; Nichols et al. 2010).

Alternatively, other suitable optimization methods for determining the parameters can be used instead of using maximum likelihood. Non-limiting examples include optimization of a cost function, such as the mean squared error between the rate function _λm calculated for each stimulus X, and the measured firing rate of the mth cell in response to stimulus X. In addition, other optimization methods can be used for the parameter estimation step (e.g., instead of gradient ascent), such as line search or simplex method. Other optimization techniques can also be used (see, for example, Pun L 1969).

Compared to using a single type of stimulation (e.g., only WN or NS), discovering the parameters of spatiotemporal transformations, or more generally, discovering the parameters for the encoder (also called the input/output model for the cell), using WN and NS stimulation provides a unique set of parameters.

A long-standing challenge in developing input/output models for retinal ganglion cells, or other retinal cells, is that models often work well for one type of stimulus but not for others. For example, a model that works best for WN stimulation may not work well for NS stimulation, and vice versa.

Strategies for addressing this problem have focused on using biological approaches, that is, incorporating appropriate mechanisms into the model to adapt it to different image statistics. Approaches include quasi-linear models with explicit adaptation components (e.g., parameters that depend on the input statistics (see, for example, Victor (1987), where the time constant of the filter explicitly depends on the input contrast), or nonlinear models, where adaptation is a general property of nonlinear dynamics (see Famulare and Fairhall (2010). However, these strategies cannot be implemented in the data-driven approach for the wide range of stimuli required by the present invention: for quasi-linear models, the number of parameters is too large relative to the amount of data that can be provided by experimental retinal recordings, which may hinder their use, and for nonlinear models, making progress is very difficult because it is not clear what functional form should be used for the dynamics (e.g., so that it accurately captures the responses to WN and NS).

As shown in the examples herein, the method used herein is very effective, that is, it is able to produce very reliable mappings of input/output relationships for a wide range of stimuli, including both artificial and natural stimuli. It is effective in most cases because WN and NS are complementary. In particular, in the temporal and spatial domains, NS stimuli have a higher weight in the low-frequency direction compared to WN stimuli (and WN stimuli have a higher weight in the high-frequency direction compared to NS stimuli). Their complementarity has a major advantage. Compared to each individual stimulus set, the combined stimulus set can be sampled in a diverse space of inputs, which drives optimization at different locations in the parameter space. These parameters are not the average of the parameters found in WN and NS alone, but a unique set of model parameters that describe the response to these two stimulus sets and other stimuli (gratings, etc.). The latter makes the model generalizable; that is, the latter allows the encoder to work well under a wide range of stimulus conditions (including artificial and natural stimuli), that is, when exposed to the same stimulus, the response it produces is the same or substantially similar to that produced by normal retinal cells.

Although we have described and constructed encoders in modular form with a specific set of algorithmic steps, it is obvious that algorithms or devices with substantially similar input/output relationships can be constructed using different steps, or in non-modular form, such as combining any two or three steps into a single computational unit, such as an artificial neural network.

As for the encoder of the present invention, it is possible to generate a data set that does not require the collection of physiological data, which can be used, for example, to develop parameters for alternative spatiotemporal transformations or to train neural networks to produce the same or similar outputs as using methods known in the art. Therefore, the explicit description of the encoder can be used in the development of prostheses and other devices, such as, but not limited to, biomimetic devices (e.g., devices that provide supernormal capabilities) and robotic devices (e.g., artificial vision systems).

For example, such an artificial neural network can use an input layer, where each node of the input layer receives input from pixels of an image, followed by one or more hidden layers, where the nodes of the hidden layers receive input from the nodes of the input layer and/or from each other, and then an output layer, where each node of the output layer receives input from the nodes of the hidden layer. The activity of the output nodes corresponds to the output of the encoder. To train such a network, any standard training algorithm can be used, such as backpropagation, where the training input consists of the stimuli used to construct the encoder (i.e., white noise and images of natural scenes), and the training output consists of the output of the encoder. This example illustrates that it is not necessary to collect more physiological data, and alternative methods can be developed (Duda and Hart 2001).

Parameters can be developed using a variety of models involving the relationships between neurons. Parameters can be developed for neuronal models where neurons are considered independent, coupled, or correlated. Coupled models allow for the inclusion of terms that allow a spike in one neuron to influence the probability of subsequent spikes in other neurons (Nichols et al. 2010; Pillow JW et al. 2008).

Determine the type of signal that drives bipolar cells to drive ganglion cells to produce normal or near-normal retinal output.

As mentioned above, sensors target ganglion cells. Here, sensors targeting bipolar cells are described, specifically using ChR2 as an example.

Provided herein is a method for determining the type of light stimulation delivered to a ChR2-expressing bipolar cell to produce a normal ganglion cell firing pattern. Using a ganglion cell encoder for the input/output relationship, or a ganglion cell encoder as described above, the light pattern that drives the bipolar cell can be reverse engineered. In short, the transformation from an image to ganglion cell output is known, and this transformation can be used to determine the light pattern that can be presented to a ChR2-expressing bipolar cell to produce the same ganglion cell output.

The method is as follows. In a multi-electrode recording experiment, an arbitrary light pattern is presented to a ChR2-expressing bipolar cell, and the ganglion cell response is recorded; this data is used to determine the conversion between the ChR2-expressing bipolar cell and the ganglion cell. This conversion is then reversed. The reverse conversion starts from the arbitrary target ganglion cell output and returns to the light pattern present in the ChR2-expressing bipolar cell.

To implement the above method, the spatiotemporal conversion from bipolar cells to ganglion cells is determined by the following formula:

λ _m (t)=N _m ((S*L _m )(t)) (3)

Here, S is the input to the ChR2-expressing bipolar cell, L and N are linear or nonlinear filters that convert bipolar cells to ganglion cells, and λ is the firing rate of the ganglion cell. To obtain the parameters L and N, we light-type-drive the ChR2-expressing bipolar cell, record the ganglion cell response, and optimize the model parameters as described in the previous section. Using the model parameters, we can determine the input required for ChR2 to produce the target ganglion cell output. Formally, the reverse transformation involved is expressed using Equation 3. For example, the following formula can be used:

This formula gives the next input S(t), which is a function of the target output λ(t) and the input S(t-aΔt) that has been delivered before this point in time. When the filter function L is non-zero, the sum is calculated within the range of times with a span of a. The algorithm of the present invention is based on

λ _m (t) = N _m ((S*L _m )(t))

Convolution is represented as a discrete sum and implemented using a simple algebraic method.

The above formula represents a formal inversion. In practical applications, the time step Δt and the number of intervals A are chosen empirically, without the need for experimentation. It should also be noted that the nonlinearity N may not have a unique inversion, but this is not a problem, as for these purposes, only one solution is needed, not a unique solution—that is, only some form that can drive the bipolar cells to produce the correct output (i.e., normal or near-normal output). Therefore, any inversion can be chosen based on the task at hand. It is important to note that the ganglion cell encoder is the foundation of this method. By understanding the ganglion cell input/output (stimulus/response) relationship, the ganglion cell encoder can clearly determine the type of light required to drive the bipolar cells to produce the normal ganglion cell discharge pattern, that is, the same or substantially similar discharge pattern as that produced by normal retinal ganglion cells in response to the same stimulus.

sensor:

The sensor is capable of receiving an input signal and driving a neuron to discharge or change its voltage based on the received signal. In a preferred embodiment, the sensor is targeted to a single cell and is, for example, but not limited to, a light-sensitive protein or an electrode targeted to a single cell. In other embodiments, the sensor is targeted to a small group of cells; a small group of cells can consist of one cell, a group of cells, or about 100 cells. In a preferred embodiment, a group of sensors is used and each sensor is targeted to a single cell or a small group of cells as described above. We refer to this group of sensors as a high-resolution sensor. More than one sensor can be targeted to a given cell or a small group of cells; for example, channelrhodopsin-2 and halorhodopsin can be targeted to a single cell.

The sensor can drive any retinal cell, including but not limited to retinal ganglion cells and retinal bipolar cells, to discharge or generate a voltage change. An interface device can be used to connect the encoder and the sensor.

The sensor can use any suitable mechanism and can include electrodes, optogenetic stimulators, thermal stimulators, photothermal stimulators, etc. (Wells et al. 2005) In one embodiment, a sensor, such as an electrode, is implanted in the patient's eye in a manner that stimulates retinal ganglion cells or retinal bipolar cells. In another embodiment, direct photoactivation, such as a light absorption-based system, is used for the sensor.

Other sensors are also within the scope of these techniques, as are combinations of sensors or multiplexing of sensors.The sensor can be a light-responsive element, including but not limited to, a protein, such as a light-sensitive protein, or a light-responsive chemical entity.

Light-sensitive proteins that function as sensors are light-gated ion channels that can transport ions across membranes in response to light (Zhang et al. 2009; Lagali et al. 2008). Light-sensitive proteins can respond to visible light, ultraviolet light, or infrared light. Examples of light-sensitive proteins include channelrhodopsin-1, channelrhodopsin-2, LiGluR, ChETA, SFO (step-function opsin), OptoXR (photosensitive G protein-coupled receptor), channelrhodopsin-1, channelrhodopsin-2 (ChR2), ChIEF, NpHr, eNpHR, and combinations thereof. Light-sensitive proteins or active fragments thereof can be used as sensors. (European Patent Application No. 19891976.)

Examples of light-sensitive chemical entities that can be used as sensors include synthetic photosensitizable azobenzene-regulated K+ (SPARK), depolarizing SPARK (D-SPARK), photoswitchable affinity labels (PALs), CNB-glutamate, MNI-glutamate, BHC-glutamate, and combinations thereof.

In one embodiment, the sensor is a light-responsive element in a retinal ganglion cell. The code generated by the encoder can be represented by a bit stream (e.g., a stream of 0s and 1s, where 0 = no spike, 1 = spike). Subsequently, the bit stream is converted into a stream of light pulses (e.g., 0 = no light, 1 = light). Since ganglion cells contain light-responsive elements (such as light-sensitive proteins, such as ChR2), they convert light pulses into changes in membrane voltage, and since ganglion cells are action potential neurons, light pulses cause spikes to be generated, i.e., action potentials to be generated. If the intensity of the light pulse is appropriate, for example, in the range of 0.4-32 mW/ ^mm2 , the light pulse and the subsequent action potential are almost one-to-one matched (as shown in Example 13). In this way, the discharge type of the ganglion cell follows the signal of the encoder.

In another embodiment, the sensor is a light-responsive element in the retinal bipolar cells. In this case, the ganglion cells are driven indirectly: light stimulates the bipolar cells, which in turn send signals to the ganglion cells directly or indirectly (e.g., through apodocytes), causing them to fire. In this case, the stimulation provided to the bipolar cells can be discrete pulses or continuous waves. When a light-sensitive element such as ChR2 is exposed to light, it causes the bipolar cells to produce a voltage change and release neurotransmitters to their downstream neurons, ultimately causing the ganglion cells to fire.

Background firing in certain cells may interfere with the ability of light-sensitive proteins (e.g., ChR2) to track encoder outputs. In one embodiment, to correct background firing in retinal ganglion cells, ChR2 and halorhodopsin (or its equivalent) can first be expressed in each cell. When activated with yellow light, halorhodopsin will hyperpolarize the cell, inhibiting firing. When the cell is about to fire, the yellow light is turned off and blue light appears. Blue light activates channelrhodopsin-2 (ChR2), which depolarizes the cell, causing an action potential to be emitted. In this way, the cell can be illuminated by yellow light to inhibit background firing, and the light can be converted from yellow to blue to produce a discharge. In other embodiments, the same bidirectional control strategy can also be used for non-peak cells - yellow light hyperpolarizes the cell and blue light depolarizes the cell.

Furthermore, as discussed above, sometimes the encoder generates a series of spikes in rapid succession (i.e., a pulse train), which a sensor such as ChR2 may not be able to track well. To address this issue, a Poisson variant of the code can be generated. This version of the code has the same meaning to the brain as the normal code, but is adapted to the dynamics of the sensor. For example, the encoder can be adapted so that the resulting code is not rapid and continuous, but is more adapted to the dynamics of ChR2. Alternatively, a variant of ChR2 can be used that tracks spikes more closely. The specific strategy for this is described in the spike generation section above.

Carriers used for photosensitive elements

Genes, such as those encoding light-sensitive proteins, can be introduced into retinal cells using viral and non-viral vectors and methods. Viral vectors include, but are not limited to, adenoviruses, adeno-associated viruses, retroviruses, lentiviruses, herpes viruses, vaccinia viruses, poxviruses, baculoviruses, and bovine papillomaviruses, as well as recombinant viruses such as recombinant adeno-associated viruses (AAV), recombinant adenoviruses, recombinant retroviruses, recombinant poxviruses, and other viruses known in the art (Ausubel et al. 1989; Kay et al. 2001; and Walther and Stein 2000; Martin et al. 2002; van Adel et al. 2003; Han et al. 2009; U.S. Patent Publication No. 20070261127). Methods for assembling recombinant vectors are well known (see published PCT application WO2000015822 and other references cited therein).

One embodiment is an adeno-associated virus. A variety of different serotypes have been reported, including AAV1, AAV2, AAV3, AAV4, AAV5, and AAV6. The AAV sequences used to generate the vectors and capsids used in the present invention, as well as other constructs, can come from a variety of sources. For example, sequences can be provided by AAV type 5, AAV type 2, AAV type 1, AAV type 3, AAV type 4, AAV type 6, or other AAV serotypes, or other adenoviruses, including currently identified human AAV types and unidentified AAV serotypes. Many of these viral serotypes and strains can be obtained from the American Type Culture Collection (Manassas, Va.) or can be obtained from multiple scientific research institutions or commercial sources. Alternatively, it may be necessary to use synthetic sequences in the process of preparing the vectors and viruses of the present invention using techniques known in the art; these techniques can utilize AAV sequences that have been published and can be obtained from multiple databases. The source of the sequences used to prepare the constructs of the present invention is not intended to limit the present invention. Similarly, the choice of species and serotype of AAV to provide these sequences is within the skill of the art and does not limit the invention described below. AAV can be self-complementary. (Koilkonda et al. 2009)

The vectors of the present invention can be constructed and produced using the materials and methods described herein and common knowledge known to those skilled in the art. Such engineering methods for constructing any embodiment of the present invention are well known to those skilled in the art of molecular biology and include genetic engineering, recombinant virus engineering and production, and synthetic biology techniques. See, for example, Sambrook et al. and Ausubel et al., cited above; and published PCT application WO1996013598. In addition, suitable methods for producing rAAV expression cassettes in adenoviral capsids have been described in U.S. Patent Nos. 5,856,152 and 5,871,982. Methods for delivering genes to ocular cells are also well known in the art. See, for example, Koilkonda et al., 2009 and U.S. Patent Publication No. 20100272688.

Genes can also be delivered by other non-viral methods known in the art, including, but not limited to, plasmids, cosmids, and phages, nanoparticles, polymers (e.g., polyethyleneimine), electroporation, liposomes, and Transit-TKO transfection reagent (Mirus Bio, Madison, USA). Cai et al. 2010; Liao et al. 2007; Turninovich et al. 2010. Wright has reviewed in detail techniques for transfecting genes into target cells within the eye (Wright 1997). Encapsulated cell technology developed by Neurotech (Lincoln, RI, USA) can also be used.

Regulatory sequences

The vector may include appropriate expression control sequences, including but not limited to, transcription initiation, termination, promoter and enhancer sequences; efficient RNA processing signals, such as splicing and polyadenylation signals; sequences that stabilize cytoplasmic mRNA; sequences that enhance translation efficiency (i.e., Kozak conserved sequences); sequences that enhance protein stability; and, if necessary, sequences that enhance protein processing and/or secretion. A large number of different expression control sequences, such as natural, constitutive, inducible and/or tissue-specific, are known in the art and can be used to drive gene expression depending on the type of expression target. One of ordinary skill in the art can select appropriate expression sequences without undue experimentation.

For eukaryotic cells, expression control sequences typically include a promoter, an enhancer, such as those from immunoglobulin genes, SV40, cytomegalovirus, and a polyadenylation sequence, which may include splice donor and acceptor sites. The polyadenylation sequence is typically inserted after the transgene sequence and before the 3-ITR sequence. In one embodiment, bovine growth hormone polyA is used.

In the methods of the present invention, another regulatory element used in the vector is an internal ribosome entry site (IRES). An IRES sequence or other suitable system can be used to produce more than one polypeptide from a single gene transcript. An IRES (or other suitable sequence) is used to produce proteins containing more than one polypeptide chain, or to express two different proteins from or in the same cell. An example of an IRES is the poliovirus internal ribosome entry sequence, which supports transgene expression in retinal cells.

The promoter introduced into the vector can be selected from a large number of constitutive or inducible promoters that are capable of expressing the selected transgene in ocular cells. In one embodiment, the promoter is cell-specific. The term "cell-specific" refers to the ability of a particular promoter used in a recombinant vector to direct expression of a selected gene in a specific ocular cell type. In one embodiment, the promoter specifically expresses the transgene in retinal ganglion cells. In another embodiment, the promoter specifically expresses the transgene in bipolar cells.

As discussed above, each type of retinal ganglion cell or retinal bipolar cell uses its own unique code. In one embodiment of the present invention, only one type of ganglion cell is targeted. Expression of a light-sensitive protein can be controlled by a cell-specific promoter. For example, in ON bipolar cells, expression can be controlled by the mGluR6 promoter (Ueda et al. 1997). For example, a light-sensitive protein can be expressed in retinal ganglion cells using a ganglion cell-specific gene promoter, such as Thy-1 (Arenkiel et al. 2007; Bamstable et al. 1984).

In one embodiment, the sensor can be targeted to a specific type of retinal cell by using a specific dual-vector cre-lox system described herein (for an overview of cre-lox methodology, see Sauer (1987)). For example, ChR2 can be targeted to the OFF ganglion cell subtype as follows: In one viral vector, the inverted ChR2 gene is inserted in reverse, flanked by loxP sites, under the control of the calbindin promoter; calbindin is expressed in the OFF retinal ganglion cell subtype and certain apodocytes (Huberman et al., 2008). Subsequently, another viral vector expressing Cre recombinase is introduced under the control of the Thy-1 promoter (a promoter expressed in retinal ganglion cells) (Barnstable et al. 1984). Since the Thy 1 promoter expresses Cre recombinase only in ganglion cells, the inverted ChR2 will be inverted and expressed only in these cells, not in apodocytes. Expression of ChR2 in the correct orientation occurs only in cells where both the calbindin promoter and the Thy 1 promoter are active, i.e., in the OFF ganglion cell subtype. (It is worth noting that both the Thy 1 and calbindin promoters can be activated in areas outside the retina, but this will not result in expression of the genes in the vector since the vector is intended for use only in the eye, specifically the retina).

This concept can also be implemented in reverse (how useful it is depends on the promoters we have): For example, we can use Thy-1 to drive ChR2 in ganglion cells. We put it in the correct orientation and flank it with lox sequences. Then, we use another promoter, such as the GABA A receptor promoter, to activate Cre recombinase in certain subsets of ganglion cells. Cre will invert ChR2 in these cells, turning it off—so ChR2 will only be active in cells that express Thy-1 and the other promoter. It doesn't matter if Cre is activated in other cell types because they don't have ChR2, so it won't be turned off either.

These same methods can be applied to other types of retinal ganglion cells. Targeting can be achieved by replacing a promoter in the place of the calcium-binding protein promoter, such as the SPIG1 promoter (Yonehara et al. 2008, Yonehara et al. 2009), the DRD4 promoter (Huberman et al. 2009), the neurofilament promoter (Nirenberg and Cepko, 1993), and other promoters that drive expression in a subset of ganglion cells, such as those identified by Siegert et al. (2009). The dual-vector Cre-Lox system described herein can also be easily extended to target other cell types. Promoter analysis can be used to identify functional fragments and derivatives of promoters (McGowen et al. 1998; 4:2; Bookstein et al. 1990).

In one embodiment, multiple types of retinal neurons are targeted, and different sensors, such as different ChR2 derivatives, can be expressed in different types of cells. Different sensors, such as different ChR2 derivatives, can differ in their properties, including excitation wavelength. Thus, codes can be delivered to specific cell types by presenting them at different wavelengths. For example, if we place blue-sensitive sensors only in OFF cells, we can then selectively drive OFF cells by delivering blue light pulses generated by the OFF cell code. Other cell types will not respond to blue light and will not be driven by the OFF cell code.

The structure of the ganglion cell layer (GCL) of the primate retina also allows for targeting of specific cell types. The ganglion cell bodies are within the GCL. The GCL is thinnest near the fovea and contains several layers of cell bodies. The cell bodies of different cell types are located in different locations within the GCL. For example, ON cell bodies are closer to the retinal surface (closer to the vitreous) than OFF cell bodies (Perry and Silveira, 1988). In this way, it is possible to preferentially target ON cells. This can be achieved, for example, by low-dose infection with a viral vector (e.g., AAV carrying ChR2); low-dose infection will preferentially target cells close to the surface. This method is not limited to the fovea and can be applied to any area of the retina containing multiple sublayers of the GCL.

In another embodiment, light-responsive elements can be expressed in bipolar cells. For example, light-responsive elements, such as a gene encoding channelrhodopsin-2, can be targeted to bipolar cells using the mGluR6ChR2 plasmid (Ueda et al. 1997; U.S. Patent Publication No. 20090088399) or other highly efficient adeno-associated viruses (Morgans CW et al. 2009; Cardin JA, et al. 2010; Petrs-Silva et al. 2009; Petersen-Jones et al. 2009; Mancuso et al. 2009). Bipolar cell-specific promoters can also be used, such as the promoter of glutamate receptor genes expressed in ON bipolar cells (see Lagali et al. 2008) or the promoter of dystrophin (Fitzgerald et al. 1994). Promoter analysis can be used to identify functional fragments and derivatives of promoters (McGowen et al. 1998; 4:2; Bookstein et al. 1990).

Examples of constitutive promoters that can be introduced into the vectors of the present invention include, but are not limited to, CMV-mediated early enhancer/chicken β-actin (CβA) promoter-exon 1-intron 1 element, RSV LTR promoter/enhancer, SV40 promoter, CMV promoter, 381 bp CMV-mediated early gene enhancer, dihydrofolate reductase promoter, phosphoglycerate kinase (PGK) promoter, and 578 bp CBA promoter-exon 1-intron 1. (Koilkonda et al. 2009). Promoter analysis can be used to identify functional fragments and derivatives of promoters (McGowen et al. 1998; 4:2; Bookstein et al. 1990).

Alternatively, the transgenic product is expressed by introducing an inducible promoter to control the amount of the product in the eye cells and the time of production. If it has been confirmed that the gene product is toxic to the cells when it accumulates excessively, such a promoter can be used. Inducible promoters include those well known in the art and discussed above, including but not limited to, zinc-inducible sheep metallothionein (MT) promoter; dexamethasone (Dex)-inducible mouse mammary tumor virus (MMTV) promoter; T7 promoter; insect ecdysone promoter; tetracycline repressible system; tetracycline inducible system; RU486 inducible system; and rapamycin inducible system. Any type of strictly regulated inducible promoter can be used. Other types of inducible promoters that can be used in this article are promoters that can be regulated by specific physiological states, such as temperature, acute phase, specific cell differentiation state, or only in replicating cells.

The selection of these and other common vectors and regulatory elements is routine, and many such sequences are commercially available. See, for example, Sambrook et al. 1989 and Ausubel et al. 1989. Of course, not all vectors and expression control sequences function equally well in expressing all transgenes of the present invention. However, those skilled in the art can make choices among these expression control sequences without departing from the scope of the present invention. Using the guidance provided herein, one skilled in the art can select an appropriate promoter/enhancer sequence. Such selection is a routine choice and not a limitation on the molecule or construct. For example, one or more expression control sequences can be selected, operatively linked to the transgene of interest, and expression control sequences added, and the transgene introduced into the vector. The vector can be packaged into infectious particles or virions according to any method known in the art for packaging vectors.

The vector is preferentially evaluated for contamination by conventional methods and then formulated into a pharmaceutical composition for retinal injection. The vector is described in detail above for targeting ocular cells and contains the desired photosensitizer and cell-specific promoter. Such formulations involve the use of pharmaceutically and/or physiologically acceptable excipients or carriers, particularly those suitable for vitreous, retinal, or subretinal injection, such as buffered saline or other buffers, such as HEPES, to maintain a pH at an appropriate physiological level. A variety of such known vectors are provided in published PCT application WO2002082904, which is incorporated herein by reference. If the virus requires long-term storage, it can be frozen in the presence of glycerol.

According to the methods of the present invention, i.e., methods for treating ocular diseases characterized by retinal degeneration, the pharmaceutical compositions described above are administered to a subject suffering from such blinding diseases via intravitreal, retinal, or subretinal injection. Methods for administering vectors to the eye are well known in the art. See, for example, Koilkonda et al., 2009 and U.S. Patent Publication No. 20100272688.

An effective amount of a nucleic acid sequence encoding a photosensitive element of interest, carried by a vector under the control of a cell-specific promoter sequence, can range from about 1 x ¹⁰⁹ to 2 x ¹⁰¹² infectious units per about 150 to about 800 microliters. Infectious units are assayed as described in McLaughlin et al., 1988. More desirably, an effective amount is between about 1 x ¹⁰¹⁰ and 2 x ¹⁰¹¹ infectious units per about 250 to about 500 microliters. Other dosages within this range can be selected by the attending medical professional, taking into account the physiological state of the subject, the treatment being administered, the age of the subject, the specific ocular disease, and the extent of disease progression (if progressive) of the subject, preferably a human.

The subsequent dosage of the pharmaceutical composition of the present invention can also be determined. For example, depending on the duration of transgene expression in the target cells of the eye, booster administration can be performed every 6 months, or once a year after the initial administration.

Such booster doses and their need are monitored by the attending medical staff using tests such as those described herein for retinal and visual function and behavioral visual testing. Other similar tests can also be used to determine how the condition of the treated subject changes over time. The attending medical staff can select the appropriate test. Alternatively, the method of the present invention can also involve administering a large volume of a solution containing the virus via single or multiple injections to achieve a level of visual function similar to that of a normal retina.

The code can be converted into light pulses by a light source such as, but not limited to, an LED array, a DLP chip, a scanning laser beam, or an LCD of a suitable source. The interface for the light-sensitive element will be described in more detail below.

In another embodiment, the sensor is an electrode. The electrical pulses generated by the encoder, through the electrode, drive ganglion cells to discharge according to the encoded pulses, either directly or via bipolar cells, or a combination thereof. The implanted electrodes may be, but are not limited to, those described in U.S. Patent Nos. 6,533,798 and 7,149,586; and U.S. Patent Publication Nos. 20080249588, 20090326623, and 20080221653.

Examples of vectors using AAV and light-sensitive proteins that can be used in this prosthesis include, but are not limited to, sc-mGluR6-hChR2-GFP, mGluR6-hChR2-GFP, sc-smCBA-CHR2-GFP, sc-smCBA-CHR2-GFP, and Flex-CBA-Chief-GFP. (Bill Hauswirth, personal communication) Newer vectors that are active in bipolar cells and use the L7 promoter, such as AAV2 or AAV2-Y444F or AAV2-Y444,500,730F, can also be used. (See, e.g., Sheridan C 2011; published PCT applications WO1998048027, WO2001094605, WO2002082904, WO2003047525, WO2003080648, WO2003093479, WO2003104413, WO2005080573, WO2007127428, WO2010011404)

equipment:

A prosthetic device implementing the methods described herein includes the following elements, which may be interconnected physically, wirelessly, optically, or otherwise as known in the art.

(1) Camera

High-fidelity images are obtained using a camera. In one embodiment, the camera is based on a charge-coupled device (CCD), such as the Point Grey Firefly MV (752x480 pixels, 8 bits/pixel, 60 frames per second) (Point Grey Research, Richmond, BC, Canada). Sending images from the camera to the processing device in real time requires a high-bandwidth connection. For example, using a USB 2.0 interface between the camera and the processing device can achieve data transfer rates exceeding 20 MB/sec.

Any device capable of capturing visual images with high spatial and temporal resolution and then transferring these images to a processing device may be used in place of a camera. These devices include, but are not limited to, devices based on charge coupled devices (CCDs); active pixel sensors (APSs) such as complementary metal oxide semiconductor (CMOS) sensors, thin film transistors (TFTs), photodiode arrays; and combinations thereof.

The camera can be connected to the processing device interface via any connection with high-speed data transmission capability, including but not limited to, a serial interface such as IEEE 1394 or USB 2.0; a parallel interface; an analog interface such as NTSC or PAL; a wireless interface; the camera and the processing device can be integrated into the same board.

(2) Processing equipment

The processing device executes the encoder, which converts the image into code in real time.

The processing device, such as a handheld computer, can be implemented by any device that can receive an image stream and convert it into an output in real time. This includes, but is not limited to, a general purpose microprocessor (GPP)/digital signal processor (DSP) combination; a standard personal computer, or a portable computer such as a laptop; a graphics processing unit (GPU); a field programmable gate array (FPGA) (or a field programmable analog array (FPAA) if the input signal is analog); an application specific integrated circuit (ASIC) (if an upgrade is required, the ASIC chip must be replaced); an application specific standard product (ASSP); a stand-alone DSP; a stand-alone GPP; and combinations thereof.

In one embodiment, the processing device is a dual-core processor-based handheld computer (Beagleboard, Texas Instruments, Dallas, TX) that integrates a general-purpose microprocessor (GPP) and a digital signal processor (DSP) onto a single chip. Compared to a typical portable computer, its motherboard is capable of high-speed parallel computing, but uses less power (~2 watts or less, compared to 26 watts for a standard laptop). This allows real-time computational transformations on a portable device that uses a single battery for long-term power. For example, a typical laptop battery has a capacity in the range of 40-60 watt-hours, which can enable the processor to operate continuously for approximately 20-30 hours. In another embodiment, the processing device is small in size so that it can be attached to glasses worn by the patient.

For a given location in space, an encoder applies a specific transformation to a series of input images, generating an encoded output that drives a target cell at the desired location in space. In one embodiment, the target cell is a retinal ganglion cell, and the encoder output is a series of electrical pulses at a specified time, at which the retinal ganglion cell fires. The timing of each pulse is calculated with submillisecond resolution. Most computation occurs in the DSP, while the GPP is used to direct image data from the camera to the processor's memory and synchronize the camera with the DSP.

In one embodiment, the target cells are retinal ganglion cells and the output format of the processing device is as follows: for a given time t, the output is a matrix of bits where the element at position (x, y) corresponds to the state of the ganglion cell at position (x, y): 1 if the cell should have spiked at time t, 0 if the cell should not have spiked at time t. The size of the matrix is graded so that it matches the number of ganglion cells that can be stimulated. The output of the encoder is then stored in memory and converted into a signal that drives the sensor through the output interface (see the description below under the heading "(4) Output Interface"). The conversion is performed in blocks. In one embodiment, the output of the encoder is stored as 16.66 ms and then converted into blocks. The range of blocks that can be used is 5 ms to 66.66 ms, with the minimum block length in time being determined by the time delay between the onset of stimulation and the first spike of the ganglion cell.

(3) Sensor

The sensor receives the signal from the device through the output interface and activates the target cells, which have been specified in the encoder. The sensor has been described in detail in the "Sensor" section above.

(4) Output interface

The output interface translates the encoded output (from the processing device) into a form that can drive the sensor. Several output interfaces can be used, depending on the sensor selected. For example, if the retinal ganglion cell encoder is paired with a light-sensitive sensor (such as ChR2) expressed in the retinal ganglion cells, the output interface can be a digital light processing (DLP) device. The DLP device can output light pulses that correspond to the encoded ganglion cell output received from the encoding device. The light pulses then drive the sensors in the ganglion cells, causing the ganglion cells to discharge according to the instructions of the encoder. In this example, the function of the output interface is as follows: the output of the encoder is transmitted from the processing unit to the output interface (DLP). The output interface then converts the binary data representing the number of action potentials into light pulses through a digital micromirror device (DMD), which is paired with a high-intensity light-emitting diode (LED). The DMD is a grid mirror whose position can be converted with high resolution over time and space. When the encoder indicates that the ganglion cell at position (x, y) should fire an action potential, the mirror image at position (x, y) in the device switches to the "on" position for a short period of time (e.g., on a millisecond timescale), and then back to the "off" position. This briefly reflects light from the LED onto the retina, causing a light pulse at position (x, y). This light pulse drives the retinal nerve cell at position (x, y) to fire.

In one embodiment, the device is paired with ChR2, a light-sensitive sensor expressed in retinal ganglion cells, and the output interface is a digital light processing (DLP) device described above (TI DLP Pico Projector Development Kit v2.0, Texas Instruments, Dallas, TX). The standard light source on the DLP device can be replaced by a high-intensity LED with an intensity sufficient to activate ChR2 (Cree XP-E Blue LED, Cree, Durham, NC). As described above, the DLP includes a digital micromirror device (DMD) (DLP1700A, Texas Instruments, Dallas, TX), which is composed of a grid of mirrors. When the retinal ganglion cell at that location should fire, each mirror in the grid can be switched to reflect light from the LED to the retina. Data from the encoder device is transmitted to the output interface via a high-definition multimedia interface (HDMI, 22 MB/sec). The position of each mirror on the DMD is controlled with high temporal resolution - when the encoder indicates that the ganglion cell should fire an action potential, the mirror at the corresponding position will be switched to the "on" position within a short period of time (1.4 ms). The conversion of the mirror state causes the device to output a light pulse to the corresponding position, which drives the target retinal ganglion cell to emit an action potential. The conversion time of the mirror can be shortened or extended, for example from 0.1ms to 10ms, which depends on the amount of light required to activate the cell. In this embodiment, the mirror array on the DMD has 480X 320 mirrors, so that it can independently target more than 150,000 positions (e.g., cells). The DLP can also have more mirrors, such as 1024X 768 mirrors, as in the DLP5500A (Texas Instruments, Dallas, TX), so that it can independently stimulate more positions. The data conversion performed between the encoding device and the interface is performed according to standard technical specifications, as proposed in the Texas Instruments application report DLPA021-January 2010-"Using the DLP Pico 2.0 Kit for Structured Light Applications".

DLP is an example of a potential output interface. The output interface can also be implemented using any device capable of activating a paired sensor. Examples of light-activated sensors include, but are not limited to, digital micromirror devices; LED arrays; spatial light modulators; optical fibers; lasers; xenon lamps; scanning mirrors; liquid crystal displays, and combinations thereof. (Golan L, et al. 2009; Grossman N, et al. 2010)

For electrode-based sensors, the output interface can consist of any device capable of driving current into the electrodes, as is known in the art.

(5) One or more of the techniques described herein, or any portion thereof, including the encoder (which may include preprocessing, spatiotemporal conversion, spike generation, and burst elimination steps), and optimization of encoder parameters, can be performed by computer hardware or software, or a combination thereof. Based on the methods and figures described herein, these methods can be implemented in a computer program using standard programming techniques. Program code is applied to input data to perform the functions described by the invention and generate output information. The output information is applied to one or more output devices such as a monitor. Each program can be executed in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if necessary, the programs can be executed in assembly language or machine language. In general, the language can be a compiled or interpreted language. Furthermore, the program can be run on an application-specific integrated circuit that has been pre-programmed for the purpose.

Preferably, each computer program of this type is stored in a storage medium or device (e.g., ROM or disk), and the medium or device is readable by a general or special purpose programmable computer, and when a computer reads a storage medium or device to execute the program described herein, the computer is set and operated. During program execution, the computer program may also be present in a cache or main memory. Analysis, pre-processing, and other methods described herein may also be implemented as a computer-readable storage medium, and a computer program is set, wherein the storage medium is set so that the computer performs the function described in the present invention in a specific and predefined manner. In certain embodiments, the computer-readable medium is tangible and substantially has a non-transient property, for example, the information recorded in this way is recorded in a form different from that of merely transmitting a signal.

In some embodiments, the program product may include a signal-bearing medium. The signal-bearing medium may include one or more instructions that, when executed, such as by a processor, may provide the functionality described above. In some implementations, the signal-bearing medium may include a computer-readable medium, such as, but not limited to, a hard drive, a compact disc (CD), a digital laser disc (DVD), a digital tape, a memory, etc. In some implementations, the signal-bearing medium may include a recordable medium, such as, but not limited to, a memory, a read/write (R/W) CD, a R/W DVD, etc. In some implementations, the signal-bearing medium may include a communication medium, such as, but not limited to, a digital and/or analog communication medium (e.g., an optical cable, a waveguide, a wired communication link, a wireless communication link, etc.). Thus, for example, the program product may be transmitted via an RF signal-bearing medium, wherein the signal-bearing medium is transmitted via a wireless communication medium (e.g., a wireless communication medium that complies with the IEEE 802.11 standard).

It will be appreciated that any of the signals and signal processing techniques may be optical or digital or analog in nature, or a combination thereof.

As described above, the output of the encoder is stored in blocks to convert the signal and use it to drive the sensor (via the output interface). For example, in one embodiment, where the output interface uses a DLP to generate light pulses, the output of the encoder is translated into a signal that controls the state of the mirror in the DLP (either reflected toward or from the retina). The conversion is performed in blocks. In one embodiment, the output of the encoder is stored as 16.66ms and converted into blocks. The range of blocks that can be used is 5ms to 66.66ms, with the minimum block length in time being determined by the time delay between the onset of the stimulus and the first spike emitted by the ganglion cell (in a normal WT retina). Another advantage of block storage is that it allows the pulse train elimination step described in the section entitled "Encoder" to be performed.

Methods for testing encoder and prosthesis performance

The following describes a procedure for testing the performance of the encoders and prostheses. Performance can be tested by at least three different methods: performance in a forced-choice visual discrimination task, or accuracy in a Bayesian stimulus reconstruction test, or performance in an error type test. The term "test stimulus" as used herein refers to a stimulus or stimulus presented to an animal to evaluate the performance of an encoder or encoder + sensor (i.e., a retinal prosthesis). The term "reconstruction stimulus" as used herein refers to a reconstructed stimulus using the methods described herein. The term "activated retina" refers to a retina that has been treated with encoders + sensors; this includes sensors targeting ganglion cells or bipolar cells.

It is important that the tasks used to test the performance of the prosthesis fall within a range where meaningful information is difficult to obtain, as shown by the tasks used in Example 8. In short, the task must be difficult enough (i.e., a sufficiently rich set of stimuli must be used) so that the normal retinal response can provide information about the stimuli, but it cannot perform the task perfectly. For example, in the tasks shown in the examples, an 80% correct score using responses from the normal retina is considered satisfactory. If the task used is too difficult, so that the performance of the normal retina approaches chance, then the match is difficult to use for performance analysis. Conversely, if the task chosen is too simple (e.g., only requires a rough discrimination, such as distinguishing black from white, and in which the correct score of the normal retina response is close to 100%), the prosthetic method will deviate far from the natural code of the retina and will not provide results close to normal vision. Therefore, it is crucial to use appropriate challenge tests, such as those used in the examples listed. The use of challenge tests can also determine whether the performance of the prosthesis is better than that of the retina (i.e., entering the field of "bionic vision").

To assess performance in a forced-choice visual discrimination task, a confusion matrix, known in the art (Hand DJ, 1981), was used. A confusion matrix shows the probability of responding to a presented stimulus based on the stimulus being decoded. The vertical axis of the matrix shows the presented stimulus (i), and the horizontal axis shows the decoded stimulus (j). The matrix element at position (i, j) gives the probability of decoding stimulus i as stimulus j. If j = i, the stimulus was correctly decoded; otherwise, it was incorrectly decoded. In short, elements located on the diagonal indicate correct decoding; elements not located on the diagonal indicate confusion.

In this task, an array of stimuli is presented, specifically, stimuli comprising natural scenes (see below for stimulus requirements for this task), and the extent to which the stimuli can be distinguished from one another is measured based on their responses to ganglion cells and/or encoders. For the data generated in Figure 8, which were used to set performance criteria for the discrimination task described herein, the responses of ganglion cells were recorded using a multi-electrode array as described in Pandarinath et al., 2010, and the stimuli were presented on a computer monitor.

A training set is obtained to establish the response distribution ("training set"), and another set is obtained to decode and calculate the confusion matrix ("test set").

To decode the responses in the test set, we need to determine which stimulus s _j is most likely to produce a response. Specifically, we need to determine which stimulus s _j is the largest with respect to p(r|s _j ). Bayes' theorem is used, which gives p(s _j |r) = p(r|s _j )p(s _j )/p(r), where p(s _j |r) is the probability of stimulus s _j being present, specifying a specific response r; p(r|s _j ) is the probability of specifying a specific response r for stimulus s _j ; and p(s _j ) is the probability of stimulus s _j being present. p(s _j ) represents the set of all stimuli in this experiment that are equal. Thus, Bayes' theorem shows that p(s|r _j ) is maximized when p(r|s _j ) is maximized. When p(s _j ) is consistent, as in this case, specifying a response, the method of finding the most likely stimulus for a response is called maximum likelihood decoding (Kass et al. 2005; Pandarinath et al. 2010; Jacobs et al. 2009). For each stimulus _si that exists and results in a response r, which is decoded as _sj , the element at position (i, j) in the confusion matrix is the increment.

To establish the response distribution required for the decoding calculation used to prepare the confusion matrix (i.e., to assign p(r|s _j ) to any response r), the following procedure is performed. In the examples of this document that generate the confusion matrix, the response r is defined as a spike train spanning 1.33 seconds after stimulus onset, with bins of 66.7 ms. Assuming the spike generation process is a non-homogeneous Poisson process, the probability p(r|s _j ) for the entire 1.33 s response is calculated by multiplying the probabilities of each 66.7 ms bin. The probability assigned to each bin is determined according to Poisson statistics, based on the average training set responses for stimulus s _j in that bin. Specifically, if the number of spikes in response r in that bin is n, and the average number of spikes in the training set responses in that bin is h, then the probability assigned to that bin is ( ^hn /n!)exp(-h). The product of these probabilities for each bin determines the response distribution for the decoding calculation used to prepare the confusion matrix.

Once the confusion matrix has been calculated, overall performance on the forced-choice visual discrimination task is quantified using the "fraction correct," which is the fraction of decoding responses that correctly identified the stimulus during the total task. The fraction correct is the average of the diagonal of the confusion matrix.

In this procedure, four sets were analyzed. For each, responses from the WT retina were used as the training set and a different set of responses was combined as a test set, as outlined below:

(1) The first set should consist of responses from WT retinas, in order to obtain the correct fraction that is generated by responses of normal ganglion cells.

(2) The second set should consist of responses from the encoders (the responses from the encoders, as described in this document, are streams of electrical pulses, in this case spanning 1.33 seconds after the stimulus is present and in chunks of 66.7 milliseconds, which are the responses of WT ganglion cells). Given the distribution of responses from a normal WT retina, the responses from this test set can be used to determine how well the encoders perform. This is based on the fact that the brain has established an interpretation of the responses of the normal WT retina (i.e., the responses of the normal encoding). When using the responses from the encoders as the test set, it is possible to test how well the brain works after replacing the normal retinal responses (our replacement of the retinal code).

(3) The third set should consist of responses from the retina of a blind animal driven by the encoder + sensor (ChR2), with the same duration and bin size of the responses as above. This set provides a test of how the encoder performs after the encoder output passes through the sensor in real tissue.

(4) Finally, the last set consists of responses from the retina of a blind animal driven only by the sensor (ChR2), with the same duration and bin size as above. This allows for a test of how well the standard optogenetic approach performs. This is essentially a control experiment that shows that the discrimination task provides a suitable test, as explained in the previous paragraph for testing suitable difficulty.

As shown in Example 8, in a forced-choice visual discrimination task, the encoder's performance was 98.75% of that of a normal retina, the performance of the complete system, i.e., the encoder + sensor of the current embodiment, was 80% of that of a normal retina, and the performance of the standard approach (sensor only) was 10% lower than that of a normal retina (8.75%). Thus, based on testing in vitro or in animal models as described above, the performance of the prosthesis in a forced-choice visual discrimination task, when measured as "fraction correct," will be at least about 35%, 50%, 60%, 70%, 80%, 90%, 95%, 99%, or 100% of the performance of a normal retina, or better than a normal retina. Notably, 35% is approximately four times higher than the performance of the optogenetic approach in Example 8. Likewise, because the sensor can be used in conjunction with other sensors or for other purposes, such as, but not limited to, robotic vision, the performance of the encoder itself will be at least approximately 35%, 50%, 60%, 70%, 80%, 90%, 95%, 99% or 100% of, or better than, the performance of a normal retina when tested as described above.

The performance of the encoder can also be tested using reconstructed stimuli. Reconstructed stimuli use standard maximum likelihood methods to determine the most likely stimulus given a set of spike trains (for review, see Paninski, Pillow, and Lewis, 2007). Although the brain does not reconstruct stimuli, reconstruction is still a convenient method that allows comparison of methods and gives an approximate estimate of the likelihood of visual restoration with various methods.

The stimulus should be a full gray screen for 1 second, followed by an image for 1 second, preferably a human face. Each pixel of the stimulus must span a reasonable visual space area so that the characteristics of the image can be recognized. In this case, the image is each face of a human face. Selecting a standard of 35X 35 pixels is sufficient, as shown in Figure 9. This is consistent with the spatial frequency used for facial recognition, which should be at least 8 cycles per face, that is, at least 32 pixels in each dimension are required to meet the sampling requirements (Rolls et al., 1985). In the embodiment shown in Figure 9, mice were used, and each pixel corresponds to a visual space of 2.6 degrees x 2.6 degrees. This in turn corresponds to approximately 12-20 ganglion cells in the mouse retina.

Stimulus reconstruction consists of searching the space of all possible stimuli to find the most likely stimulus for a given population response r. To find the most likely stimulus given r, Bayes' theorem p(s|r) = p(r|s) * p(s) / p(r) is used. Since the prior stimulus probability p(s) is assumed to be constant for all s, the maximum value of p(s|r) is equal to the maximum value of p(r|s).

To determine p(r|s), we assume that the cell responses are conditionally independent. That is, we assume that p(r|s) is the product of probabilities p( _rj |s), where p( _rj |s) is the probability that the jth cell responds with _rj , given a stimulus s. The theoretical basis for this assumption is that the bias of conditional independence is known to be small and contributes only minimally to the information carried (Nirenberg et al., 2001; Jacobs et al., 2009) and the fidelity of stimulus decoding.

To calculate p( _rm |s) for a given cell m, the response _rm is set to the spike train of the mth cell after stimulus onset, spanning 1 second and binned in 0.57 ms. Since the spike generation process is assumed to be a non-homogeneous Poisson process, the probability of a response for the entire 1 second, p( _rm |s), is calculated by multiplying the probabilities assigned to each bin. The probability of each bin being assigned is determined using Poisson statistics based on the expected firing rate of the cell in each bin to stimulus s. The expected firing rate of the cell is calculated using Equation 1 (see the "Encoder" section in the "Spatiotemporal Conversion Step"), _λm (t;X), where X is stimulus s and t is the bin duration. Finally, the probability of a population response, p( _r |s), is calculated by multiplying the response probabilities of each cell, p(rj|s).

To find the stimulus _sj that is most likely to produce the population response r, a standard gradient ascent technique is used. The goal is to find the stimulus _sj that maximizes the probability distribution p(r|s). Since the stimulus space is high-dimensional, gradient ascent is used because it provides an efficient method for searching in high-dimensional spaces. The steps are as follows: Start the search at a random point in the stimulus space _sk . Evaluate the probability distribution p(r| _sk ) for this stimulus, and calculate the slope of this probability distribution with respect to each stimulus dimension. Then, create a new stimulus sk ₊₁ by varying the stimulus _sk in a direction that increases the probability (determined by the slope of the probability distribution). This process is repeated until the stimulus probability begins to increase only marginally, i.e., p(r|s) reaches a peak. It is important to note that since the probability distribution is not strictly log-concave, it is possible that it will stop at a local maximum. To prove that this is not the case, reconstructions must be performed using multiple random starting points to ensure convergence to the same peak.

To compare the performance of the prosthetic method, three sets of responses must be reconstructed: 1) responses from the encoder, 2) responses from the blind retina where the ganglion cells are driven by the encoder + sensor (ChR2), and 3) responses from the blind retina where the ganglion cells are driven by the sensor alone (i.e., ChR2 alone). Reconstruction should be performed on processing clusters in blocks of 10 x 10 or 7 x 7 pixels in order to compare the results in the examples (particularly Figure 9).

To obtain a large enough dataset for complete reconstruction, it may be necessary to systematically move the image across the retinal region being recorded. This allows for the acquisition of responses from a single or small number of retinas to all parts of the image. In Figure 9, responses from approximately 12,000 ganglion cells were recorded for each image. Their behavior should be identical or substantially similar to that shown in Figure 9B. It is possible to identify not only that the image is of an infant's face, but also that it is the face of a specific infant, a challenging task.

To quantify the differences in performance between the methods, the reconstructions from each method must be compared to the original image. This is determined by calculating the standard Pearson correlation coefficient between the values of each pixel in the reconstructed image and the ground truth image. For this test, a correlation coefficient of 1 indicates that all original image information is fully preserved, while a correlation coefficient of 0 indicates that the similarity between the reconstruction and the ground truth image does not exceed chance.

As shown in Figure 9, the results are as follows: for the encoder alone, the correlation coefficient is 0.897; for the encoder plus sensor, the correlation coefficient is 0.762; and for the sensor alone (equivalent to the prior art), the correlation coefficient is 0.159. This is consistent with the results obtained in the discrimination task, showing that the performance of the encoder plus sensor is several times better than that of the prior art.

Thus, when tested in vitro or in an animal model, the performance of the prosthesis, as measured by reconstruction accuracy, may be as follows: the Pearson correlation coefficient between the reconstruction of the encoder + sensor (retinal prosthesis) response and the original image is at least about .35, .50, .60, .70, .80, .90, .95, or 1.0. Similarly, when tested as described above, the Pearson correlation coefficient between the reconstruction of the encoder response and the original image is at least about .35, .50, .60, .70, .80, .90, .95, or 1.0, or better than that of a normal retina. Notably, .35 is >2-fold better than the performance of the optogenetic method described in Example 8.

Another test performed on the confusion matrix data is the test for error type, i.e., "error type test", which uses a standard test method in the field, i.e., mean square error (MSE). To test the performance of the encoder and encoder + sensor (i.e., prosthetic method), the error type of the above sets (2), (3), and (4) was evaluated because this quantity was calculated with reference to set (1). The degree of match of the error type of each set ((2), (3), or (4)) with the WT (i.e., normal) (set (1)) was quantified by the mean square error (MSE), which is defined as the mean square of the difference between the elements of the confusion matrix determined by one of the test sets ((2), (3), or (4)) and the WT (set (1)). The theoretical basis of this test is that the type of decoding error indicates that the brain may be confused about the stimulus when receiving the output of the retina, i.e., it cannot distinguish between the stimuli. As shown in Example 8, the performance of the normal (WT) retina is within a range - some stimuli can be easily distinguished by the response of real cells, while some cannot. For example, as shown in the confusion matrix in the upper right corner of Figure 8 in Example 8, the population response of WT ganglion cells can clearly distinguish 10 of the 15 presented stimuli (represented by the 10 bright squares along the diagonal of the matrix); in contrast, the population response of WT ganglion cells is uncertain for the remaining 5 stimuli (represented by the squares away from the diagonal). Error type detection provides a method to quantify the degree of response of encoders, encoders + sensors, and sensors alone, and whether the same stimulus can be distinguished or difficult to distinguish can also be quantified. The method tests the degree of match between the confusion matrix of set (2), (3), or (4) and set (1); in particular, it calculates the mean squared error between the elements of the detection confusion matrix (set (2), (3), or (4)) and the WT (set (1)). To develop retinal prostheses that provide normal or near-normal vision, the neural signals delivered to the brain (i.e., the firing patterns of ganglion cells) need to be the same as the information provided by normal cells, i.e., when the prosthesis is used, stimuli that can be distinguished normally are distinguished, and stimuli that can be perceived normally are perceived in a similar manner.

When the error types were detected using the data from Example 8, the results were as follows: the performance of the encoder produced an MSE of 0.005; this was a very good match for the error types of a normal retina. The performance of the complete system (encoder + sensor) produced an MSE of 0.013, which was also very close. The sensor alone produced an MSE of 0.083, which is a very high value, indicating that it was a poor match for the normal error types. Thus, when tested in vitro and in animal models, the match to the error types of the real retina, as measured by MSE, can be at most about 0.04, 0.03, 0.02, 0.01, or 0.005. Notably, 0.04 indicates that the match is at least twice as good as the optogenetic approach in Example 8 (because 0.04 is half of 0.083), while the encoder + sensor produced a match of 0.013, which is far better than this.

To test prostheses using the methods described herein, mammalian retinas are obtained with sensors adapted for the same retinal cell types as wild-type retinas of the same species. The assays described above are then performed. For all of the above analyses, results should be consistent from retinas of the same type, for example, using at least about five retinas.

Clinical application test

vision

The World Health Organization (WHO) defines low vision as best-corrected distance visual acuity of less than 20/60 but greater than or equal to 20/400 in the better eye, or a visual field with its widest dimension less than 20 degrees but greater than 10 degrees. Blindness is defined as best-corrected distance visual acuity of less than 20/400 or less in the better eye, or a visual field with its widest dimension less than 10 degrees. In the United States, legal blindness is defined as best-corrected distance visual acuity of less than or equal to 20/400 in the better eye, or a visual field with its widest dimension less than 20 degrees. Most states in North America require a best-corrected visual acuity of 20/40 in both eyes for an unrestricted driver's license (Riordan-Eva 2010).

In the early 1980s, the visual acuity measurement method used in clinical research was the Early Treatment Diabetic Retinopathy Study (ETDRS) chart, which had five letters per line, with letter and line spacing equal to letter size, and letter size increasing in a geometric progression (Kniestedt and Stamper, 2003; Ferris FL et al., 1982; ETDRS report number 7). Subsequently, an electronic equivalent test method, the so-called electronic visual acuity assessment (EVA), was developed and validated and is now widely used (Beck RW et al., 2003; Cotter SA et al., 2003).

Vision testing methods are known in the art. The standard testing process using the ETDRS visual acuity chart is as follows.

A. Eye chart: Modified Bailey-Lovie

Standard visual acuity testing uses ETDRS charts 1 and 2. Regardless of the subject's visual acuity, visual acuity testing begins at a distance of 4 meters. Visual acuity testing uses two sets of ETDRS charts, each with a different alphabetical order. Chart 1 is always used for testing the right eye, and Chart 2 for testing the left eye.

B. Eye chart and room lighting

If the EVA is nonfunctional, each clinic must have/use an ETDRS light box to illuminate the ETDRS eye chart during study vision testing. The light box can be mounted on the wall or placed on a stand (available at lighting stores for the blind in New York City) at a height such that the top of the third row of letters (0.8 LogMAR) on the chart is 49 + 2 inches (124.5 + 5.1 cm) from the floor. The lighting in the room should be approximately 50 foot-candles with uniform illumination between the light box and the subject. The distance between the center of the testing chair and the eye chart should be 4.0 meters.

C. Best corrected visual acuity test

The right eye is tested first, followed by the left eye. The subject is seated so that the center of the test chair is 4.0 meters from the ETDRS eye chart. This testing distance should be used initially for all subjects, even if they cannot be refractioned at this distance. In addition to the shading sheet on the test frame, the left eye is covered with an eye patch or an eye plate placed under the test frame. Using the lens correction obtained from subjective refraction in the test frame, the subject is asked to read the letters on the ETDRS eye chart 1 from top to bottom with their right eye. Emphasize to the subject that each response will be scored, so allow them ample time to read each letter for optimal recognition. Inform the subject that all characters they are required to read are letters, not numbers.

As the subject reads the chart, the examiner notes the letters they correctly identify and circles the corresponding letters on the ETDRS score sheet (or study record sheet). Letters read incorrectly or for which no guess was made are not marked on the record sheet. Each letter read correctly is scored one point. After the 4-meter distance test is completed, the score for each row (including zero if no letter was read correctly in that row) and the total score for each eye are recorded on the record sheet.

If the number of correct letters read at a distance of 4 meters is less than 20, the test should be repeated at a distance of 1 meter, and the scores at both the 4-meter and 1-meter distances should be recorded in the ETDRS scorebook (or study record sheet). Both eyes of the subject should first be tested at a distance of 4 meters, and then the subject should be moved to the 1-meter distance for testing. It is highly recommended to count the total number of correct letters read at a distance of 4 meters immediately after the 4-meter distance test is completed to determine the subjects who need to be tested at a distance of 1 meter. Before the actual test at the 1-meter distance is carried out, a range of +0.75D should be added to the correction in the test frame to compensate for the new distance. The subject should be tested in a seated position when testing at a distance of 1 meter.

Test the left eye in the same way as the right eye, but use Chart 2. The left eye should never see Chart 1, and the right eye should never see Chart 2, even when the two charts are swapped or the shade is changed.

Amblyopia detection (light perception test)

If the subject cannot discern any letters on the eye chart (i.e., score = 0), light perception testing should be performed using an indirect ophthalmoscope as the light source. This testing process can be performed using the researcher's routine method as follows:

Maintain room lighting at a level normal for visual acuity testing. Remove the test frame, and the subject should close the contralateral eye tightly and completely cover the periorbital area and the bridge of the nose with the palm of their hand. The indirect ophthalmoscope should be focused at a distance of 3 feet, and the rheostat set to 6 volts. Direct light into and out of the eye at least four times from a distance of 3 feet; the subject should be asked to respond when they see the light. If the examiner is confident that the subject can perceive light, their vision should be recorded as light perception; otherwise, their vision should be recorded as no light perception.

Calculating vision scores

After each visual acuity test, the visual acuity score for that visit was calculated. The visual acuity score was determined by the number of letters read correctly, as follows:

If 20 or more letters are read correctly at a distance of 4 meters, the visual acuity score is equal to the number of letters read correctly at a distance of 4 meters (N) + 30. If more than one letter is read correctly but less than 20 letters are read correctly at a distance of 4 meters, the visual acuity score is equal to the number of letters read correctly at a distance of 4 meters plus the number of letters read correctly in the first 6 lines at a distance of 1 meter.

If no letters are read correctly at both the 4-meter and 1-meter distance tests, the visual acuity score is recorded as 0, and the light perception test is performed as described above.

Visual acuity is scored using the ETDRS chart as follows, with each eye scored separately, at a distance of 4 meters:

OK Sharpness Read the correct letters 1(top) 20/200 2 20/160 3 20/125 4 20/100 5 20/80 6 20/63 7 20/50 8 20/40 9 20/32 10 20/25 11 20/20 12 20/16 13 20/12.5 14 20/10 Total number of letters

If the total number of letters read correctly is 20 or more, the score is equal to the total number of letters read correctly plus 30. If the total number of letters read correctly is less than 20, the patient should be retested using the same eye chart at a distance of 1 meter, and the score should be recorded.

OK Sharpness Read the correct letters 1(top) 20/800 2 20/640 3 20/500 4 20/400 5 20/320 6 20/250 Correct total

The visual acuity letter score is equal to the number of letters read correctly at a distance of 4.0 meters plus the number of letters read correctly in the first 6 lines at a distance of 1.0 meters.

If 20/20 is used as the standard for normal vision, then the visual acuity measurement, such as "20/20", can also be expressed as a percentage of normal vision according to the following table:

The effectiveness of the treatment can then be expressed as an increase in the number of correct letters read, which can be easily converted to the number of lines measured on the EVA test or the ETDRS chart, or effectiveness can be expressed as the achievement of a certain percentage of normal vision.

For example, treatment with this device can improve visual acuity by at least 15 letters on the ETDRS chart or EVA test. Fifteen letters correspond to three rows on the ETDRS chart. If a patient has low vision of 20/100, treatment with this method can improve their vision to 20/50, which is 76% of normal vision, or near normal vision.

Depending on the patient's original visual acuity and the efficacy of a specific course of treatment, treatment with this device can increase visual acuity by at least 18 letters, at least 21 letters, at least 24 letters, at least 27 letters, at least 30 letters, or at least 33 letters on the ETDRS chart or EVA test.

Based on the above in vitro results, as well as the examples regarding performance in a forced-choice visual discrimination task, accuracy of a Bayesian stimulus reconstruction test, and performance in error type detection, and the in vivo test results described in the examples, treatment with the method of the present invention can improve visual acuity to 34%, 41%, 45%, 49%, 53%, 58%, 64%, 70%, 76%, 80%, 84%, 87%, 91%, 96% and 100% of normal vision.

Objective electrophysiological testing in humans may include the following:

One test is the flash visual evoked response (VEP), in which improved performance consists of a change from no response to a response. The presence of a response is an objective indicator of visual signals reaching the brain (Chiappa 1997). This test provides only a crude assessment of visual function; it indicates that signals are reaching the brain but does not provide information about resolution.

Based on the above in vitro results, as well as the examples regarding performance in a forced-choice visual discrimination task, accuracy in a Bayesian stimulus reconstruction test, and performance in error type detection, as well as the in vivo test results described in the examples, treatment with the device can achieve positive results in flash visual evoked responses.

The second test focuses on pattern signals, or pattern VEPs, which can be elicited by either transient or steady-state stimulation. Improvement is manifested by: (a) a change from no response to a response, or (b) a decrease in the minimum check size by a factor of 2 or more that elicits a detectable response, or (c) a 2-fold or greater increase in spatial frequency that elicits a detectable response. (a) is an objective indicator of visual signal arrival at the brain, as in the flash VEP test described above. (b) and (c) are objective indicators of a 2-fold improvement in visual resolution (acuity), thus indicating improved visual function and light sensitivity. Although the VEP is a standard clinical test, our application differs from its clinical use, where latency is the primary metric (Chiappa 1997). Our goal was to use this test to assess visual acuity, not conduction delay. By measuring the VEP as a function of checkerboard size or grating spatial frequency, visual acuity can be assessed (Bach et al. 2008).

Based on the above in vitro results, as well as the examples regarding performance in a forced-choice visual discrimination task, accuracy in a Bayesian stimulus reconstruction test, and performance in error type detection, and the in vivo test results described in the examples, treatment with the device can achieve the following results in a pattern VEP test: (a) a change from no response to a response, or (b) a reduction in the minimum detection size by 1/2 or more, which causes a detectable response, and (c) an increase in spatial frequency by 2 times or more, which causes a detectable response.

Scanning VEP, where the improvement consists of: (a) an increase in spatial frequency of 2 or more that elicits a detectable response, or (b) a decrease in minimum contrast of 1/2 or more that elicits a detectable response. (a) The principle of acuity measurement is the same as that of pattern VEP described above; (b) it is an objective measure of contrast sensitivity (grayscale discrimination), which is also related to visual function. Scanning VEP is a reliable and objective method for assessing acuity (Norcia and Tyler 1985) and contrast sensitivity (Norcia AM et al. 1989).

Based on the above in vitro results, as well as the examples regarding performance in a forced-choice visual discrimination task, accuracy in a Bayesian stimulus reconstruction test, and performance in error type detection, and the in vivo test results described in the examples, treatment with the device can achieve the following results in scanning VEP: (a) an increase in spatial frequency of 2 times or more, which elicits a detectable response, or (b) a reduction in minimum contrast of 1/2 or more, which elicits a detectable response.

For the above tests, the acuity criterion of "2-fold" was chosen because it is approximately equivalent to an improvement of 3 lines on the standard Snellen or ETDRS visual acuity chart (e.g., from 20/400 to 20/200), a change that is generally considered statistically and functionally significant. Similarly, the contrast sensitivity criterion of "2-fold" was chosen because it is approximately equivalent to an improvement of 2 steps on the standard Pelli-Robson contrast sensitivity chart (Pelli DG et al. 1988), an improvement that is also considered statistically and functionally significant.

Other methods for measuring clinical efficacy are known in the art (Maguire et al. 2008). Objective assessments include, but are not limited to, pupillary light reflex (PLR), full-field electroretinography (ERG) (including bilateral full-field ERG), and nystagmus testing. Analysis should follow the International Society for Clinical Electrophysiology of Vision standard guidelines. Pupillary responses should be recorded simultaneously from both eyes (Kawasaki et al. 1995). Nystagmus testing can be performed qualitatively and quantitatively by analyzing motion paths in video recordings at baseline and at specific time points after treatment. Interpupillary distance can be measured directly from video frames. Objective measures include, but are not limited to, standard visual acuity (VA) tests, dynamic visual fields, and locomotion tests that assess the subject's ability to navigate an obstacle course. For locomotion tests, a different maze can be used for each session, and the number of obstacles avoided or encountered, the number of landmarks identified, and the time spent in the maze can be assessed (Simonelli et al. 2010).

The examples are provided to illustrate, not to limit, the invention as claimed.

Example 1 Method for constructing an encoder

Encoder constructed using Linear-Nonlinear-Poisson (LNP) cascade

The encoder parameters were constructed based on responses to two stimulus sets: binary white noise (WN) and a grayscale natural scene image recorded in New York's Central Park (NS). Both stimuli were presented at a 15 Hz frame rate and had the same mean luminance (0.24 μW/cm ² at the retina) and contrast (root mean square (RMS) contrast of 0.087 μW/cm ² ). For the preprocessing step, we chose a = 0 and b = 255/0.48 μW/cm ² to map the visual stimuli to a value range of 0-255 (see the "Encoder" section above).

To determine the spatiotemporal transformations, we used the same linear-nonlinear model described in the previous section (also see Victor and Shapley 1979; Paninski et al. 2007; Pillow et al. 2008; Nirenberg et al. 2010). Model parameters were determined by maximum likelihood, which is the probability that the model will produce the experimentally observed spike train evoked by the stimulus, as shown in Nirenberg et al. 2010; similar methods are described in Paninski et al. 2007; Pillow et al. 2008. Maximum likelihood optimization is common knowledge in the art.

In the example data below, neurons are modeled independently. For each neuron m, the firing rate λ _m is determined according to Equation 1. The linear filter for each neuron is assumed to be the product of a spatial function (a 10×10 pixel array, centered on the receptive field) and a temporal function (18 time bins, each 67 ms, for a total duration of 1.2 sec). As described in Nirenberg et al. (2010), following Pillow et al. (2008), dimensionality is reduced by assuming the temporal function is the sum of 10 spikes and a basis function (a raised cosine in logarithmic time).

The nonlinearity is parameterized using a cubic spline function with 7 knots. The knots are spaced to cover the value range obtained from the linear filter output of the encoder.

As previously described, parameters were fitted using standard optimization procedures, as described in Pillow et al., 2008, Paniniski et al., 2007, and Nireberg et al. As shown in Equation 2, the quantity maximized is the log-similarity of the spike train observed under the model. Because each neuron is independent, the optimization of each neuron's parameters can be performed independently. To maximize the log-similarity, we used the same procedure described above following Equation 2, which we briefly reiterate here: we initially assumed that the nonlinearity N was exponential, since in this case, the log-similarity Z has no local maxima (Paninski et al., 2007). After optimizing the linear filter and the exponential nonlinearity (via coordinate ascent), the nonlinearity was replaced by a spline. The final encoder parameters were determined by alternating stages of maximizing the log-similarity with respect to (i) the spline parameters and (ii) the filter parameters until a maximum was reached, as described in (Nirenberg et al., 2010), which also discusses the rationale for this approach.

Models that take into account historical dependencies and correlations were also constructed. The configuration of interneuronal correlations was constructed using a coupling kernel, following the method of (Pillow et al. 2008).

For the spike generation step, for each cell m, we created a nonhomogeneous Poisson process with instantaneous firing frequency λm. We considered a time interval (block) length Δt = 0.67 ms.

It is worth noting that Figures 3-6, 13, and 14 compare the performance of the encoders with real cells. Figures 7-9 also compare the performance of the encoders when used in conjunction with sensors. For these experiments, the output of the encoders passed through an interface that generated light pulses to drive ChR2 in ganglion cells. Two methods were used to obtain the output from the encoders and generate the light pulses that drive ChR2. In the first method, the output of the encoders was used to control an LCD panel (Panasonic PT-L104, Panasonic, Secaucus, NJ). The LCD panel was placed in front of a set of seven high-intensity blue LEDs (Cree XP-E Blue, Cree, Durham NC). The squares of the LCD panel delivered the encoder output to a given area of ganglion cells. For each frame, if the encoder commanded the ganglion cells to spike in that frame, the squares were set to the highest intensity (255). If the ganglion cells were not expected to spike in that frame, the squares were set to the lowest intensity (0). If the LCD panel intensity was locally strong (255), the light from the blue LEDs was allowed to pass through, otherwise the light was blocked. The output of the LCD panel was focused onto the retina. The light intensity at 255 positions on the retina was 0.5 mW/ ^mm² . Each frame lasted 16.7 ms. Another approach is used when precise spike timing is required. In this approach, an LED (Cree XP-E Blue, Cree, Durham, NC) directly drives the ganglion cells. The LED output state is controlled by a computer-generated 5V TTL pulse, which is transmitted through a control/amplification circuit described by Campagnola et al. 2008. When the TTL pulse is high (5V), the LED turns on, and when the pulse is low (0V), the LED turns off. The encoder output is used to drive the TTL pulse via a computer parallel interface using client software. When the encoder specifies that a spike should occur, the TTL pulse is driven high (5V) for 1 ms and then turned off again. In the on state, the LED pulse intensity on the retina is 1 mW/ ^mm² .

Sensor-only responses to visual stimulation (Figures 8C, 9D) were recorded using two methods. For natural images (Figure 8C), ganglion cells were driven with LEDs, which were in turn controlled by TTL pulses. Using a pulse code module, the LED output was set to match the intensity of the natural image at the location of the ganglion cell's receptive field. The TTL pulse width was 1 ms. A linear scale was used between intensity and pulse frequency, with more pulses within a frame indicating brighter intensity and fewer pulses indicating darker intensity. The LED pulse frequency was updated every 66.7 ms to match the intensity of the natural image for the corresponding frame. The highest intensity of the image was mapped to the encoder's peak firing rate for that specific ganglion cell—typically between 8 and 12 pulses per 66.7 ms frame. For infant face responses (Figure 9D), ganglion cells were driven with an LCD panel. The brightness of the LCD panel (0-255) was set to match the intensity of the infant face image (0-255) at a given location in the ganglion cell's receptive field. As mentioned in the previous section, the LED intensity at the retina is 1 mW/mm ² and the LCD intensity at maximum brightness is 0.5 mW/mm ² .

Example 2 - Determination of time-space conversion parameters

We describe the procedure for determining the parameters of the spatiotemporal transformation. The parameters discussed are the same as in the previous section "Encoder".

In this example, first, an experiment was conducted and ganglion cell responses to WN and NS stimulation were collected (see "Example 1 - Method for Constructing an Encoder" for the stimulation example). Next, the negative correlation between the number of ganglion cell action potentials and the stimulus intensity was calculated to determine the initial setting value of the linear filter _Lm . Further down, it was assumed that the linear filter was separable and was the product of a spatial function and a temporal function. The spatial function was parameterized as a 10×10 grid weight, and the temporal function was parameterized as the sum of 10 weighted temporal basis functions. At this stage, the nonlinearity _Nm was assumed to be an exponential function to ensure the absence of local maxima. Next, the similarity of this set of parameters was calculated for the responses of the ganglion cells to the specific stimulation and recording. The next step was to determine the optimal parameters for the spatial function, the temporal function, and the exponential nonlinearity by maximizing the similarity of these parameters using gradient ascent (for detailed descriptions, see: Paninski et al., 2007, Pillow et al., 2008, Nirenberg et al., 2010). After these parameters were optimized, the exponential nonlinearity was replaced by a 7-knot cubic spline, which was able to more accurately describe the cell's response. Then, the spline parameters are optimized to maximize the similarity. Subsequently, the parameters of the spatial and temporal functions are optimized to maximize the similarity given the new spline parameters. These two steps (optimizing the spline parameters while holding the spatial and temporal functions constant, then optimizing the spatial and temporal functions while holding the spline parameters constant) are repeated until the change in similarity between the two steps is less than an arbitrarily chosen small number.

Example 3 Comparison of the amount of information carried by virtual retinal cells and real retinal cells

To construct the dataset, we recorded the responses of several hundred mouse retinal ganglion cells (515 cells) to a wide range of stimuli, including natural and artificial stimuli, which for this experiment were checkered patterns, natural scenes, and drifting gratings. For each cell, we constructed its encoder (also known as its virtual retinal cell or model cell). This was implemented as follows. WN (white noise) and NS (natural or naturalistic stimuli) were presented to the retina and the responses of the ganglion cells were recorded, parameterizing the stimulus/response relationship for each cell as described above. The encoder was then tested with additional natural scenes and drifting grating responses. In this way, all tests were performed with novel stimuli.

Figure 3 shows the results of this information analysis. We recorded from hundreds of ganglion cells and simulated their responses. We then presented a large array of stimuli—not the stimuli used to construct the encoders—to both model and real cells. We calculated the amount of information about the stimulus carried by each virtual cell and compared it with the information carried by the corresponding real cell. As shown, the virtual cells carried almost all the information carried by the real cells. To ensure sufficient data for each analysis, each stimulus set was presented to at least 100 real cells. We then evaluated the reliability of the results through multiple calculations. As expected, the amount of information carried by the real cells increased with increasing temporal resolution, and as shown in the figure, the information carried by the virtual cells followed closely behind.

Example 4 Comparison of the Information Quality Carried by Virtual Retinal Cells and Real Retinal Cells

Figure 4 shows that the quality of information carried by virtual cells and real cells is also the same. For each cell in Figure 4, we compared the post-stimulus distribution generated by the virtual cell's response with the post-stimulus distribution generated by the real cell's response. Figure 4A shows several examples, and Figure 4B shows a histogram of the results for all cells in the dataset.

To understand what the matrices show, we detail one of them—the one in the upper left corner of Figure 4-1, Panel A. The vertical axis represents the presented stimulus, while the horizontal axis represents the "decoded" stimulus (i.e., the poststimulus distribution). In the top row, there is a single bright square located at the far left. This means that when the presented stimulus is the lowest temporal frequency grating, the decoded stimulus will be correct—i.e., the poststimulus distribution is sharp (represented by the single bright square) and the peak is located at the correct position (the position corresponding to the lowest temporal frequency). In contrast, in the bottom row of the matrix, there is no single bright square; instead, there is a stretch of red squares located to the right of the row. This means that when the presented stimulus is the highest temporal frequency grating, the decoding may be short—the poststimulus is wide and provides only limited information about the stimulus. This suggests that the stimulus may be a high-frequency grating, but it does not indicate what specific high frequency it is.

The significance of this graph is twofold. First, it demonstrates that real cells exhibit multiple and diverse afterstimuli (e.g., there are many types of ganglion cells with varying visual responses and sensitivities to stimuli). Second, the virtual cells accurately replicate these afterstimuli, with some providing information about low frequencies, others about high frequencies, or displaying complex patterns. In almost all cases, over hundreds of cells, the encoders captured the behavior of the real cells. The afterstimuli generated by each virtual cell closely matched those produced by the real cell. This provides strong evidence that the virtual cells can function as surrogates for real cells.

Example 5 Predicting the response of retinal ganglion cells by encoder

We used the encoder to make a series of predictions about the behavior of ganglion cells and tested them. In this case, the predictions focused on comparing the differences in how ON cells and OFF cells signal movement, particularly in slow motion.

It is implemented as follows: First, a cell encoder is constructed. As described above, WN and NS stimuli are presented to the wt (wild type) retina, the responses of the ganglion cells are recorded and the stimulus/response relationship is parameterized. The ganglion cell population includes ON and OFF cells. Parameters are generated for ON and OFF cells respectively, and these parameters are used to generate ON and OFF encoders. A visual discrimination task is implemented to make predictions. We provide different sensors whose drifting gratings change at a temporal frequency and obtain responses. We then decode the responses (using Bayesian (i.e., maximum likelihood) as described in this article). In each trial of this task, we focus on the most likely frequency of the grating given the response. We then calculate the fraction of the number of correct answers obtained across all trials. In order to make specific predictions for ON cells and OFF cells, we implement this task using cell populations consisting of only ON cells or only OFF cells. Because ganglion cells are known to behave differently under scotopic (nighttime light) and photopic (daytime light) conditions, we also implemented this task using encoders whose parameter values have been determined under these two conditions (Purpura et al. 1990; Troy et al. 2005; Troy et al. 1999).

Several results quickly emerged. The first was that, under scotopic conditions, ON cells discriminated lower temporal frequencies (slow motion) better than OFF cells. The second was that, also under scotopic conditions, OFF cells discriminated higher temporal frequencies better than ON cells. The third was that these differences existed only under scotopic conditions: both cells performed almost equally well under photopic conditions. The final result was that both cells performed well only over a narrow range of frequencies under scotopic conditions but over a wide range under photopic conditions.

The predictions were then tested. First, electrophysiological tests were performed. Identical stimuli were presented to the retina on a multi-electrode array, and the responses of the ganglion cells were recorded. These responses were then decoded using the same method used to decode the responses of the virtual cells: the maximum likelihood method. As shown in Figure 5, the predictions made by the real cells were identical to those made by the virtual cells, demonstrating that the virtual cells could be used as surrogates for real cells in baseline experiments.

Example 6 Predicting Animal Behavior by Encoder

Finally, we made behavioral predictions. To this end, we used an optokinetic response task because it is a) simple, b) easily quantifiable, and c) allows us to selectively probe a single cell type, ON cells (only ON cells project to the accessory visual system (AOS), which drives this behavior) (Dann and Buhl 1987; Giolli et al. 2005). In this task, experimental animals (wt mice) are presented with a drifting grating, which they either track or fail to track. To predict behavior, we focused on the presence or absence of the grating in both the encoder and the animal. We used the same method used to test predictions in electrophysiological experiments—i.e., decoding the response data using the maximum likelihood method. The only difference was that, in contrast to behavior, the encoder responses were decoded into only two conditions (grating present vs. grating absent) because this corresponds to the choice in the behavioral task. Finally, the animals and encoders were presented with stimuli representing either photopic conditions (daylight exposure) or scotopic conditions (nightlight exposure) and contrast sensitivity was measured, defined as the contrast at which 75% of the stimuli were correctly decoded, which meets the criteria of binary forced-choice psychology. As shown in Figure 6, the encoder successfully predicted the shift in visuokinetic behavior.

Example 7 Retinal Ganglion Cell Discharge Types Generated by Encoders

We presented natural scene images to three groups of animals and recorded the responses of ganglion cells from the retinas of the three groups of animals: a) normal animals (briefly: retinas from wild-type (WT) mice; natural scene images were presented to the retinas, and the firing patterns of the ganglion cells were recorded) (Figure 7, upper panel), b) blind animals treated with retinal prostheses (briefly: retinas from commercially available double-transgenic mice that have retinal degeneration but whose retinal ganglion cells still express channelrhodopsin-2; natural scene images were presented to the retinas, and the firing patterns of the ganglion cells were recorded) (Figure 7, middle panel), and c) blind animals treated with current optogenetic prosthesis methods (briefly: retinas from the same double-transgenic mice as above; natural scene images (uncoded) were presented to the retinas, and the firing patterns of the ganglion cells were recorded) (Figure 7, lower panel).

In the normal retina, images are converted by the retinal circuitry into action potential patterns (also called spike trains). Figure 7, top, shows a spike train from a normal retina. In the retina of a blind animal treated using the encoder/sensor approach, images are converted by the encoder/sensor into spike trains ( Figure 7, middle). As shown, the spike trains produced by this approach closely match those produced by normal ganglion cells. This is because the encoders closely reproduce the ganglion cell spike trains, and ChR2 has sufficiently fast kinetic responses to the encoder outputs. Therefore, the input/output relationship of the normal retina is mimicked. In contrast, Figure 7, bottom, shows the output of a standard optogenetic approach (which uses only the sensor, i.e., only ChR2, as described in Lagali et al. 2008; Tomita et al. 2010; Bi A et al. 2006; Zhang et al. 2009; Thyagarajan et al. 2010). In this case, the stimulus (an image of a natural scene) directly activates ChR2. While this approach can cause ganglion cells to fire, the resulting firing pattern is not typical.

Example 8 Performance of Encoders and Retinal Prostheses

We evaluated the performance of the encoder and prosthesis using three approaches: a discrimination task (Figure 8), image reconstruction (Figure 9), and a behavioral task (visuomotor) (Figure 10). The testing methods and results are as follows.

Performance in visual discrimination tasks

First, a discrimination task was performed. Briefly, an array of stimuli was presented, and then the degree to which they could be distinguished from each other based on the responses of ganglion cells (or encoders) was measured. For ganglion cell recordings, stimuli were presented on a computer monitor, and the ganglion cell responses were recorded using a multielectrode array as described by Pandarinath et al., 2010.

To decode a response in the test set, we need to determine which stimulus s _j is most likely to produce that response. Specifically, we need to determine the stimulus s _j for which p(r|s _j ) is maximized. According to Bayes' theorem, p(s _j |r) = p(r|s _j )p(s _j )/p(r), where p(s _j |r) is the probability of stimulus s _j occurring, given a specific response r; p(r|s _j ) is the probability of obtaining a specific response r for a given stimulus s _j ; and p(s _j ) is the probability of stimulus s _j occurring. In this experiment, p(s _j ) was set equal for all stimuli. Bayes' theorem shows that p(s|r j ) is maximized when p(r|s _j ) is maximized. When p(s _j ) is consistent, as in this experiment, the method for obtaining the most likely stimulus for a given response is referred to as maximum likelihood decoding ( _Kass et al. 2005; Pandarinath et al. 2010; Jacobs et al. 2009). For each presentation of stimulus _si such that response r is decoded as _sj , the entry at position (i, j) in the confusion matrix is incremented.

The following procedure is performed to establish the response distribution required for the decoding calculations used to form the confusion matrix (i.e., specifying p(r|s _j ) for any response r). Response r is defined as a spike train spanning 1.33 seconds after stimulus onset, with bins of 66.7 ms. Because the spike generation process is assumed to be a non-homogeneous Poisson process, the probability p(r|s _j ) for the response over the entire 1.33 s is calculated by multiplying the probabilities of each 66.7 ms bin. The probability of assignment to each bin is determined according to Poisson statistics, based on the average training set responses for stimulus s _j in that bin. Specifically, if the number of spikes in response r in that bin is n, and the average number of spikes in the training set responses in that bin is h, then the probability of assignment to that bin is ( ^hn /n!)exp(-h). The product of the probabilities of each bin determines the response distribution used for the decoding calculations to form the confusion matrix. These results, similar to those shown in FIG8 , were obtained with a range of block sizes (50 to 100 ms) and random assignments of the training and test sets.

After the confusion matrix is calculated, the overall performance of the forced choice visual discrimination task is quantified by the "correct score", which is the score of the number of responses that correctly identify the decoding of the stimulus during the total task. The correct score is the average value of the diagonal of the confusion matrix. In this program, 4 sets were analyzed. For each of them, the response from the WT retina was used as the training set, and the different response sets were combined as the test. Four sets were generated.

(1) The first set consisted of responses from WT retinas. The fraction correct was obtained using responses from normal ganglion cells.

(2) The second set consists of responses from the encoders (the responses from the encoders, as described in this document, are streams of electrical pulses, in this case spanning 1.33 seconds after the stimulus is present and in chunks of 66.7 milliseconds, which are the responses of WT ganglion cells). Given the distribution of responses from a normal WT retina, when the responses from the encoders are used as the test set, a measure of the performance of the encoders can be obtained. In other words, we are basing this on the assumption that the brain can interpret the responses of a normal WT retina (i.e., the responses encoded normally). When the responses from the encoders are used as the test set, a measure of how well the brain works can be obtained for representative normal retinal responses (representative retinal codes).

(3) The third set consists of responses from blind animals driven by the encoder plus the sensor (ChR2 in ganglion cells), with the same duration and bin size as above. This set provides a test of how the encoder performs after the output passes through the sensor in real tissue. (This experiment is not perfect because the sensor is very close to the encoder, but it still provides a test of how the complete system (encoder + sensor) performs.

(4) Finally, the last set consists of responses from blind animals driven only by the sensor (ChR2 in ganglion cells), with the same duration and bin size as above. This provides a test of how well the standard optogenetic approach performs.

Figure 8 shows these results. Figure 8A shows the confusion matrix generated when the test set was obtained from a normal WT retina. On the left is the matrix for individual ganglion cells, and on the right is the matrix for a group of 20 cells. As shown, each individual cell carries a considerable amount of information; as a group, the cells in the group can distinguish almost all stimuli in the set. The accuracy is 80%. Figure 8B shows the confusion matrix generated when the test set was obtained from the encoders (note that these encoders were constructed from the WT retina input/output relationships in Figure 8A). The accuracy is very close to that of the WT retina, at 79%. Figure 8C shows the results for the complete system (encoders and sensors). Individual cells do not carry the same amount of information, but when formed into groups, they perform very well. The accuracy is 64%. Finally, Figure 8D shows the results of the standard optogenetic method. Individual cells carry almost no information, and even as a group, the amount of information they carry is still very limited. The accuracy is 7%, close to chance. Therefore, adding an encoder, that is, adding the retinal neuron code in the present invention, even to a very small cluster of only 20 cells, can have a very large effect and significantly improve the performance of the prosthesis.

Finally, to summarize the data, the performance percentages of the encoder-only approach (Figure 8B), the encoder-and-sensor approach (Figure 8C), and the standard optogenetic approach (Figure 8D) were compared to the normal retina (Figure 8A). The results were as follows: the encoder-only approach achieved 98.75% of the performance of the normal retina; the performance of the complete system, that is, the encoder-sensor approach in this example, was 80% of the performance of the normal retina; and the standard approach (sensor-only approach) achieved less than 10% of the performance of the normal retina (only 8.75%).

Reconstructing stimuli from ganglion cell (or encoder) responses

Next, stimulus reconstruction was performed. Stimulus reconstruction uses a standard maximum likelihood method to determine the most likely stimulus given a set of spike trains (see Paninski, Pillow, & Lewi, 2007 for a review). Although the brain does not reconstruct the stimulus, reconstruction is still a convenient method that allows comparison of prosthetic methods and gives an approximate estimate of the likelihood of visual restoration with various methods.

The stimulus consists of a full gray screen for 1 second, followed by a given image for 1 second, preferably a face. It should be noted that each pixel of the stimulus must span a reasonable area of visual space so that the features of the image, in this case a face, can be recognized. A criterion of 35 x 35 pixels per face is sufficient, as shown in Figure 9. This is consistent with the requirement that the spatial frequency used for facial recognition should be at least 8 cycles per face, so at least 32 pixels are required in each dimension to be sufficient for sampling (Rolls et al., 1985). In the example shown in Figure 9, which uses mice, each pixel corresponds to 2.6 degrees x 2.6 degrees of visual space. This in turn corresponds to approximately 12-20 ganglion cells in the mouse retina.

Reconstructing the stimulus consists of searching through all possible stimuli in space to determine the most likely stimulus for a given population response r. To determine the most likely stimulus for a given r, Bayes' theorem p(s|r) = p(r|s)*p(s)/r(r) is used. Since the prior stimulus probability p(s) is assumed to be constant for all s, the maximum value of p(s|r) is equal to the maximum value of p(r|s).

To determine p(r|s), we assume that the cell responses are conditionally independent. That is, we assume that p(r|s) is the product of the probabilities p( _rj |s), where p( _rj |s) is the probability that the jth cell responds with _rj for a given stimulus s. The theoretical basis for this assumption is that conditional independence is known to have a small bias and has little impact on the information carried (Nirenberg et al., 2001; Jacobs et al., 2009) and the fidelity of stimulus decoding.

To calculate p( _rm |s) for a given cell m, the response _rm is the spike train of the mth cell spanning 1 second and binned by 0.57 ms after stimulus onset. Since the spike generation process is assumed to be a non-homogeneous Poisson process, the probability of responding for the entire 1 second, p( _rm |s), is calculated by multiplying the probabilities of each bin. The probability of assigning a cell to each bin is determined using Poisson statistics based on the cell's predicted firing rate in response to stimulus s within each bin. The predicted firing rate of the cell is calculated using the quantity _λm (t;X) in Equation 1 (see the "Encoder" section under "Spatiotemporal Conversion Step"), where X is set to stimulus s and t is the bin duration. Finally, the probability of the cell population responding, p( _r |s), is calculated by multiplying the individual cell response probabilities, p(rj|s).

To determine the most likely stimulus _sj for the population response r, a standard gradient ascent technique is used. To find the stimulus _sj that maximizes the probability distribution p(r|s), and because the stimulus space is high-dimensional, gradient ascent provides an efficient method for searching in high-dimensional space. The procedure is summarized as follows: Start the search at a random point in the stimulus space _sk . Evaluate the probability distribution p(r| _sk ) for this stimulus, and calculate the slope of the probability distribution for each dimension of this stimulus. Then, modify the stimulus _sk by increasing the probability (determined by the slope of the probability distribution) to create a new stimulus sk ₊₁ . This process is repeated until the probability of the stimulus begins to increase only marginally, i.e., when p(r|s) reaches a peak. It is important to note that because the probability distribution is not strictly log-concave, local maxima are possible. To verify that this is not the case, reconstructions are performed using multiple random starting points to confirm convergence to the same peak.

To compare the performance of the prosthetic approach, three sets of responses must be reconstructed: 1) responses from the encoder, 2) responses from a blind retina where the ganglion cells are driven by the encoder plus the sensor (ChR2), and 3) responses from a blind retina where the ganglion cells are driven by the sensor alone (i.e., ChR2 alone). Reconstruction should be performed on the processing cluster of the present invention in blocks of 10 x 10 or 7 x 7 pixels.

The results are shown in Figure 9. To obtain a large enough dataset for complete reconstruction, the image was systematically moved across the area of the retina being recorded, so that responses to all parts of the image could be obtained from a single or a small number of retinas. In each figure, the responses of approximately 12,000 ganglion cells were recorded for each image. Figure 9A shows the original image. Figure 9B shows the image generated by the responses of only the encoders. It can not only identify that the image is a baby's face, but also that it is the face of a specific baby, which is a very challenging task. Figure 9C shows the image generated by the encoder/sensor responses. Although not as good as the original image, it is very close. Finally, Figure 9D shows the image generated by the responses of the standard method (i.e., only ChR2). This image is much more limited. The results in this figure once again show that the addition of retinal encoding has a very large impact on the performance.

To measure the performance differences between the methods, the images reconstructed by each method were compared with the original images. This was determined by calculating the standard Pearson correlation coefficient between the values of each pixel in the reconstructed image and the values of each pixel in the true image. For this test, a correlation coefficient of 1 indicates that all original image information is fully preserved, while a correlation coefficient of 0 indicates that the similarity between the reconstruction and the true image does not exceed chance.

The results are as follows: for the encoder alone, the correlation coefficient is 0.897; for the encoder plus sensor, the correlation coefficient is 0.762; and for the sensor alone (equivalent to the prior art), the correlation coefficient is 0.159. This is consistent with our results in the discrimination task, showing that the encoder plus sensor performs several times better than the prior art.

Performance in visuomotor tasks

Finally, we conducted a series of behavioral experiments using optokinetic tasks. The results are shown in Figure 10. Briefly, animals were presented with a drifting sinusoidal grating on a display screen, and their eye position was recorded using the ISCAN PCI pupil/corneal reflection tracking system (ISCAN, Woburn, MA). The recordings were then analyzed and the relationship between movement and stimulus movement was analyzed. Figure 10A, left column, shows baseline drift (no stimulus). Eye position drift in blind animals is similar to that observed in blind humans. Figure 10B (center column) shows results from commercially available double-knockout transgenic mice from our laboratory, which have retinal degeneration and still express channelrhodopsin-2 in their retinal ganglion cells. These mice were given a primary stimulus. This model mimics standard optogenetics. Figure 10C (right column) shows results using a retinal prosthesis. Results from commercially available double-knockout transgenic mice from our laboratory, which have retinal degeneration and still express channelrhodopsin-2 in their retinal ganglion cells, show that the primary stimulus was replaced by the encoder output. As shown in the figure, mice treated with a standard optogenetic approach failed to track the drifting grating, while mice treated with a retinal prosthesis were able to do so. When the image was translated into a code used by ganglion cells, the animals were able to track it.

Example 9: Conversion of images into light pulses

Figure 12 illustrates an exemplary encoder that converts an image into light pulses. Figure 12A shows an exemplary image of a natural scene in Central Park. Figure 12B shows the preprocessed image. The average light intensity and contrast have been rescaled to match the working range of the spatiotemporal conversion. In this exemplary image, no rescaling of the average or contrast is required. Figure 12B also shows the position of the exemplary encoder that produces the output of Figures 12C-E. Figure 12C shows the output of the spatiotemporal conversion step. The preprocessed image is convolved with the spatiotemporal kernel of the exemplary cell and the discharge frequency is generated by nonlinearization. Figure 12D shows the output of the spike potential generation step. The discharge frequency generated by the spatiotemporal conversion is passed through the spike potential generator to generate a series of electrical pulses. Figure 12E shows the light pulses corresponding to the output of the spike potential generation step.

Example 10 Example of Mouse and Monkey Retinal Ganglion Cell Encoder Parameter Sets

This example provides parameter sets for two sample encoders: a mouse encoder and a monkey encoder. The parameter sets consist of spatial parameters, temporal parameters, and nonlinear (spline) parameters. Furthermore, we provide basis functions for constructing temporal functions (see the "Encoder" section in the "Spatiotemporal Conversion Step" for a detailed description).

An exemplary set of mouse ganglion cell encoder parameters

Spatial Parameters - Each datum is the weight of a position in a 10x10 grid. Each position in the grid is separated by 2.6 degrees of visual angle. For readability, the sample weights below have been rescaled by a factor of ^10³ .

Temporal parameters - Contains 10 spatial parameters. Each number is the weight of 10 spatial basis functions (see below for details).

Time Basis Functions—Contains 10 time basis functions {F ₁ , F ₂ , …, F ₁₀ }. Each function has 18 values, where each time value determines a basis function for a given time step, with each time step being 66.7 ms apart. The first value represents the function with a lag of 66.7 ms, and the last value represents the function with a lag of 1.2 s.

It is important to note that these numbers are not model parameters; that is, they are pre-selected and not fitted to the data. _F1 through _F5 are pulses, and _F6 through _F10 are raised cosines in logarithmic time. Their values are given here for the reader's convenience.

The spline parameters—the nonlinearization—are standard cubic splines, i.e., piecewise cubic polynomials. As a standard, a spline is defined in terms of its component polynomials {P ₁ , P ₂ , …P ₆ } and knots {b ₁ , b ₂ , …b ₇ }. Each P _n is used to compute the nonlinearity between knots b _n and b _n+1 . Therefore, in this civilization, the number of polynomials p _tot = 6, resulting in p _tot + 1 = 7 knots. Each polynomial P _n is defined with four coefficients [A _n , B _n , C _n , D _n ]. For a given point x, when b _n ≤ x ≤ _{b n+1} , the value of the nonlinearity y is given by:

y=((A _n (xb _n )+B _n )(xb _n )+C _n )(xb _n )+D _n

For x values less than _b1 , n=1 in the above formula. If the x value is greater than _b7 , n=7 in the above formula.

Nodes: [-4.2105 -2.6916 -1.1727 0.3461 1.8650 3.3839 4.9027]

P ₁ =[0.2853 -0.1110 -2.9797 4.8119]

P ₂ =[-0.2420 1.1890 -1.3423 1.0298]

P ₃ =[-0.2063 0.0863 0.5947 0.8860]

P ₄ =[3.3258 -0.8538 -0.5712 1.2653]

P ₅ =[-6.3887 14.3006 19.8527 10.0815]

P ₆ =[3.2260 -14.8100 19.0790 50.8402]

Example set of monkey ganglion cell encoder parameters

Spatial Parameters - Each data point is the weight of a position in a 10x10 grid, with each position of the grid separated by 0.3 degrees of visual angle. For ease of reading, the sample weights below are rescaled by a factor of 10 ³ .

Temporal parameters - Contains 10 spatial parameters. Weights of 10 spatial basis functions for each data point (see below for details).

Time Basis Functions—Contains 10 time basis functions {F ₁ , F ₂ , …, F ₁₀ }. Each function has 30 values, where each value determines the basis function for a given time step, with each time step being 16.7 ms apart. The first value represents the function with a 16.7 ms lag, and the last value represents the function with a 0.5 s lag.

The spline parameters—the nonlinearity—are standard cubic splines, i.e., piecewise cubic polynomials. As a standard, a spline is defined in terms of its component polynomials {P ₁ , P ₂ , …P ₆ } and knots {b ₁ , b ₂ , …b ₇ }. Each P _n is used to compute the nonlinearity between knots b _n and b _n+1 . Therefore, in the present invention, the number of polynomials p _tot = 6, resulting in p _tot + 1 = 7 knots. Each polynomial P _n has 4 coefficients [A _n , B _n , C _n , D _n ]. For a given point x, when b _n ≤ x ≤ _{b n+1} , the value of the nonlinearity y is determined by the following equation:

y=((A _n (xb _n )+B _n )(xb _n )+C _n )(xb _n )+D _n

Nodes: [-7.9291 -5.9389 -3.9486 -1.9584 0.0318 2.0221 4.0123]

P ₁ =[-1.0067 3.4136 4.5376 -25.8942]

P2＝[-0.2910 -2.5970 6.1628 -11.2780]

P ₃ =[2.4072 -4.3345 -7.6326 -11.5935]

P ₄ =[-2.7537 10.0384 3.7195 -24.9763]

P ₅ =[1.6687 -6.4032 10.9543 0.4804]

P ₆ =[-1.0485 3.5605 5.2966 10.0743]

Example 11 Monkey retinal ganglion cell discharge patterns generated by encoders

Images of natural scenes were presented and responses from macaque retinal ganglion cells were recorded (in brief, retinas were harvested from monkeys; images of natural scenes were presented to the retinas, and responses from the ganglion cells were recorded) (Figure 13, top). Furthermore, the images were presented to encoders generated for these monkey ganglion cells (following the steps outlined in the "Encoder" section) (Figure 13, center).

In a normal retina, images are converted into action potential patterns, also known as spike trains, by retinal circuitry. A normal ganglion cell spike train is shown in Figure 13, top. The responses generated by the encoder closely match these responses (Figure 13, center). Thus, the input/output relationship of a normal retina can be simulated.

Example 12: Performance of Monkey Encoders in a Visual Discrimination Task

The performance of a group of monkey encoders was assessed using a discrimination task ( FIG14 ). The task was performed according to the method described in Example 8 (see “Performance in the discrimination task” section).

Two analyses were performed according to the procedure in Example 8. Each analysis used the responses of the monkey retina as the training set. The test set used two sets of responses:

(1) The first set consisted of monkey retinal responses to obtain the correct fraction generated by normal ganglion cell responses.

(2) The second set consists of the encoder responses (as noted throughout this paper, the encoder response is a stream of electrical pulses, in this case lasting 1.33 seconds after stimulus presentation, with chunks of 6.7 ms, which is similar to monkey ganglion cell responses).

When the encoder responses are used as the test set, an indicator of encoder performance is obtained for a given monkey retinal response distribution. In other words, based on the assumption that the brain can interpret the monkey retinal responses (i.e., naturally encoded responses), using the encoder responses as the test set provides an indicator of how well the brain interprets the normal retinal responses (the retinal code of the present invention) described in the present invention. Figure 14 shows the results. Figure 14A shows the confusion matrix generated when the test set was obtained from a normal monkey retina. The left side shows the matrix for individual ganglion cells, and the right side shows the matrix for a group of 10 cells. As shown in the figure, individual cells each carry a certain amount of information; as a group, cells can discriminate almost all stimuli in the set. The accuracy score is 83%. Figure 14B shows the confusion matrix generated when the test set was obtained from the encoders (these encoders are based on the input/output relationship of the monkey retina, as shown in Figure 14A). The accuracy score obtained from the encoder responses is 77%, which is very close to the accuracy score of 83% produced by normal monkey ganglion cells. In other words, it is 77/83 = 92.8% of the accuracy score produced by normal monkey ganglion cells. Therefore, the output of the encoder, the monkey retinal neural code in the present invention, closely matches the representation of the monkey retina.

Example 13 Fidelity of sensor output to encoder output

Figure 15 shows the ganglion cell responses that the encoder + sensor can generate following the encoder's high-fidelity output. The encoder is generated as described above. The stimulus, an image of an infant's face, is input into a processing device that drives the encoder and generates a code. In a double-transgenic blind mouse expressing ChR2, the code drives an LED located on the retina through an interface. Electrodes record the retinal response. Figure 15A shows the light pulses and the corresponding ganglion cell outputs. For each pair of rows, the top row shows the number of light pulses, while the bottom row shows the number of action potentials generated by the ChR2-expressing ganglion cells. The following Figure 15B is an enlarged view of the circled area in Figure 15A, which shows that the light pulses correspond one-to-one with the action potentials. As shown in the figure, the action potential can be generated immediately after the light pulse, thus, the encoder has high fidelity.

Example 14 Prosthetic Treatment

A 53-year-old male patient suffered from macular degeneration. His EVA test score was 48, meaning his vision was 20/200, thus confirming his low vision. His vision had been slowly deteriorating, and he was extremely concerned about eventual complete blindness. Medical staff discussed retinal prosthesis treatment options with the patient and decided to use the retinal prosthesis of the present invention.

A kit containing a gene therapy drug as described above and a device having a camera, a processor, and an interface were used.

To reduce the risk of an ocular immune response during treatment, the patient was given a short course of glucocorticoids and scheduled for a clinic visit at the end of the course. During this visit, the patient received gene therapy via intravitreal injection under local anesthesia. The patient was given a rAAV vector carrying the channelrhodopsin-2 cDNA with a promoter sequence that targets retinal ganglion cells.

The patient recovered and was sent home. Weekly follow-up was performed to ensure the eye was healing and to monitor viral shedding. The eye healed normally, and no viral shedding was observed.

During the fourth week, patients are fitted for the first time with the hardware that is part of their treatment: a pair of glasses that contain a processor and batteries. Each lens of the glasses is a camera that records an image; the inner surface of each lens is a light array.

Initial visual acuity testing was performed with and without the device. Without glasses, the patient's vision remained at 20/200; after wearing the device, this improved to 20/80 as measured by EVA. Weekly testing and practice with the full device were performed; by week six, the patient's vision had improved to 20/50 with glasses. The patient now has near-normal vision.

Example 15 Prosthetic Treatment

A 60-year-old female patient suffered from macular degeneration. She scored a 3-letter mark on the EVA test—her vision was 20/800, and she was determined to be legally blind. Medical staff discussed the use of a retinal prosthesis with the patient and decided to treat her with the retinal prosthesis of the present invention.

A kit containing gene therapy drugs and a device with a camera, processor, and interface were used.

To reduce the risk of an ocular immune response during treatment, the patient was given a short course of glucocorticoids and scheduled for a clinic visit at the end of the course. During this clinic visit, the patient received gene therapy via intravitreal injection under local anesthesia.

The patient recovered and was sent home. Weekly follow-up was performed to ensure smooth eye recovery and monitor viral vector dissemination. The eye healed normally, and no viral infection was found.

During the fourth week, patients are fitted for the first time with the hardware that is part of their treatment: a pair of glasses that contain a processor and batteries. Each lens of the glasses is a camera that records an image; the inner surface of each lens is a light array.

Initial vision testing was performed with and without the device. Without glasses, the patient's vision remained at 20/800; after wearing the device, this improved to 20/100, as measured by standard vision testing. Weekly testing and practice time with the full device were conducted; by week six, the patient's vision had improved to 20/40 while wearing glasses.

The scope of the present invention is not limited by the content of the above specific display and description. Those skilled in the art will recognize that the materials, configurations, structures and scales that can be used to replace the above-mentioned embodiments. Many references, including patents and various publications, are cited and discussed in the specification of the present invention, and a list of references is attached. The citation and discussion of these references are only for the purpose of making the description of the present invention clearer, and do not admit that the above-mentioned references are prior art of the present invention described in the present invention. All references cited and discussed in the specification are incorporated herein by reference in their entirety.

Although various embodiments of the invention have been described and illustrated in this application, it is easy for those skilled in the art to think of a variety of other means and/or structures to achieve the functions described in this application, and/or obtain the results described in this application, and/or one or more advantages described in this application, and each such change and/or modification is considered to be within the scope of the embodiments of the invention described in this application. More generally, it is easy for those skilled in the art to understand that all parameters, dimensions, materials, and configurations described in this application are exemplary, and the actual parameters, dimensions, materials, and/or configurations will depend on the specific application or application used for the invention teachings. Those skilled in the art can understand or be able to determine the use of experiments that do not exceed routine experimentation, as well as multiple equivalents of the specific embodiments of the invention described in this application. Therefore, it is understood that the aforementioned embodiments are presented only by way of example, and within the scope of the appended claims and their equivalents, the embodiments of the invention may be implemented in a manner other than that specifically described and required in this application. The embodiments of the invention of this application are directed to the various features, systems, articles, materials, kits, and/or methods described in this application. In addition, the scope of the invention of the present application includes any combination of two or more of these features, systems, articles, materials, kits, and/or methods, as long as these features, systems, articles, materials, kits, and/or methods are not mutually inconsistent.

The above embodiments may be implemented in any of a variety of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, which may be provided by a single computer or distributed across multiple computers.

Furthermore, it should be understood that a computer may be present in any of a variety of form factors, such as a rack-mounted computer, a desktop computer, a laptop computer, a tablet computer, etc. In addition, a computer may be embedded in a device not generally considered a computer but having suitable processing capabilities, including a personal digital assistant (PDA), a mobile phone, or any other suitable portable or fixed electronic device.

Furthermore, a computer may have one or more input and output devices. These devices may be used together with other items to present a user interface. Examples of output devices that can be used to provide a user interface include a printer, or a display screen for outputting a visual presentation, and a speaker or other sound generating device for outputting an auditory presentation. Examples of input devices that can be used for a user interface include a keyboard and a pointing device, such as a mouse, a touchpad, and a number pad. As another example, a computer may receive input information through speech recognition or other auditory modalities.

These computers can be interconnected via one or more networks in any suitable manner, including local area networks or wide area networks, such as enterprise networks, and intelligent networks (IN) or the Internet. These networks can be based on any suitable technology and can operate according to any suitable scheme, including wireless networks, wired networks, or fiber optic networks.

A computer for performing at least a portion of the functions described herein may include a memory, one or more processing units (also referred to herein as "processors"), one or more communication interfaces, one or more display units, and one or more user input devices. The memory may include any computer-readable medium and may store computer instructions (also referred to herein as "processor-executable instructions") to implement the various functions described herein. The processing unit may be used to execute these instructions. The communication interface may be connected to a wired or wireless network, a bus, or other communication method so that the computer can transmit messages to other devices and/or receive messages from other devices. A display unit may be provided, for example, to allow a user to see various information related to the execution of the instructions. User input devices may be provided, for example, to allow a user to make manual adjustments, make selections, enter data or various other information, and/or interact with the processor in any of a variety of ways during the execution of the instructions.

The various methods or processes outlined herein can be encoded into software that can be executed in one or more processors using any of a variety of operating systems or platforms. Additionally, such software can be written using any of a variety of suitable programming languages and/or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that is executed in a framework or virtual machine.

In this regard, various aspects of the invention may be embodied in a computer-readable storage medium (or multiple computer-readable storage media) (e.g., computer memory, one or more floppy disks, compact disks, optical disks, magnetic tape, flash memory, a circuit configuration in a field programmable logic gate array or other semiconductor device, or other non-transitory medium or tangible computer storage medium) encoding one or more programs that, when executed in one or more computers or other processors, can perform methods for implementing various embodiments of the invention discussed above. The computer-readable medium or media may be portable so that the one or more programs stored therein can be loaded into one or more different computers or other processors to implement various aspects of the invention discussed above.

The terms "program" or "software" as used in this application broadly refer to any type of computer code or set of computer-executable instructions that can be used to instruct a computer or other processor to implement the various aspects of the present invention discussed above. In addition, it should be understood that, according to one aspect, one or more computer programs that, when executed, can implement the methods of the present invention need not be located in a single computer or processor, but can be distributed in the form of modules among multiple different computers or processors to perform various aspects of the present invention.

Computer-executable instructions can take many forms, such as program modules, that are executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. Typically, the functionality of program modules can be combined or distributed in any manner desired for various implementations.

Furthermore, data structures can be stored in any suitable form within a computer-readable medium. To simplify the description, data structures can be shown as having fields associated with locations within the data structure. Similarly, the relationships can be implemented by storing the fields at locations assigned to a computer-readable medium, which conveys the relationships between the fields. However, any suitable mechanism can be used to establish relationships between data structure field information, including establishing relationships between data elements through the use of indicators, tags, or other mechanisms.

Furthermore, various inventive concepts may be implemented as one or more methods, of which one embodiment has been provided. The actions performed as part of a method may be arranged in any suitable manner. Thus, the order in which actions are performed in the construction of the embodiments may differ from that described, and may include running some actions simultaneously, even though the actions are shown as being sequential in the illustrative embodiments.

As used herein, a natural scene is understood to be an image of a natural environment, such as that described in Geisler WS, Visual Perception and Statistics of Attributes of Natural Scenes. Annals. Annu. Rev. Psychol. 59: 167-92 (2008). In some embodiments, a natural scene may be replaced by any suitable complex image, such as an image characterized by a spatial and/or temporal frequency power spectrum that generally conforms to the inverse square law. In some embodiments, such as when using short clips, the frequency spectrum of the complex image may deviate somewhat from the inverse square law. For example, in some embodiments, the complex image may have a power spectrum of the form of space or time 1/f^x, where f is the frequency and x is in the range of, for example, 1-3, or any range therein (e.g., 1.5-2.5, 1.75-2.25, 1.9-2.1, etc.).

A white noise image refers to a noise image whose spatial frequency power spectrum is basically flat.

As used herein, the term "light" and related terms (eg, "optical," "visual") are understood to include electromagnetic radiation both within and outside the visible spectrum, including, for example, ultraviolet and infrared radiation.

The indefinite articles "a" and "an" in this specification and claims should be understood to mean "at least one" unless explicitly indicated to the contrary.

The phrase "or" in this specification and in the claims should be understood to mean that one or both of the combined elements are, that is, in some cases, present simultaneously, in other cases, separately. Multiple elements listed with "or" should be understood to be constructed in the same manner, that is, "one or more" combined elements. In addition to the elements identified with "or", other elements may be present at will, whether or not relevant to the specific identification of those elements. Thus, as a non-limiting example, reference may be made to "A or B", which, when used in conjunction with an open-ended expression such as "comprising" may mean that, in one embodiment, only A (optionally including elements other than B), in another embodiment, only B (optionally including elements other than A), in yet another embodiment, both A and B (optionally including other elements), etc.

"Or" as used in the specification and claims of this application should be understood to have the same meaning as above. For example, when separating items in a list, "or" or "or" should be interpreted as inclusive, that is, including at least one, and also including a plurality of the listed elements, and optionally including other unlisted items. Unless the term clearly indicates the opposite meaning, such as "only one" or "just one", or when "consisting of..." is used in the claims, it means only including one quantitative element or listed elements. In general, if it is preceded by an exclusive term, such as "either of the two", "one of them", "only one of them" or "just one of them", the term "or" should only be interpreted as an exclusive selection (i.e., "one or the other, but not both") in this application. When "essentially consisting of..." is used in the claims, it means the meaning commonly used in the field of patent law.

In the claims, as well as in the foregoing description, all transitional phrases such as "comprises," "comprising," "with," "having," "containing," "involving," "having," "consisting of," and the like are to be construed as open-ended, i.e., meaning including, but not limited to, "consisting of." Only the transitional phrases "consisting of" and "consisting essentially of" shall be closed or semi-closed transitional phrases as provided in the U.S. Patent Office Manual of Patent Examining Procedure, Section 2111.03.

All definitions defined and used herein should be understood to control over dictionary definitions, definitions in documents incorporated herein by reference, and/or ordinary meanings of the defined terms.

Without departing from the spirit and scope of the present invention, those skilled in the art may make changes, modifications, or other supplementary explanations to the description of the present invention. Although certain embodiments of the present invention have been described and illustrated, those skilled in the art will readily appreciate that various changes and modifications may be made thereto without departing from the spirit and scope of the present invention. The materials and drawings mentioned in the foregoing description are for illustration only and are not intended to be limiting.

References

Ahuja A, Dorn J, Caspi A, McMahon M, Dagnelie G, Dacruz L, Stanga P, HumayunM, Greenberg R (2010) Blind subjects implanted with the Argus II retinalprothesis are able to improve performance in a spatial-motor task. Br JOphthalmol.

Arenkiel et al., In vivo light-induced activation of neural circuitry in transgenic mice expressing channelrhodopsin-2. Neuron (2007) 54(2): 205-18.

Ausubel et al., Current Protocols in Molecular Biology, John Wiley&Sons, New York, 1989

Bach,M et al(2008)Visual evoked potential-based acuity assessment innormal vision,artificially degraded vision,and in patients.Br J Ophthalmol92:396-403

Barnstable et al.,Thy-1 antigen:a ganglion cell specific marker inrodent retina. Neuroscience (1984)11(4):847-55.

Bi A, Cui J, Ma Y-P, Olshevskaya E, Pu M, Dizhoor AM, Pan Z-H (2006) Ectopicexpression of a microbial-type rhodopsin restores visual responses in mice with photoreceptor degeneration. Neuron 50:23-33.

Bomash I,Roudi Y,Nirenberg S.(2010)A virtual retina that works on foreign array of stimuli including natural scenes:A tool to simplify the problem of population coding.Society for Neuroscience.Program No.891.5.

Bookstein R et al(1990)Promoter deletion and loss of retinoblastomagene expression in human prostate carcinoma.Proc.Nati.Acad.Sci.USA 87(19):7762-7766

Busskamp V, et al(2010)Genetic reactivation of cone photoreceptorsrestores visual responses in retinitis pigmentosa.Science 329:413-417.

Cai et al. (2010)Gene delivery to mitotic and postmitotic photoreceptors via compacted DNA nanoparticles results in improved phenotypein a mouse model of retinitis pigmentosa. FASEB J. 24:1178-1191.

Campagnola L, Wang H, Zylka MJ. (2008) Fiber-coupled light-emitting diode for localized photostimulation of neurons expressing channelrhodopsin-2. Journal of Neuroscience Methods.169:27-33.

Cardin JA, et al (2010) Targeted optogenetic stimulation and recording of neurons in vivo using cell-type-specific expression of Channelrhodopsin-2. Nat Protoc 5(2):247-54.

Chader GJ, Weiland J, Humayun MS (2009) Artificial vision: needs, functioning, and testing of a retinal electronic prosthesis. Prog Brain Res175:317-332.

Chiappa,K.(1997)Evoked Responses in Clinical Medicine,Third Edition,Lippincott-Raven

Chichilnisky EJ.(2001)A simple white noise analysis of neuronal lightresponses. Network 12(2):199-213

Chopdar A,Chakravarthy U,Verma D(2003)Age related maculardegeneration.BMJ 326:485-488.

Cover T and Thomas J. (2006) Elements of Information Theory, 2nd Edition. Hoboken, NJ: Wiley

Dann JF,Buhl EH. (1987) Retinal ganglion cells projecting to the accessory optic system in the rat. J Comp Neurol 262(1):141-58.

Dedek K,et al(2008)Ganglion cell adaptability:does the coupling of horizontal cells play a role? PLoS One.3(3):e1714.

Douglas RM et al(2005)Independent visual threshold measurements in the two eyes of freely moving rats and mice using a virtual-reality optokinetic system.Vis Neurosci.22(5):677-84.

Duda RO,Hart PE(2001)Pattern Classification(2nd Edition)Wiley,NY,

Enroth-Cugell et al., (1966) The contrast sensitivity of retinalganglion cells of the cat. J Physiol 187(3):517--52.

European Patent Application No.19891976

Famulare M,Fairhall A.(2010)Feature selection in simple neurons:howcoding depends on spiking dynamics. Neural Comput 22(3):581-98

Field et al., (2007) Information processing in the primate retina:circuitry and coding. Annu Rev Neurosci 30:1--30.

Fitzgerald et al. (1994) Retinal signal transmission in Duchennemuscular dystrophy. J Clin Invest 93:2425-30.

Foley JM,Legge GE(1981)Contrast detection and near-thresholddiscrimination in human vision.Vision Res.21(7):1041-53.

Fried S,Werblin F,McMahon MJ(2006)US Pat.2006/0129207 Mimickingneural coding in retinal ganglion cells with short pulse electricalstimulation.In:(US,ed).

Friedman DS, O'Colmain BJ, B, Tomany SC, McCarty C, de Jong PTVM, Nemesure B, Mitchell P, Kempen J, Eye Diseases Prevalence Research Group (2004) Prevalence of age-related macular degeneration in the United States. ArchOphthalmol 122:564-572.

Gerding H, Benner FP, Taneri S (2007) Experimental implantation of epiretinal retina implants (EPI-RET) with an IOL-type receiver unit. J NeuralEng 4:S38-49.

Giolli RA, Blanks RHI, Lui F. (2005) The accessory optic system: basic organization with an update on connectivity, neurochemistry, and function. ProgBrain Res 151:407-40.

Golan L,Reutsky I,Farah N&Shoham S.(2009)Design and characteristics of holographic neural photo-stimulation systems,Journal of Neural Engineering6066004,(2009)

Graham-Rowe D(2009)A Brighter Future for Retinal Implants.In:Technology Review,http://www.technologyreview.com/biomedicine/23539/.Boston,MA:MIT.

Grinstead CM and Snell JL(1997)Introduction to Probability.AmericanMathematical Society; 2nd Revised edition

Grossman N,Poher V,Grubb MS,Kennedy GT,Nikolic K,McGovern B,PalminiRB,Gong Z,Drakakis EM,Neil,MAA,Dawson MD,Burrone J,Degenaar P.(2010)Multi-site optical excitation using ChR2 and micro-LED array.J.Neural Eng,7(1):1-13.

Han et al, (2009), Millisecond-Timescale Optical Control of NeuralDynamics in the Nonhuman Primate Brain, Neuron 62,191-198.

Hand DJ.(1981)Discrimination and classification.Wiley Series inProbability and Mathematical Statistics.

Huberman AD,Manu M,Koch SM,Susman MW,Lutz AB,Ullian EM,Baccus SA,Barres BA(2008)Architecture and activity-mediated refinement of axonalprojections from a mosaic of genetically identified retinal ganglioncells.Neuron.2008 Aug 14;59(3):425-38.

Huberman AD,Wei W,Elstrott J,Stafford BK,Feller MB,Barres BA(2009)Genetic Identification of an On-Off Direction-Selective Retinal Ganglion CellSubtype Reveals a Layer-Specific Subcortical Map of PosteriorMotion.Neuron.62(3):327-334.

Izhikevich EM (2007) Dynamic systems in neuroscience: the geometry of excitability and bursting. Cambridge, MA: MIT Press

Izhikevich EM(2010)Hybrid spiking models.Review.Phil.Trans.R.Soc.A(2010)368,5061–5070

Jacobs AL et al(2009), Ruling out and ruling in neural codes. Proc NatlAcad Sci U S A.106(14):5936-41.

Jeffreys,Harold(1961).The Theory of Probability.The Oxford University Press.

Kass RE,Ventura V,Brown EN.(2005)Statistical issues in the analysis of neuronal data.J Neurophysiol 94(1):8-25.

Kawasaki et al.,Variability of the relative afferent pupillarydefect. Am J Ophthalmol (1995).120:622–633.

Kay MA,Glorioso JC,Naldini L.(2001)Viral vectors for gene therapy: the heart of turning infectious agents into vehicles of therapeutics. Nat Med.7(1):33-40.Review.

Kelly S, Shire D, Chen J, Doyle P, Gingerich M, Drohan W, Theogarajan L, Cogan S, Wyatt J, Rizzo JI (2009) Realization of a 15-channel, hermetically-encased wireless subretinal prosthesis for the blind. In, pp 200--203.

Kibbel S, Harscher A, Wrobel W-G, Zrenner E, Rothermel A (2009) Design and Performance of an improved active subretinal chip. In: World Congress on Medical Physics and Biomedical Engineering, September 7-12, 2009, Munich, Germany (Kim SI, Suh TS, O, Schlegel WC, eds), pp 192-195:Springer BerlinHeidelberg.

Koilkonda RD,Hauswirth WW,Guy J. (2009) Efficient expression of self-complementary AAV in ganglion cells of the ex vivo primate retina. Mol Vis. 15:2796-802.

Kuffler SW.(1953)Discharge patterns and functional organization ofmammalian retina.J Neurophysiol 16(1):37-68.

Lagali PS,Balya D,Awatramani GB,Münch TA,Kim DS,Busskamp V,Cepko CL,Roska B(2008)Light-activated channels targeted to ON bipolar cells restorevisual function in retinal degeneration.Nat Neurosci 11:667-675.

Lesica NA et al. (2007) Adaptation to stimulus contrast and correlations during natural visual stimulation. Neuron 55(3):479-491.

Lettvin et al., (1959) What the frog's eye tells the frog's brain. Proceedings of the Institute of Radio Engineers 47(11):1940-51.

Liao et al. (2007) In vivo gene delivery in the retina using polyethylenimine. BioTechniques 2007,42:285-288.

Loewenstein JI, Montezuma SR, Rizzo JF, III (2004) Outer Retinal Degeneration: An Electronic Retinal Prosthesis as a Treatment Strategy. ArchOphthalmol 122:587-596.

Maguire et al. Safety and efficacy of gene transfer for Leber's congenital amaurosis. N Engl J Med (2008) 358:2240–2248.

Mancuso et al., (2009) Gene therapy for red-green color blindness inadult primates. Nature 461(7265):784-7.

Martin et al. 2002. Gene delivery to the eye using adeno-associatedviral vectors. Methods 28:267-275.

McGowan MH et al(1998)Characterization of the Mouse Aldose ReductaseGene and Promoter in a Lens Epithelial Cell Line. Mol Vis 1998;4:2

McLaughlin SK, Collis P, Hermonat PL, Muzyczka N. (1988) Adeno-associated virus general transduction vectors: analysis of proviral structures. J Virol. 62(6):1963-73.

Meytlis M,Bomash I,Pillow JW,Nirenberg S.(2009)Assessing theimportance of correlated firing using large populations of neurons.Society for Neuroscience.Program No.165.3.

Morgans CW, et al (2009) TRPM1 is required for the depolarizing lightresponse in retinal ON-bipolar cells. Proc Natl Acad Sci U S A 106(45):19174-8.

Nanduri D, Humayun M, Greenberg R, McMahon M, Weiland J (2008) Retinalprosthesis phosphene shape analysis. In: 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp 1785--1788. Vancouver, BC.

Nichols Z,Meytlis M,Nirenberg S.(2010)Correlations play a negligiblerole in coding white noise and natural scene stimuli in complete retinalpopulations.Submitted.

Nirenberg S(2000)Photoablation of cells expressing beta-galactosidase.Methods Mol Biol.135:475-80

Nirenberg S and Cepko,C(1993).Targeted ablation of diverse cellclasses in the nervous system in vivo.J Neurosci.13(8):3238-51.

Nirenberg S and Latham PE.(1998)Population coding in theretina.Curr.Opin.Neurobiol.8(4):488-493

Nirenberg S and Meister M.(1997).The light response of retinalganglion cells is truncated by a displaced amacrine circuit.Neuron 18:637-650

Nirenberg S et al(2001)Retinal ganglion cells act largely asindependent encoders.Nature 411(6838):698-701.

Nirenberg S et al. (2010) Heterogeneous response dynamics in retinalganglion cells: the interplay of predictive coding and adaptation. J Neurophysiol 103(6):3184-94

Norcia,AM,and Tyler,CW(1985)Spatial frequency sweep VEP:visual acuityduring the first year of life.Vision Res.25(10):1399-408

Norcia, AM, et al (1989). Measurement of spatial contrast sensitivity with the swept contrast VEP. Vision Res. 1989; 29 (5): 627-37.

Pandarinath et al(2010a)A novel mechanism for switching a neuralsystem from one state to another.Front Comput Neurosci.31;4:2.

Pandarinath et al(2010b)Symmetry breakdown in the ON and OFF pathways of the retina at night: functional implications. J Neurosci 30(30):10006-14.

Paninski L,Pillow J,Lewi J.(2007)Statistical models for neuralencoding,decoding,and optimal stimulus design.Prog Brain Res.165:493-507.

Paninski L.(2004)Maximum likelihood estimation of cascade point-process neural encoding models. Network 15(4):243-62

Panzeri S,et al(2007)Correcting for the sampling bias problem inspike train information measures.J Neurophysiol.98(3):1064-72.Review.

Pelli DG,Robson JG,&Wilkins AJ(1988)The design of a new letter chart for measuring contrast sensitivity.Clinical Vision Sciences 2,187-199

Perry VH, Silveira LC. (1988) Functional lamination in the ganglion celllayer of the macaque's retina. Neuroscience. 25(1):217-23.

Petrs-Silva et al., (2009) High-efficiency transduction of the mouseretina by tyrosine-mutant AAV serotype vectors. Mol Ther 17(3):463-71.

Petersen-Jones et al., (2009) AAV retinal transduction in a largeanimal model species: comparison of a self-complementary AAV2/5 with a single-stranded AAV2/5vector. Mol Vis 15:1835-42.

Pillow JW,Shlens J,Paninski L,Sher A,Litke AM,Chichilnisky EJ,Simoncelli EP.(2008)Spatio-temporal correlations and visual signaling in acomplete neuronal population.Nature 454(7207):995-9

Prusky GT,et al(2004)Rapid quantification of adult and developingmouse spatial vision using a virtual optomotor system.Invest Ophthalmol VisSci.45(12):4611-6.

Published PCT application WO1996013598

Published PCT application WO1998048027

Published PCT application WO2000015822

Published PCT application WO2001094605

Published PCT application WO2002082904

Published PCT application WO2003047525

Published PCT application WO2003080648

Published PCT application WO2003093479

Published PCT application WO2003104413

Published PCT application WO2005080573

Published PCT application WO2007127428

Published PCT application WO2010011404

Pun L(1969), Introduction to Optimization Practice ,ISBN 471-70233-1

Purpura K, Tranchina D, Kaplan E, Shapley RM. (1990) Light adaptation in the primate retina: analysis of changes in gain and dynamics of monkey retinalganglion cells. Vis Neurosci 4(1):75--93.

Rolls ET, Baylis GC, Leonard CM. Role of low and high spatial frequencies in the face-selective responses of neurons in the cortex in the superior temporal sulcus in the monkey. Vision Res. 1985; 25(8):1021-35.

Sambrook et al, "Molecular Cloning. A Laboratory Manual", 2d ed., ColdSpring Harbor Laboratory, New York (1989)

Sauer B. (1987) Functional expression of the cre-lox site-specific recombination system in the yeast Saccharomyces cerevisiae. Mol Cell Biol. 7(6):2087-96.

Shapley RM, Victor JD. (1981) How the contrast gain control modifies the frequency responses of cat retinal ganglion cells. J Physiol. 318:161-79.

Sharpee TO et al.(2008)On the Importance of Static Nonlinearity inEstimating Spatiotemporal Neural Filters With Natural Stimuli.J Neurophysiol99(5):2496-509

Sheridan C(2011)Gene Therapy finds its niche Nature Biotechnology 29(2):121-128

Siegert S,Scherf BG,Punta KD,Didkovsky N,Heintz N,Roska B(2009).Genetic address book for retinal cell types.Nature Neuroscience.12:1197-1204.

Simoncelli et al.(2004)Characterization of neural responses withstochastic stimuli.The cognitive neurosciences:327-38

Simonelli et al. (2010) Gene Therapy for Leber's Congenital Amaurosis Safe and Effective Through 1.5 Years After Vector Administration, Molecular Therapy 18 3,643–650.

Sinclair JR,et al(2004).Selective ablation of a class of amacrinecells alters spatial processing in the retina.J Neurosci.24(6):1459-67.

Soucy ER et al. (1998) A novel signaling pathway from rodphotoreceptors to ganglion cells in mammalian retina. Neuron 21:481-493

Stone et al., (1993) Response properties of ganglion cells in theisolated mouse retina. Vis Neurosci 10(1):31-9.

Strong SP,et al(1998)On the application of information theory toneural spike trains.Pac Symp Biocomput.621-32.

Thyagarajan S,van Wyk M,Lehmann K,S,Feng G,H(2010)Visualfunction in mice with photoreceptor degeneration and transgenic expression ofchannelrhodopsin 2 in ganglion cells.J Neurosci 30:8745-8758.

Tomita H, Sugano E, Isago H, Hiroi T, Wang Z, Ohta E, Tamai M (2010) Channelrhodopsin-2 gene transduced into retinal ganglion cells restores functional vision in genetically blind rats. Exp Eye Res 90:429-436.

Troy JB,Bohnsack DL,Chen J,Guo X,Passaglia CL.(2005)Spatiotemporalintegration of light by the cat

Troy JB,Bohnsack DL,Diller LC.(1999)Spatial properties of the cat X-cell receptive field as a function of mean light level.Vis Neurosci 16(6):1089--104.

Turchinovich et al. (2010) Non-viral siRNA delivery into the mouseretina in vivo. BMC Ophthalmology 10:25.

U.S. Patent 7149586

U.S. Patent 5,856,152

U.S. Patent 5,871,982

U.S. Patent 6533798

U.S. Patent Publication No. 20080221653

U.S. Patent Publication No. 20080249588

U.S. Patent Publication No. 20090088399

U.S. Patent Publication No. 20090326623,

U.S. Patent Publication No. 20100272688

U.S. Patent Publication No. 20070261127

Ueda et al, (1997) The mGluR6 5'upstream transgene sequence directs acell-specific and developmentally regulated expression in retinal rod and ON-type cone bipolar cells. J Neurosci. 17(9):3014-23.

van Adel et al. (2003) Delivery of ciliary neurotrophic factor vialentiviral-mediated transfer protects axotomized retinal ganglion cells for an extended period of time. Hum.Gene Ther. 14:103-115.

Victor JD, Shapley RM. (1979) The nonlinear pathway of Y ganglion cells in the cat retina. J Gen Physiol. 74(6):671-89.

Victor JD.(1987)The dynamics of the cat retinal X cell center. The Journal of Physiology 386(1):219.

Walther W,Stein U.(2000)Viral vectors for gene transfer:a review of their use in the treatment of human diseases.Drugs.60(2):249-71.Review.

Wassle H.(2004)Parallel processing in the mammalian retina.Nat RevNeurosci 5(10):747-57.

Wells et al.(2005)Optical stimulation of neural tissue in vivo.OpticsLetters 30(5):504-506,

Winter JO, Cogan SF, Rizzo JFI (2007) Retinal prostheses: current challenges and future outlook. J Biomater Sci Polym Ed 18:1031-1055.

Wright AF.(1997)Gene therapy for the eye.Br J Ophthalmol 81(8):620-623 Review.

Yonehara K,Ishikane H,Sakuta H,Shintani T,Nakamura-Yonehara K,et al.(2009)Identification of Retinal Ganglion Cells and Their Projections Involvedin Central Transmission of Information about Upward and Downward ImageMotion.PLoS ONE 4(1):e4320.

Yonehara K, Shintani T, Suzuki R, Sakuta H, Takeuchi Y, et al. (2008) Expression of SPIG1 Reveals Development of a Retinal Ganglion Cell SubtypeProjecting to the Medial Terminal Nucleus in the Mouse. PLoS ONE 3(2):e1533.

Zhang Y, Ivanova E, Bi A, Pan Z-H (2009) Ectopic expression of multiple microbial rhodopsins restores ON and OFF light responses in retinas with photoreceptor degeneration. J Neurosci 29:9186-9196.

Zrenner E,et al(2009)Subretinal Microelectrode Arrays Allow BlindRetinitis Pigmentosa Patients to Recognize Letters and Combine them toWords.BMEI'09.2nd International Conference on Biomedical Engineering andInformatics.Issue Date:17-19 Oct.2009.ISBN:978-1-4244-4132-7.Pages 1-4.

Claims (11)

1. An apparatus comprising: A receiving device for receiving stimuli; Processing apparatus for processing the stimuli via an encoder to generate encoded data; and An activation device for activating multiple retinal cells according to the encoded data; The Pearson correlation coefficient between the test input stimulus and the corresponding reconstructed stimulus is at least 0.35, wherein the reconstructed stimulus is reconstructed from the encoded data generated by the encoder in response to the test input stimulus. 2. The device according to claim 1, wherein the processing apparatus comprises: A device for converting the stimulus into a set of codes using a set of encoders; and A device for converting the code into a signal via an interface. 3. The device according to claim 2, wherein the activation device comprises a high-resolution sensor driven by a signal from the interface. 4. The device according to claim 1, wherein the Pearson correlation coefficient between the test stimulus and the corresponding reconstructed stimulus is at least 0.65, wherein the reconstructed stimulus is reconstructed from the encoded data generated from the encoder in response to the test input stimulus. 5. The device according to claim 1, wherein the processing apparatus comprises: A means for preprocessing the stimulus into multiple values; A device for converting the plurality of values into a plurality of firing frequencies of retinal ganglion cells in the retina; and A means for generating a code representing the peak potential of the discharge frequency. 6. The device according to claim 5, wherein the device further comprises means for eliminating the altered code by means of a burst of pulses. 7. The device according to claim 2, characterized in that the code is converted into an output containing multiple visible light pulses using an interface. 8. The device according to claim 1, wherein the encoder features are a set of parameters, and wherein the values of the parameters are determined using response data obtained through experiments exposing mammalian retinas to white noise and natural environmental stimuli. 9. The device of claim 8, wherein the encoder is configured such that the Pearson correlation coefficient between the test input stimulus and the corresponding reconstructed stimulus is at least 0.65, the reconstructed stimulus being reconstructed in response to the encoded data generated from the encoder by the test input stimulus. 10. The device of claim 8, wherein the encoder is configured such that the Pearson correlation coefficient between the test input stimulus and the corresponding reconstructed stimulus is at least 0.95, the reconstructed stimulus being reconstructed in response to the encoded data generated from the encoder by the test input stimulus. 11. A system for restoring or improving the vision of an individual in need, comprising: Devices for receiving stimuli; Processing apparatus, comprising: A storage medium stores a set of encoders for generating a set of codes from the stimulus, wherein the set of codes is configured to simulate the input/output transitions of individual retinal cells, respectively. At least one processor, and The storage medium for storing the code; An interface for converting the code into output, and A high-resolution light-response sensor for independently activating a plurality of separate retinal cells, wherein the high-resolution light-response sensor is coupled to the interface for activating the respective individual retinal cells according to the output, and wherein activation of the respective plurality of individual retinal cells results in a retinal ganglion cell response to a wide range of stimuli, including natural ambient white noise stimuli, the retinal ganglion cell response being substantially similar to the response of retinal ganglion cells from a normal retina to the same stimuli. The encoders are configured such that the Pearson correlation coefficient between the test input stimulus and the corresponding reconstructed stimulus is at least 0.35, wherein the reconstructed stimulus is reconstructed from encoded data generated from the encoders in response to the test input stimulus.