Temporal envelope and fine structure

Examples of sinusoidally amplitude- and frequency-modulated signals

The neural representation of stimulus envelope, ENV_n, has typically been studied using well-controlled ENV_p modulations, that is sinusoidally amplitude-modulated (AM) sounds. Cochlear filtering limits the range of AM rates encoded in individual auditory-nerve fibers. In the auditory nerve, the strength of the neural representation of AM decreases with increasing modulation rate. At the level of the cochlear nucleus, several cell types show an enhancement of ENV_n information. Multipolar cells can show band-pass tuning to AM tones with AM rates between 50 and 1000 Hz.[15][16] Some of these cells show an excellent response to the ENV_n and provide inhibitory sideband inputs to other cells in the cochlear nucleus giving a physiological correlate of comodulation masking release, a phenomenon whereby the detection of a signal in a masker is improved when the masker has correlated envelope fluctuations across frequency (see section below).[17][18]

Responses to the temporal-envelope cues of speech or other complex sounds persist up the auditory pathway, eventually to the various fields of the auditory cortex in many animals. In the Primary Auditory Cortex, responses can encode AM rates by phase-locking up to about 20–30 Hz,[19][20][21][22] while faster rates induce sustained and often tuned responses.[23][24] A topographical representation of AM rate has been demonstrated in the primary auditory cortex of awake macaques.[25] This representation is approximately perpendicular to the axis of the tonotopic gradient, consistent with an orthogonal organization of spectral and temporal features in the auditory cortex. Combining these temporal responses with the spectral selectivity of A1 neurons gives rise to the spectro-temporal receptive fields that often capture well cortical responses to complex modulated sounds.[26][27] In secondary auditory cortical fields, responses become temporally more sluggish and spectrally broader, but are still able to phase-lock to the salient features of speech and musical sounds.[28][29][30][31] Tuning to AM rates below about 64 Hz is also found in the human auditory cortex [32][33][34][35] as revealed by brain-imaging techniques (fMRI) and cortical recordings in epileptic patients (electrocorticography). This is consistent with neuropsychological studies of brain-damaged patients[36] and with the notion that the central auditory system performs some form of spectral decomposition of the ENV_p of incoming sounds. The ranges over which cortical responses encode well the temporal-envelope cues of speech have been shown to be predictive of the human ability to understand speech. In the human superior temporal gyrus (STG), an anterior-posterior spatial organization of spectro-temporal modulation tuning has been found in response to speech sounds, the posterior STG being tuned for temporally fast varying speech sounds with low spectral modulations and the anterior STG being tuned for temporally slow varying speech sounds with high spectral modulations.[37]

One unexpected aspect of phase locking in the auditory cortex has been observed in the responses elicited by complex acoustic stimuli with spectrograms that exhibit relatively slow envelopes (< 20 Hz), but that are carried by fast modulations that are as high as hundreds of Hertz. Speech and music, as well as various modulated noise stimuli have such temporal structure.[38] For these stimuli, cortical responses phase-lock to both the envelope and fine-structure induced by interactions between unresolved harmonics of the sound, thus reflecting the pitch of the sound, and exceeding the typical lower limits of cortical phase-locking to the envelopes of a few 10's of Hertz. This paradoxical relation[38][39] between the slow and fast cortical phase-locking to the carrier “fine structure” has been demonstrated both in the auditory[38] and visual[40] cortices. It has also been shown to be amply manifested in measurements of the spectro-temporal receptive fields of the primary auditory cortex giving them unexpectedly fine temporal accuracy and selectivity bordering on a 5-10 ms resolution.[38][40] The underlying causes of this phenomenon have been attributed to several possible origins, including nonlinear synaptic depression and facilitation, and/or a cortical network of thalamic excitation and cortical inhibition.[38][41][42][43] There are many functionally significant and perceptually relevant reasons for the coexistence of these two complementary dynamic response modes. They include the ability to accurately encode onsets and other rapid ‘events’ in the ENV_p of complex acoustic and other sensory signals, features that are critical for the perception of consonants (speech) and percussive sounds (music), as well as the texture of complex sounds.[38][44]

The perception of ENV_p depends on which AM rates are contained in the signal. Low rates of AM, in the 1–8 Hz range, are perceived as changes in perceived intensity, that is  loudness fluctuations (a percept that can also be evoked by frequency modulation, FM); at higher rates, AM is perceived as roughness, with the greatest roughness sensation occurring at around 70 Hz;[45] at even higher rates, AM can evoke a weak pitch percept corresponding to the modulation rate.[46] Rainstorms, crackling fire, chirping crickets or galloping horses produce "sound textures" - the collective result of many similar acoustic events - which perception is mediated by ENV_n statistics.[47][48]

The auditory detection threshold for AM as a function of AM rate, referred to as the temporal modulation transfer function (TMTF),[49] is best for AM rates in the range from 4 – 150 Hz and worsens outside that range[49][50][51] The cutoff frequency of the TMTF gives an estimate of temporal acuity (temporal resolution) for the auditory system. This cutoff frequency corresponds to a time constant of about 1 - 3 ms for the auditory system of normal-hearing humans.

Correlated envelope fluctuations across frequency in a masker can aid detection of a pure tone signal, an effect known as comodulation masking release.[18]

AM applied to a given carrier can perceptually interfere with the detection of a target AM imposed on the same carrier, an effect termed modulation masking.[52][53] Modulation-masking patterns are tuned (greater masking occurs for masking and target AMs close in modulation rate), suggesting that the human auditory system is equipped with frequency-selective channels for AM. Moreover, AM applied to spectrally remote carriers can perceptually interfere with the detection of AM on a target sound, an effect termed modulation detection interference.[54] The notion of modulation channels is also supported by the demonstration of selective adaptation effects in the modulation domain.[55][56][57] These studies show that AM detection thresholds are selectively elevated above pre-exposure thresholds when the carrier frequency and the AM rate of the adaptor are similar to those of the test tone.

Human listeners are sensitive to relatively slow "second-order" AMs cues correspond to fluctuations in the strength of AM. These cues arise from the interaction of different modulation rates, previously described as "beating" in the envelope-frequency domain. Perception of second-order AM has been interpreted as resulting from nonlinear mechanisms in the auditory pathway that produce an audible distortion component at the envelope beat frequency in the internal modulation spectrum of the sounds.[58][59][60]

Interaural time differences in the envelope provide binaural cues even at high frequencies where TFS_n cannot be used.[61]

Diagram of the common part of the envelope perception model of Torsten Dau and EPSM.

The most basic computer model of ENV processing is the leaky integrator model.[62][49] This model extracts the temporal envelope of the sound (ENV_p) via bandpass filtering, half-wave rectification (which may be followed by fast-acting amplitude compression), and lowpass filtering with a cutoff frequency between about 60 and 150 Hz. The leaky integrator is often used with a decision statistic based on either the resulting envelope power, the max/min ratio, or the crest factor. This model accounts for the loss of auditory sensitivity for AM rates higher than about 60–150 Hz for broadband noise carriers.[49] Based on the concept of frequency selectivity for AM,[53] the perception model of Torsten Dau[63] incorporates broadly tuned bandpass modulation filters (with a Q value around 1) to account for data from a broad variety of psychoacoustic tasks and particularly AM detection for noise carriers with different bandwidths, taking into account their intrinsic envelope fluctuations. This model of has been extended to account for comodulation masking release (see sections above).[64] The shapes of the modulation filters have been estimated[65] and an “envelope power spectrum model” (EPSM) based on these filters can account for AM masking patterns and AM depth discrimination.[66] The EPSM has been extended to the prediction of speech intelligibility[67] and to account for data from a broad variety of psychoacoustic tasks.[68] A physiologically based processing model simulating brainstem responses has also been developed to account for AM detection and AM masking patterns.[69]

Phase locking recorded from a neuron in the cochlear nucleus in response to a sinusoidal acoustic stimulus at the cell's best frequency (in this case 240 Hz). The stimulus was approximately 20 dB above the neuron's best frequency. The neural outputs (action potentials) are shown in the upper trace and the stimulus waveform in the lower trace.

The neural representation of temporal fine structure, TFS_n, has been studied using stimuli with well-controlled TFS_p: pure tones, harmonic complex tones, and frequency-modulated (FM) tones.

Auditory-nerve fibres are able to represent low-frequency sounds via their phase-locked discharges (i.e., TFS_n information). The upper frequency limit for phase locking is species dependent. It is about 5 kHz in the cat, 9 kHz in the barn owl and just 4 kHz in the guinea pig. We do not know the upper limit of phase locking in humans but current, indirect, estimates suggest it is about 4–5 kHz.[70] Phase locking is a direct consequence of the transduction process with an increase in probability of transduction channel opening occurring with a stretching of the stereocilia and decrease in channel opening occurring when pushed in the opposite direction. This has led some to suggest that phase locking is an epiphenomenon. The upper limit appears to be determined by a cascade of low pass filters at the level of the inner hair cell and auditory-nerve synapse.[71][72]

TFS_n information in the auditory nerve may be used to encode the (audio) frequency of low-frequency sounds, including single tones and more complex stimuli such as frequency-modulated tones or steady-state vowels (see role and applications to speech and music).

The auditory system goes to some length to preserve this TFS_n information with the presence of giant synapses (End bulbs of Held) in the ventral cochlear nucleus. These synapses contact bushy cells (Spherical and globular) and faithfully transmit (or enhance) the temporal information present in the auditory nerve fibers to higher structures in the brainstem.[73] The bushy cells project to the medial superior olive and the globular cells project to the medial nucleus of the trapezoid body (MNTB). The MNTB is also characterized by giant synapses (calyces of Held) and provides precisely timed inhibition to the lateral superior olive. The medial and lateral superior olive and MNTB are involved in the encoding of interaural time and intensity differences. There is general acceptance that the temporal information is crucial in sound localization but it is still contentious as to whether the same temporal information is used to encode the frequency of complex sounds.

Several problems remain with the idea that the TFS_n is important in the representation of the frequency components of complex sounds. The first problem is that the temporal information deteriorates as it passes through successive stages of the auditory pathway (presumably due to the low pass dendritic filtering). Therefore, the second problem is that the temporal information must be extracted at an early stage of the auditory pathway. No such stage has currently been identified although there are theories about how temporal information can be converted into rate information (see section Models of normal processing: Limitations).

It is often assumed that many perceptual capacities rely on the ability of the monaural and binaural auditory system to encode and use TFS_n cues evoked by components in sounds with frequencies below about 1–4 kHz. These capacities include discrimination of frequency,[74][4][75][76] discrimination of the fundamental frequency of harmonic sounds,[75][4][76] detection of FM at rates below 5 Hz,[77] melody recognition for sequences of pure tones and complex tones,[74][4] lateralization and localization of pure tones and complex tones,[78] and segregation of concurrent harmonic sounds (such as speech sounds).[79] It appears that TFS_n cues require correct tonotopic (place) representation to be processed optimally by the auditory system.[80] Moreover, musical pitch perception has been demonstrated for complex tones with all harmonics above 6 kHz, demonstrating that it is not entirely dependent on neural phase locking to TFS_BM (i.e., TFS_n) cues.[81]

As for FM detection, the current view assumes that in the normal auditory system, FM is encoded via TFS_n cues when the FM rate is low (<5 Hz) and when the carrier frequency is below about 4 kHz,[77][82][83][84] and via ENV_n cues when the FM is fast or when the carrier frequency is higher than 4 kHz.[77][85][86][87][84] This is supported by single-unit recordings in the low brainstem.[73] According to this view, TFS_n cues are not used to detect FM with rates above about 10 Hz because the mechanism decoding the TFS_n information is “sluggish” and cannot track rapid changes in frequency.[77] Several studies have shown that auditory sensitivity to slow FM at low carrier frequency is associated with speech identification for both normal-hearing and hearing-impaired individuals when speech reception is limited by acoustic degradations (e.g., filtering) or concurrent speech sounds.[88][89][90][91][92] This suggests that robust speech intelligibility is determined by accurate processing of TFS_n cues.

The separation of a sound into ENV_p  and TFS_p appears inspired partly by how sounds are synthesized and by the availability of a convenient way to separate an existing sound into ENV and TFS, namely the Hilbert transform. There is a risk that this view of auditory processing[93] is dominated by these physical/technical concepts, similarly to how cochlear frequency-to-place mapping was for a long time conceptualized in terms of the Fourier transform. Physiologically, there is no indication of a separation of ENV and TFS in the auditory system for stages up to the cochlear nucleus. Only at that stage does it appear that parallel pathways, potentially enhancing ENV_n or TFS_n information (or something akin to it), may be implemented through the temporal response characteristics of different cochlear nucleus cell types.[73] It may therefore be useful to better simulate cochlear nucleus cell types to understand the true concepts for parallel processing created at the level of the cochlear nucleus. These concepts may be related to separating ENV and TFS but are unlikely realized like the Hilbert transform.

A computational model of the peripheral auditory system[94][95] may be used to simulate auditory-nerve fiber responses to complex sounds such as speech, and quantify the transmission (i.e., internal representation) of ENV_n and TFS_n cues. In two simulation studies,[96][97] the mean-rate and spike-timing information was quantified at the output of such a model to characterize, respectively, the short-term rate of neural firing (ENV_n) and the level of synchronization due to phase locking (TFS_n) in response to speech sounds degraded by vocoders.[98][99] The best model predictions of vocoded-speech intelligibility were found when both ENV_n and TFS_n cues were included, providing evidence that TFS_n cues are important for intelligibility when the speech ENV_p cues are degraded.

At a more fundamental level, similar computational modeling was used to demonstrate that the functional dependence of human just-noticeable-frequency-differences on pure-tone frequency were not accounted for unless temporal information was included (notably most so for mid-high frequencies, even above the nominal cutoff in physiological phase locking).[100][101] However, a caveat of most TFS models is that optimal model performance with temporal information typically over-estimates human performance.

An alternative view is to assume that TFS_n information at the level of the auditory nerve is converted into rate-place (ENV_n) information at a later stage of the auditory system (e.g., the low brainstem). Several modelling studies proposed that the neural mechanisms for decoding TFS_n are based on correlation of the outputs of adjacent places.[102][103][104][105][106]

Amplitude modulation spectra (left) and frequency modulation spectra (right), calculated on a corpus of English or French sentences.[107]

The ENV_p plays a critical role in many aspects of auditory perception, including in the perception of speech and music.[2][7][108][109] Speech recognition is possible using cues related to the ENV_p, even in situations where the original spectral information and TFS_p are highly degraded.[110] Indeed, when the spectrally local TFS_p from one sentence is combined with the ENV_p from a second sentence, only the words of the second sentence are heard.[111] The ENV_p rates most important for speech are those below about 16 Hz, corresponding to fluctuations at the rate of syllables.[112][107][113] On the other hand, the fundamental frequency (“pitch”) contour of speech sounds is primarily conveyed via TFS_p cues,[107] although some information on the contour can be perceived via rapid envelope fluctuations corresponding to the fundamental frequency.[2] For music, slow ENV_p rates convey rhythm and tempo information, whereas more rapid rates convey the onset and offset properties of sound (attack and decay, respectively) that are important for timbre perception.[114]

The ability to accurately process TFS_p information is thought to play a role in our perception of pitch (i.e., the perceived height of sounds), an important sensation for music perception, as well as our ability to understand speech, especially in the presence of background noise.[4]

Although pitch retrieval mechanisms in the auditory system are still a matter of debate,[76][115] TFS_n information may be used to retrieve the pitch of low-frequency pure tones[75] and estimate the individual frequencies of the low-numbered (ca. 1st-8th) harmonics of a complex sound,[116] frequencies from which the fundamental frequency of the sound can be retrieved according to, e.g., pattern-matching models of pitch perception.[117] A role of TFS_n information in pitch perception of complex sounds containing intermediate harmonics (ca. 7th-16th) has also been suggested[118] and may be accounted for by temporal or spectrotemporal[119] models of pitch perception. The degraded TFS_n cues conveyed by cochlear implant devices may also be partly responsible for impaired music perception of cochlear implant recipients.[120]

TFS_p cues are thought to be important for the identification of speakers and for tone identification in tonal languages.[121] In addition, several vocoder studies have suggested that TFS_p cues contribute to the intelligibility of speech in quiet and noise.[98] Although it is difficult to isolate TFS_p from ENV_p cues,[109][122] there is evidence from studies in hearing-impaired listeners that speech perception in the presence of background noise can be partly accounted for by the ability to accurately process TFS_p,[92][99] although the ability to “listen in the dips” of fluctuating maskers does not seem to depend on periodic TFS_p cues.[123]

In infancy, behavioral AM detection thresholds[139] and forward or backward masking thresholds[139][140][141] observed in 3-month olds are similar to those observed in adults. Electrophysiological studies conducted in 1-month-old infants using 2000 Hz AM pure tones indicate some immaturity in envelope following response (EFR). Although sleeping infants and sedated adults show the same effect of modulation rate on EFR, infants’ estimates were generally poorer than adults’.[142][143] This is consistent with behavioral studies conducted with school-age children showing differences in AM detection thresholds compared to adults. Children systematically show worse AM detection thresholds than adults until 10–11 years. However, the shape of the TMTF (the cutoff) is similar to adults’ for younger children of 5 years.[144][145] Sensory versus non-sensory factors for this long maturation are still debated,[146] but the results generally appear to be more dependent on the task or on sound complexity for infants and children than for adults.[147] Regarding the development of speech ENV_p processing, vocoder studies suggest that infants as young as 3 months are able to discriminate a change in consonants when the faster ENV_p information of the syllables is preserved (< 256 Hz) but less so when only the slowest ENV_p is available (< 8 Hz).[148] Older children of 5 years show similar abilities than adults to discriminate consonant changes based on ENV_p cues (< 64 Hz).[149]

The effects of hearing loss and age on neural coding are generally believed to be smaller for slowly varying envelope responses (i.e., ENV_n) than for rapidly varying temporal fine structure (i.e., TFS_n).[150][151] Enhanced ENV_n coding following noise-induced hearing loss has been observed in peripheral auditory responses from single neurons[152] and in central evoked responses from the auditory midbrain.[153] The enhancement in ENV_n coding of narrowband sounds occurs across the full range of modulation frequencies encoded by single neurons.[154] For broadband sounds, the range of modulation frequencies encoded in impaired responses is broader than normal (extending to higher frequencies), as expected from reduced frequency selectivity associated with outer-hair-cell dysfunction.[155] The enhancement observed in neural envelope responses is consistent with enhanced auditory perception of modulations following cochlear damage, which is commonly believed to result from loss of cochlear compression that occurs with outer-hair-cell dysfunction due to age or noise overexposure.[156]  However, the influence of inner-hair-cell dysfunction (e.g., shallower response growth for mild-moderate damage and steeper growth for severe damage) can confound the effects of outer-hair-cell dysfunction on overall response growth and thus ENV_n coding.[152][157] Thus, not surprisingly the relative effects of outer-hair-cell and inner-hair-cell dysfunction have been predicted with modeling to create individual differences in speech intelligibility based on the strength of envelope coding of speech relative to noise.

For sinusoidal carriers, which have no intrinsic envelope (ENV_p) fluctuations, the TMTF is roughly flat for AM rates from 10 to 120 Hz, but increases (i.e. threshold worsens) for higher AM rates,[51][158] provided that spectral sidebands are not audible. The shape of the TMTF for sinusoidal carriers is similar for young and older people with normal audiometric thresholds, but older people tend to have higher detection thresholds overall, suggesting poorer “detection efficiency” for ENV_n cues in older people.[159][160] Provided that the carrier is fully audible, the ability to detect AM is usually not adversely affected by cochlear hearing loss and may sometimes be better than normal, for both noise carriers [161][162] and sinusoidal carriers,[158][163] perhaps because loudness recruitment (an abnormally rapid growth of loudness with increasing sound level) “magnifies” the perceived amount of AM (i.e., ENV_n cues). Consistent with this, when the AM is clearly audible, a sound with a fixed AM depth appears to fluctuate more for an impaired ear than for a normal ear. However, the ability to detect changes in AM depth can be impaired by cochlear hearing loss.[163] Speech that is processed with noise vocoder such that mainly envelope information is delivered in multiple spectral channels was also used in investigating envelope processing in hearing impairment. Here, hearing-impaired individuals could not make use of such envelope information as well as normal-hearing individuals, even after audibility factors were taken into account.[164] Additional experiments suggest that age negatively affects the binaural processing of ENV_p at least at low audio-frequencies.[165]

The perception model of ENV processing[63] that incorporates selective (bandpass) AM filters accounts for many perceptual consequences of cochlear dysfunction including enhanced sensitivity to AM for sinusoidal and noise carriers,[166][167] abnormal forward masking (the rate of recovery from forward masking being generally slower than normal for impaired listeners),[168] stronger interference effects between AM and FM [82] and enhanced temporal integration of AM.[167] The model of Torsten Dau[63] has been extended to account for the discrimination of complex AM patterns by hearing-impaired individuals and the effects of noise-reduction systems.[169] The performance of the hearing-impaired individuals was best captured when the model combined the loss of peripheral amplitude compression resulting from the loss of the active mechanism in the cochlea[166][167][168] with an increase in internal noise in the ENVn domain.[166][167][82] Phenomenological models simulating the response of the peripheral auditory system showed that impaired AM sensitivity in individuals experiencing chronic tinnitus with clinically normal audiograms could be predicted by substantial loss of  auditory-nerve fibers with low spontaneous rates and some loss of auditory-nerve fibers with high-spontaneous rates.[170]

Very few studies have systematically assessed TFS processing in infants and children. Frequency-following response (FFR), thought to reflect phase-locked neural activity, appears to be adult-like in 1-month-old infants when using a pure tone (centered at 500, 1000 or 2000 Hz) modulated at 80 Hz with a 100% of modulation depth.[142]

As for behavioral data, six-month-old infants require larger frequency transitions to detect an FM change in a 1-kHz tone compared to adults.[171] However, 4-month-old infants are able to discriminate two different FM sweeps,[172] and they are more sensitive to FM cues swept from 150 Hz to 550 Hz than at lower frequencies.[173] In school-age children, performance in detecting FM change improves between 6 and 10 years and sensitivity to low modulation rate (2 Hz) is poor until 9 years.[174]

For speech sounds, only one vocoder study has explored the ability of school age children to rely on TFSp cues to detect consonant changes, showing the same abilities for 5-years-olds than adults.[149]

Psychophysical studies have suggested that degraded TFS processing due to age and hearing loss may underlie some suprathreshold deficits, such as speech perception;[10] however, debate remains about the underlying neural correlates.[150][151] The strength of phase locking to the temporal fine structure of signals (TFS_n) in quiet listening conditions remains normal in peripheral single-neuron responses following cochlear hearing loss.[152] Although these data suggest that the fundamental ability of auditory-nerve fibers to follow the rapid fluctuations of sound remains intact following cochlear hearing loss, deficits in phase locking strength do emerge in background noise.[175] This finding, which is consistent with the common observation that listeners with cochlear hearing loss have more difficulty in noisy conditions, results from reduced cochlear frequency selectivity associated with outer-hair-cell dysfunction.[156]  Although only limited effects of age and hearing loss have been observed in terms of TFS_n coding strength of narrowband sounds, more dramatic deficits have been observed in TFS_n coding quality in response to broadband sounds, which are more relevant for everyday listening.  A dramatic loss of tonotopicity can occur following noise induced hearing loss, where auditory-nerve fibers that should be responding to mid frequencies (e.g., 2–4 kHz) have dominant TFS responses to lower frequencies (e.g., 700 Hz).[176]  Notably, the loss of tonotopicity generally occurs only for TFS_n coding but not for ENV_n coding, which is consistent with greater perceptual deficits in TFS processing.[10] This tonotopic degradation is likely to have important implications for speech perception, and can account for degraded coding of vowels following noise-induced hearing loss in which most of the cochlea responds to only the first formant, eliminating the normal tonotopic representation of the second and third formants.

Several psychophysical studies have shown that older people with normal hearing and people with sensorineural hearing loss often show impaired performance for auditory tasks that are assumed to rely on the ability of the monaural and binaural auditory system to encode and use TFS_n cues, such as: discrimination of sound frequency,[76][177][178] discrimination of the fundamental frequency of harmonic sounds,[76][177][178][179] detection of FM at rates below 5 Hz,[180][181][91] melody recognition for sequences of pure tones and complex sounds,[182] lateralization and localization of pure tones and complex tones,[78][183][165] and segregation of concurrent harmonic sounds (such as speech sounds).[79] However, it remains unclear to which extent deficits associated with hearing loss reflect poorer TFS_n processing or reduced cochlear frequency selectivity.[182]

The quality of the representation of a sound in the auditory nerve is limited by refractoriness, adaptation, saturation, and reduced synchronization (phase locking) at high frequencies, as well as by the stochastic nature of actions potentials.[184] However, the auditory nerve contains thousands of fibers. Hence, despite these limiting factors, the properties of sounds are reasonably well represented in the population nerve response over a wide range of levels[185] and audio frequencies (see Volley Theory).

The coding of temporal information in the auditory nerve can be disrupted by two main mechanisms: reduced synchrony and loss of synapses and/or auditory nerve fibers.[186] The impact of disrupted temporal coding on human auditory perception has been explored using physiologically inspired signal-processing tools. The reduction in neural synchrony has been simulated by jittering the phases of the multiple frequency components in speech,[187] although this has undesired effects in the spectral domain. The loss of auditory nerve fibers or synapses has been simulated by assuming (i) that each afferent fiber operates as a stochastic sampler of the sound waveform, with greater probability of firing for higher-intensity and sustained sound features than for lower-intensity or transient features, and (ii) that deafferentation can be modeled by reducing the number of samplers.[184] However, this also has undesired effects in the spectral domain. Both jittering and stochastic undersampling degrade the representation of the TFS_n more than the representation of the ENV_n. Both jittering and stochastic undersampling impair the recognition of speech in noisy backgrounds without degrading recognition in silence, support the argument that TFS_n is important for recognizing speech in noise.[3] Both jittering and stochastic undersampling mimic the effects of aging on speech perception.[188]

Individuals with cochlear hearing loss usually have a smaller than normal dynamic range between the level of the weakest detectable sound and the level at which sounds become uncomfortably loud.[189][190] To compress the large range of sound levels encountered in everyday life into the small dynamic range of the hearing-impaired person, hearing aids apply amplitude compression, which is also called automatic gain control (AGC). The basic principle of such compression is that the amount of amplification applied to the incoming sound progressively decreases as the input level increases. Usually, the sound is split into several frequency “channels”, and AGC is applied independently in each channel. As a result of compressing the level, AGC reduces the amount of envelope fluctuation in the input signal (ENV_p) by an amount that depends on the rate of fluctuation and the speed with which the amplification changes in response to changes in input sound level.[191][192] AGC can also change the shape of the envelope of the signal.[193] Cochlear implants are devices that electrically stimulate the auditory nerve, thereby creating the sensation of sound in a person who would otherwise be profoundly or totally deaf. The electrical dynamic range is very small,[194] so cochlear implants usually incorporate AGC prior to the signal being filtered into multiple frequency channels.[195] The channel signals are then subjected to instantaneous compression to map them into the limited dynamic range for each channel.[196]

Cochlear implants differ than hearing aids in that the entire acoustic hearing is replaced with direct electric stimulation of the auditory nerve, achieved via an electrode array placed inside the cochlea. Hence, here, other factors than device signal processing also strongly contribute to overall hearing, such as etiology, nerve health, electrode configuration and proximity to the nerve, and overall adaptation process to an entirely new mode of hearing.[197][198][199][200] Almost all information in cochlear implants is conveyed by the envelope fluctuations in the different channels. This is sufficient to give reasonable perception of speech in quiet, but not in noisy or reverberant conditions.[201][202][203][204][121][110][205][206][207][208] The processing in cochlear implants is such that the TFSp is discarded in favor of fixed-rate pulse trains amplitude-modulated by the ENVp within each frequency band. Implant users are sensitive to these ENVp modulations, but performance varies across stimulation site, stimulation level, and across individuals.[209][210] The TMTF shows a low-pass filter shape similar to that observed in normal-hearing listeners.[210][211][212] Voice pitch or musical pitch information, conveyed primarily via weak periodicity cues in the ENVp, results in a pitch sensation that is not salient enough to support music perception,[213][214] talker sex identification,[215][216] lexical tones,[217][218] or prosodic cues.[219][220][221] Listeners with cochlear implants are susceptible to interference in the modulation domain[222][223] which likely contributes to difficulties listening in noise.

Hearing aids usually process sounds by filtering them into multiple frequency channels and applying AGC in each channel. Other signal processing in hearing aids, such as noise reduction, also involves filtering the input into multiple channels.[224] The filtering into channels can affect the TFS_p of sounds depending on characteristics such as the phase response and group delay of the filters. However, such effects are usually small. Cochlear implants also filter the input signal into frequency channels. Usually, the ENV_p of the signal in each channel is transmitted to the implanted electrodes in the form an electrical pulses of fixed rate that are modulated in amplitude or duration. Information about TFS_p is discarded. This is justified by the observation that people with cochlear implants have a very limited ability to process TFS_p information, even if it is transmitted to the electrodes,[225] perhaps because of a mismatch between the temporal information and the place in the cochlea to which it is delivered[76] Reducing this mismatch may improve the ability to use TFS_p information and hence lead to better pitch perception.[226] Some cochlear implant systems transmit information about TFS_p in the channels of the cochlear implants that are tuned to low audio frequencies, and this may improve the pitch perception of low-frequency sounds.[227]