Instruction: Write an abstract for the following acadamic paper.

Input:
Paper: we have recorded the flight of a fly during take off and landing using digital high speed photography . it is shown that the dynamics of flexible wings are different for these procedures . during this observation fly flew freely in a big box and it was not tethered .
Abstract: in this fluid dynamics video , we demonstrated take off and landing of a fly . the deformation of wings is in focus in this video .

Paper: single particle tracking microscopy provides the position time series @xmath0 of individual particle trajectories in a medium @xcite . the information garnered by single particle tracking yields insights into the mechanisms and forces , that drive or constrain the motion of the particle . an early example of systematic single particle tracking is given by the work of jean perrin on diffusive motion @xcite . due to the relatively short individual trajectories , perrin used an ensemble average over many trajectories to obtain meaningful statistics . a few years later , nordlund conceived a method to record much longer time series @xcite , allowing him to evaluate individual trajectories in terms of the time average and thus to avoid averages over not perfectly identical particles . today , single particle tracking has become a standard tool to characterise the microscopic rheological properties of a medium @xcite , or to probe the active motion of biomolecular motors @xcite . particularly in biological cells and complex fluids single particle trajectory methods have become instrumental in uncovering deviations from normal brownian motion of passively moving particles .    ) as function of the lag time @xmath1 . left : normal diffusion growing like @xmath2 ( 1 ) , restricted ( confined ) diffusion with a turnover from @xmath3 to @xmath4 ( 2 ) , drift diffusion @xmath5 ( 3 ) . right : ergodic subdiffusion @xmath6 ( 1 ) , restricted ergodic subdiffusion turning over from @xmath6 to @xmath4 ( 2 ) , non - ergodic subdiffusion @xmath3 ( 1  ) , restricted non - ergodic subdiffusion turning over from @xmath3 to @xmath7 ( 2  ) , superdiffusion @xmath8 ( 3 ) . here , @xmath9 . note the double - logarithmic scale.,width=332 ]    classical diffusion patterns are sketched in the left panel of fig . [ diffusion_modes ] . accordingly , one may observe free diffusion , leading to a linear growth with time of the second moment [ line 1 in fig . [ diffusion_modes ] ] . brownian motion may also be restricted ( corralled , confined ) . confinement in a cell , for instance , could be due to the cell walls . in that case the second moment initially grows linearly with time and eventually saturates to its thermal value equalling the second moment of the corresponding boltzmann distribution [ line 2 ] . in the presence of a drift the second moment grows with the square of time [ line 3 ] . such results are typical for simple fluids . in more complex environments , different patterns may be observed , as displayed on the right of fig . [ diffusion_modes ] . here , subdiffusion may occur , for which the second moment grows slower than linearly with time [ line 1 ] . restricted subdiffusion would depart from this behaviour to reach a plateau [ line 2 ] . driven motion may lead to a superdiffusive power - law form of the second moment with an exponent between 1 and 2 [ line 3 ] . however , as we demonstrate below , subdiffusion may also be non - ergodic , and the associated time averaged second moment may grow _ linearly _ with time [ line 1  ] . similarly strange behaviour may be observed for restricted non - ergodic subdiffusion , which exhibits a _ power - law _ growth , not a saturation to a plateau [ line 2  ] . non - ergodic processes come along with a significant scatter between individual trajectories . this is an effect of the ageing nature of the process that persists for long measurement times . in the following we discuss in detail the behaviour of passive subdiffusive motion in terms of time and ensemble averages and address the peculiarities , that may arise for non - ergodic systems . non - ergodic behaviour of the above sense is indeed observed experimentally . [ christine ] shows the time averaged mean squared displacement for lipid granules in a living fission yeast cell . the motion is recorded by indirect tracking in an optical tweezers setup . initially the granule is located in the bottom of the laser trap potential such that the granule moves freely . eventually the granule ventures away from the centre of the trap and experiences the hookean trap force . as demonstrated in a detailed analysis the granule motion indeed exhibits weak ergodicity breaking , giving rise to the characteristic turnover from an initially linear scaling @xmath2 with the lag time @xmath1 , to the power - law regime @xmath10 @xcite . moreover a pronounced trajectory - to - trajectory scatter is observed , again typical for systems with weak ergodicity breaking . the initially linear scaling @xmath11 turns over to the power - law regime @xmath12 , induced by the restoring force exerted on the granules by the laser trap @xcite . note also the characteristic scatter between individual trajectories . inset : average behaviour of the shown trajectories . in both graphs the unit of @xmath13 is volt@xmath14 , i.e. , the direct output voltage of the quadrant photodiode . the voltage is directly proportional to the distance from the centre of the laser trap.,width=332 ]    free diffusion is typically quantified in terms of the second moment . the mean squared displacement @xmath15 is obtained as the spatial average over the probability density function @xmath16 to find the particle at position @xmath17 at time @xmath18 . the quantity ( [ eamsd ] ) therefore corresponds to the ensemble averaged second moment of the particle position , denoted by angular brackets , @xmath19 . in particular , the time @xmath18 enters into eq . ( [ eamsd ] ) only as a parameter . conversely , single particle trajectories @xmath0 are usually evaluated in terms of the time averaged mean squared displacement defined as @xmath20 where we use an overline @xmath21 to symbolise the time average . here @xmath1 is the so - called lag time constituting a time window swept along the time series , and @xmath22 is the overall measurement time . the time averaged mean squared displacement thus compares the particle positions along the trajectory as separated by the time difference @xmath1 .    in an ergodic system the time average of a certain quantity obtained from sufficiently long time series is equal to the corresponding ensemble mean @xcite . for instance , for the mean squared displacement ergodicity would imply @xmath23 brownian motion is ergodic , as well as certain stationary processes leading to anomalous diffusion , such as fractional brownian motion considered below . there exist , however , non - ergodic processes , which are intimately connected to ageing properties . in what follows we discuss two prominent models for anomalous diffusion , the stationary fractional brownian motion and the ageing continuous time random walk , and analyse in detail the features of the associated time averaged mean squared displacement .    generally , anomalous diffusion denotes deviations from the classical linear dependence of the mean squared displacement , @xmath24 . such anomalies include ultraslow diffusion of the form @xmath25 @xcite . in contrast , anomalous diffusion processes may become faster than ballistic , for instance for systems with correlated jump lengths or in systems governed by generalised langevin equations @xcite . here we are interested in anomalous diffusion with power - law dependence on time @xcite , @xmath26 for which the anomalous diffusion exponent belongs to the subdiffusive range @xmath9 , such that the limit @xmath27 corresponds to brownian motion . the proportionality factor @xmath28 in eq . ( [ aneamsd ] ) is the anomalous diffusion coefficient of physical dimension @xmath29 . the embedding spatial dimension is @xmath30 , and eq . ( [ aneamsd ] ) includes the complete gamma function @xmath31 . subdiffusion of the form ( [ aneamsd ] ) with @xmath9 occurs in the following biologically relevant systems . fluorescently labelled mrna in _ e.coli _ bacteria cells was observed to follow @xmath32 with @xmath33 @xcite . this result is consistent with more recent findings according to which free rna tracers in living cells exhibit @xmath34 , while dna loci show @xmath35 @xcite . telomeres in the nucleus of mammalian cells were reported to follow anomalous diffusion with @xmath36 at shorter times and value @xmath37 at intermediate times , before a turnover to normal diffusion occurs @xcite . also larger tracer particles show anomalous diffusion , such as adeno - associated viruses of radius @xmath38 nm in a cell with @xmath39 @xcite and endogenous lipid granules , of typical size of few hundred nm with @xmath40 @xcite . it should be noted that this subdiffusion observed by single particle tracking microscopy is consistent with results from other techniques , such as fluorescence correlation spectroscopy @xcite or dynamic light scattering @xcite . subdiffusion of biopolymers larger than some 10 kd in living cells is due to molecular crowding , the excluded volume effect in the superdense cellular environment @xcite . larger tracer particles also experience subdiffusion due to interaction with the semiflexible cytoskeleton @xcite . knowledge of the time or ensemble averaged mean squared displacement of an anomalous diffusion process is insufficient to fully characterise the underlying stochastic mechanism , as the associated probability density @xmath16 is no longer necessarily gaussian , and therefore no longer specified by the first and second moments , only @xcite . this property is in contrast to the universal gaussian nature of brownian motion which is effected by the central limit theorem . at the same time the very nature of the anomalous diffusion process may result in decisively different behaviours for diffusional mixing and diffusion - limited reactions @xcite . in biological cells this would imply significant differences for signalling and regulatory processes . for a better understanding of the dynamics in biological cells and other complex fluids knowledge of the underlying stochastic mechanism is therefore imperative . here we discuss the properties of continuous time random walk ( ctrw ) processes with diverging characteristic waiting time with respect to their time averaged behaviour , expanding on our earlier work @xcite . we show that for free motion the lag time dependence of the time averaged mean squared displacement , @xmath41 , is insensitive to the anomalous diffusion exponent @xmath42 for ctrw processes . in contrast , for confined ctrw subdiffusion a universal scaling behaviour emerges , @xmath43 , with dynamic exponent @xmath44 [ curves ( 1  ) and ( 2  ) in fig . [ diffusion_modes ] ] . subdiffusion governed by fractional brownian motion ( fbm ) leads to the scaling @xmath45 of the time averaged mean squared displacement , turning over to a saturation plateau under confinement , @xmath46 [ curves ( 1 ) and ( 2 ) in fig . [ diffusion_modes ] ] . we particularly emphasise the irreproducible nature of the time averaged quantities and their associated scatter around the ensemble mean for ctrw subdiffusion processes . this randomness of the time averages is captured by the distribution function of the amplitude of the time average . we show that even for relatively short trajectories this distribution is a good characteristic for the underlying process . in contrast for fbm processes the scatter typical of many single particle experiments is not found in the long time limit .    in the remainder of this work , for simplicity we restrict the discussion to the one - dimensional case ( @xmath47 ) . generalisation to higher dimensions is straightforward . the article is structured as follows . we start with a brief introduction to ctrw and fbm . we then consider the cases of unbounded motion and confined anomalous diffusion in the subsequent two sections . the distribution of the time averages will be presented thereafter . finally , we discuss the velocity autocorrelation functions for subdiffusive ctrw and fbm processes , before presenting a concluding discussion . although both ctrw and fbm give rise to an ensemble averaged mean squared displacement of the form ( [ eamsd ] ) , they are fundamentally different processes , as outlined here . we note in passing that also in the random motion on a fractal support subdiffusion arises @xcite . we will not pursue this type of anomalous diffusion in the following . ctrw theory dates back to the work of montroll and weiss @xcite , and was championed in the analysis of charge carrier motion in amorphous semiconductors by scher and montroll @xcite . ctrw has become a standard statistical tool to describe processes ranging from particle motion in actin networks @xcite to the tracer motion in groundwater @xcite . each jump of a ctrw process is characterised by a random jump length and a random waiting time elapsing before the subsequent jump . at each jump the jump length and waiting time are chosen independently . for a subdiffusive process we assume that the variance of the jump lengths is given by a finite value @xmath48 , and we consider the unbiased case @xmath49 . on a lattice of spacing @xmath50 , we would have @xmath51 . in contrast , the waiting times @xmath52 are drawn from the probability density @xmath53 for large @xmath52 , with @xmath9 . this form of @xmath54 is scale - free , that is , the average waiting time @xmath55 diverges , causing effects such as ageing @xcite and weak ergodicity breaking @xcite . in eq . ( [ wtd ] ) , the quantity @xmath56 is a scaling factor . the anomalous diffusion constant in this case becomes @xcite @xmath57    the scale - freeness of @xmath58 allows individual waiting times @xmath52 to become quite large . no matter how long a given time series is chosen , single @xmath52 values may become of the order of the length of the entire time span covered in the trajectory . an example is shown in fig . [ timeseries ] : the stalling events with large waiting times @xmath52 are quite distinct . for values of the anomalous diffusion exponent @xmath42 that are closer to 1 the stalling is less pronounced . physically , the power - law form of the waiting time distribution @xmath58 may be related to comb models @xcite or random energy landscapes @xcite ; for more details , compare refs .  .     for brownian motion ( top ) , ctrw ( middle ) , and fbm ( bottom ) the anomalous diffusion exponent for fbm and ctrw is @xmath59 . for ctrw the stalling events are outstanding , while in the case of fbm strong antipersistence occurs . note that the stalling events and the antipersistence are less pronounced for larger values of @xmath42 . typical experimental data include additional noise , such that the appearance of measured data will not display the ideal behaviour shown here.,width=332 ]      fbm is a gaussian process with stationary increments . its position is defined in terms of the langevin equation @xmath60 or , alternatively , @xmath61 the motion is driven by stationary , fractional gaussian noise @xmath62 with zero mean @xmath63 and long - ranged noise correlation @xcite @xmath64 contrasting the uncorrelated noise for normal diffusion @xmath27 : @xmath65 . in eq . ( [ fbnoise ] ) the anomalous diffusion exponent is connected to the traditionally used hurst exponent by @xmath66 , and we introduce the abbreviation @xmath67 for consistency with standard fbm notation . for subdiffusion the fractional gaussian noise is anticorrelated , decaying like @xmath68 . this implies that a given step is likely to go into the direction opposite to the previous step . the corresponding oscillating behaviour is seen in fig . [ timeseries ] . the position autocorrelation function of fbm becomes @xmath69 so that at equal times @xmath70 we recover the mean squared displacement ( [ aneamsd ] ) .    if the fractional gaussian noise is not considered external , but the validity of the fluctuation - dissipation theorem is imposed , one obtains the generalised langevin equation ( gle ) @xcite , @xmath71 where we write @xmath72 instead of the exponent @xmath42 in eq . ( [ fbnoise ] ) . in the gle ( [ gle ] ) , we defined the coupling coefficient @xmath73}$ ] according to the fluctuation - dissipation theorem , where @xmath74 is the boltzmann constant and @xmath75 the absolute temperature . in what follows we only consider the overdamped limit , in which the inertia term @xmath76 can be neglected . the gle then gives rise to the form @xmath77 of the mean squared displacement . in contrast to fbm , that is , the gle leads to subdiffusion for persistent noise with @xmath78 , while @xmath79 yields superdiffusion . fbm and the related gle are used to describe processes such as long term storage capacity of water reservoirs @xcite , climate fluctuations @xcite , economical market dynamics @xcite , single file diffusion @xcite , and elastic models @xcite . motion of this type has also been associated with the relative motion of aminoacids in proteins @xcite , and the free diffusion of biopolymers under molecular crowding conditions @xcite . let us begin with considering anomalous diffusion on an infinite domain and without drift . to find an analytical expression for the time averaged mean squared displacement ( [ tamsd ] ) we note that even a brownian process recorded over a finite time span @xmath22 will show fluctuations in the number of jumps performed during @xmath22 . to average out these trajectory - to - trajectory fluctuations we introduce the ensemble mean , @xmath80 we can then express the integrand in terms of the variance of the jump lengths , @xmath48 , and the average number of jumps @xmath81 in the time interval @xmath82 , as follows : @xmath83 for a regular random walk on average every jump occurs after the waiting time @xmath55 . thus @xmath84 , and @xmath85 where we defined the diffusion constant @xmath86 $ ] . in the brownian limit the time averaged mean squared displacement ( [ btamsd ] ) in terms of the lag time @xmath1 takes on exactly the same form as the ensemble averaged mean squared displacement ( [ eamsd ] ) as function of time @xmath18 . this is not surprising , as brownian motion is an ergodic process . for sufficient duration @xmath22 of the time records any time average converges to the corresponding ensemble average , and thus the ensemble average in expression ( [ eatamsd ] ) is no longer necessary . the number of jumps of a ctrw process with waiting time distribution of the form ( [ wtd ] ) on average grows sublinearly with time , @xmath87 $ ] @xcite . this time evolution translates into the mean squared displacement ( [ eamsd ] ) with the anomalous diffusion coefficient ( [ an_coff ] ) . combining the time dependence of @xmath88 for ctrw subdiffusion with the definition ( [ eatamsd ] ) we obtain the following result for the time averaged mean squared displacement , @xmath89 in the limit @xmath90 @xcite . this follows from expansion of the relation @xmath91 . the noteworthy feature of this result is that the linear lag time dependence of normal diffusion remains completely unaffected by the anomalous nature of the stochastic process . only the dependence on the overall measurement time @xmath22 witnesses the underlying subdiffusion . ( [ ctrwtamsd ] ) can be understood if we notice that for normal diffusion we have @xmath92 [ since according to einstein @xmath93 where @xmath94 is the mean time between jump events . now , for the subdiffusive case @xmath55 diverges and must be replaced by @xmath95 , which explains the term @xmath96 . [ ctrw_tamsd ] shows 20 trajectories of a ctrw process with @xmath97 , displaying the general trend @xmath98 .    , shown for 20 individual trajectories . the overall measurement time is @xmath99 ( a.u . ) . note the local changes of the slope in the trajectories as well as the complete stalling in the lowest curve , all bearing witness to the scale - free nature of the underlying waiting time distribution @xmath100.,width=332 ]    rewriting the result ( [ ctrwtamsd ] ) in the form @xmath101 we see that the effective diffusion coefficient @xmath102 decays as function of the measurement time @xmath22 . the longer the system evolves after its initial preparation , the less mobile it appears , consistent with the ageing property of ctrw subdiffusion . for instance , in the picture of the trapping events , in the course of time deeper and deeper traps may be encountered by the diffusing particle , such that it gets more and more stuck . this increasing immobility is mirrored in the form @xmath102 . note that eqs . ( [ ctrwtamsd ] ) and ( [ rew ] ) suggest that from measuring the @xmath1 dependence of an anomalous diffusion process one might draw the erroneous conclusion that normal diffusion were observed . the disparity between the time averaged mean squared displacement ( [ ctrwtamsd ] ) and its ensemble averaged counterpart ( [ aneamsd ] ) demonstrates the weak ergodicity breaking characteristic of a process with diverging time scale @xcite . single or few , long waiting times are also responsible for the pronounced deviations between individual realisations shown in fig . [ ctrw_tamsd ] . this apparent irreproducibility of the time averaged mean squared displacement is again intimately coupled to the weak ergodicity breaking nature of the ctrw subdiffusion process . we will discuss this feature more quantitatively in terms of the distribution @xmath103 as function of the relative deviation @xmath104 of the time averaged mean squared displacement around its ensemble mean in section [ scatter ] . in contrast to the above behaviour of ctrw subdiffusion processes , fbm is ergodic . indeed , by help of the position autocorrelation ( [ fbm_auto ] ) the time averaged mean squared displacement ( [ eatamsd ] ) becomes @xcite @xmath105 exactly matching the ensemble averaged mean squared displacement , @xmath106 . however , ergodicity is reached algebraically slowly @xcite , see also below . in fig . [ fbm_tamsd ] the comparatively minute scatter between individual trajectories supports the ergodic behaviour of fbm processes . the somewhat larger deviations at longer lag times are due to worsening statistics when @xmath107 .    . the scatter between individual trajectories is minute , mirroring the ergodicity of this process.,width=332 ] since the motion of tracer particles is typically confined , for example , by the cell walls or internal membranes , we now consider the important case of anomalous diffusion in a bounded domain .    in a finite interval @xmath108 $ ] the mean squared displacement of a brownian particle initially released well away from the boundaries will grow linearly in time and eventually turn over to a stationary plateau of magnitude @xmath109 . similarly , if the particle evolves in the confinement of an external harmonic potential of the form @xmath110 , the thermal value @xmath111 $ ] will eventually be attained . how is this behaviour modified for a ctrw subdiffusion process ? under the influence of an arbitrary external potential @xmath112 defining the force @xmath113 , ctrw subdiffusion can be described by the fractional fokker - planck equation @xcite , or , equivalently , in terms of the following coupled langevin equations @xcite :    @xmath114    here the position @xmath115 is expressed in terms of the parameter @xmath116 ( the internal time ) , and driven by the white gaussian noise @xmath117 . thus , eq .  ( [ le1 ] ) defines standard brownian motion @xmath118 , where @xmath119 is the diffusivity for the normal diffusion process in internal time @xmath116 . laboratory time @xmath18 is introduced by the so - called subordination through the process @xmath120 , given by the probability density function @xcite @xmath121^{1/\alpha } \frac{t}{s^{1 + 1/\alpha}}\right),\ ] ] where @xmath122 is a one - sided lvy stable probability density with laplace transform @xmath123 . thus , eq .  ( [ le2 ] ) transforms the brownian process @xmath118 with diffusivity @xmath119 into the subdiffusive motion with generalised diffusivity @xmath28 . on the basis of this scheme , the ensemble averaged position - position correlation in an arbitrary confining potential @xmath124 becomes @xmath125 valid in the limit @xmath126)^{1/\alpha}$ ] , where @xmath127 is the smallest non - zero eigenvalue of the corresponding fokker - planck operator @xcite . ( [ ctrw_conf_ea_auto ] ) demonstrates that , despite the confinement , the process is non - stationary . in result ( [ ctrw_conf_ea_auto ] ) we used the incomplete beta function @xmath128 and defined the first and second boltzmann moments , whose general definition is @xmath129 the partition function reads @xmath130 at large time separation , @xmath131 , the position autocorrelation decays algebraically , @xmath132 towards the value @xmath133 . the corresponding limiting behaviour of the incomplete beta function reads @xmath134    inserting expression ( [ ctrw_conf_ea_auto ] ) into the definition of the time averaged mean squared displacement ( [ eatamsd ] ) we obtain @xcite @xmath135dt.\end{aligned}\ ] ] then , with relation ( [ bet_lim ] ) we arrive at the scaling behaviour @xmath136 valid in the limits @xmath137 and @xmath138 ) ^{1/\alpha}$ ] . result ( [ conf_tamsd ] ) is quite remarkable : instead of the naively assumed saturation toward the stationary plateau a power - law growth @xmath139 is observed . only when the lag time @xmath1 approaches the overall measurement time @xmath22 the singularity in expression ( [ eatamsd ] ) causes a dip toward the plateau of the ensemble averaged mean squared displacement . additionally eqs . ( [ ctrw_conf_ea_auto ] ) and ( [ conf_tamsd ] ) are universal in the sense that the exact form of the confining potential solely enters into the prefactor through the first two moments of the boltzmann distribution corresponding to the confining potential @xmath124 . we note that the asymptotic scaling @xmath140 is consistent with the numerical analysis presented in ref . @xcite . [ confined_ctrw ] depicts the ensemble mean of the time averaged mean squared displacement ( [ eatamsd ] ) for two types of confinement , an harmonic potential and a box potential . the particle is initially placed at the bottom of the potential well and in the middle of the box potential , respectively . this is why at short lag times the particle exhibits the linear scaling @xmath141 typical for unconfined motion , before turning over to the confinement - induced scaling @xmath7 . note that this plot is fit - free , i.e. , the theoretically calculated asymptotic behaviours nicely fall on top of the simulations results . for an harmonic binding potential @xmath142 ( @xmath143 ) and a particle in a box of length @xmath144 ( @xmath145 ) , with @xmath146 , measurement time @xmath147 ( a.u . ) , @xmath148 , and @xmath149 . without fitting , the lines show the analytic results for the transition from @xmath150 for short lag times according to eq . ( [ ctrwtamsd ] ) ( @xmath151 and @xmath152 ) to @xmath7 for long lag times , eq . ( [ conf_tamsd ] ) (  ) . for long @xmath1 , @xmath153 exhibits universal behaviour , in the sense that the curve does not depend on the external field.,width=332 ]    as shown in fig . [ conf_scatter ] individual trajectories still exhibit the pronounced scatter typical for ctrw subdiffusion . visually the scatter does not change between the unbiased initial motion and the confinement - dominated part of the process . this is due to the fact that the scatter is caused by the scale - freeness of the waiting times . in our process waiting times and jump lengths are decoupled , corresponding to the subordination property of ctrw subdiffusion @xcite .    . the extent of the deviations between the trajectories does not change qualitatively between the initial unbiased motion and the later confinement - dominated regime . the squares ( @xmath143 ) represent simulations results for the ensemble average @xmath154 . same parameters as in fig . [ confined_ctrw].,width=332 ]      what happens if we introduce a cutoff in the power - law of the waiting time distribution of the form @xmath155,\ ] ] in which a characteristic time scale @xmath156 is introduced , terminating the power - law scaling ? for the waiting time density ( [ wtd_cutoff ] ) the characteristic waiting time @xmath157 becomes finite . at sufficiently short times one would expect this process to still exhibit the features of ctrw subdiffusion , while at times @xmath158 the process should converge to regular brownian motion with a gaussian propagator . in fig . [ ctrw_cutoff_conf ] we demonstrate that for a suitable choice of the cutoff time @xmath156 the behaviour of the time averaged mean squared displacement @xmath13 under confinement preserves the characteristic non - ergodic features of ctrw subdiffusion , i.e. , the turnover from the initial scaling @xmath3 to the confinement - dominated scaling @xmath159 . below we will show that at the same time the distribution of the time average around its ensemble mean is significantly altered .    ) , in the box @xmath160 $ ] with reflecting boundary conditions . the generic behaviour of initial and final scaling , @xmath161 and @xmath7 remains unaltered . we chose @xmath162 , @xmath163 , @xmath164 and overall measurement time @xmath165 ( a.u.).,width=332 ]      ergodicity remains unaffected for fbm , that is confined to an interval of size @xmath108 $ ] . namely , convergence of @xmath13 is observed toward the stationary value @xmath109 @xcite . for fbm under the influence of an harmonic potential @xmath110 the position - position correlation can be calculated exactly @xcite . inserting into the ensemble mean ( [ eatamsd ] ) of the time averaged mean squared displacement one can show that the initial behaviour @xmath166 turns over to the stationary value @xmath167 $ ] . in general , the time averaged mean squared displacement for confined fbm converges to a constant : @xmath168 the deviations of the time averaged mean squared displacement @xmath169 between individual trajectories can be quantified in terms of the probability density function @xmath170 of the dimensionless ratio @xmath171 of the time averaged mean squared displacement over its ensemble mean . for an ergodic process this distribution is necessarily sharp , @xmath172 for long measurement time @xmath22 . deviations from this form are expected for non - ergodic processes such as ctrw subdiffusion , but also for relatively short trajectories . we here address both effects and demonstrate that the distribution @xmath170 is a quite reliable means to distinguish different stochastic processes even when the recorded time series are fairly short .      for ctrw subdiffusion with power - law waiting time distribution ( [ wtd ] ) the distribution assumes the form @xcite @xmath173 for @xmath174 , where @xmath122 is again a one - sided lvy stable distribution , whose laplace transform is @xmath175 . special cases include the brownian limit ( [ phi1 ] ) for @xmath176 and the gaussian shape @xmath177 for @xmath146 . in fact , we observe an exponentially fast decay of @xmath170 with large @xmath178 for all @xmath9 , compare appendix [ app1 ] .    for short trajectories the basic shape of the distribution @xmath170 is surprisingly well preserved @xcite . this is demonstrated in fig . [ ctrw_scatter ] , in which the characteristic asymmetric shape with respect to the ergodic value @xmath179 for the case @xmath97 is reproduced for a process with @xmath180 , even for a lag time as long as @xmath181 . no significant dependence on the confinement is observed . also for larger values of @xmath42 the finite value for small @xmath178 is similarly reproduced for short trajectories @xcite , pointing at the quite remarkably reliability on the shape of the scatter distribution @xmath170 for ctrw subdiffusion . as we show below , the distribution for fbm processes with its zero value at @xmath182 can be clearly distinguished from the ctrw form .     of the time averaged mean squared displacement , with @xmath183 for a ctrw process with @xmath97 , @xmath163 , and @xmath180 ( a.u . ) . the filled and open symbols , respectively , represent the unconfined case and confined motion on the interval @xmath184 $ ] . we see quite good agreement with the expected gaussian limiting distribution of the scatter , eq . ( [ phi_gauss ] ) , centred around @xmath182 . in the inset we show the case for the largest measured lag time , @xmath185 , also in good agreement with the predicted shape of the distribution.,width=332 ]    a useful parameter to quantify the violation of ergodicity is the ergodicity breaking parameter @xcite @xmath186 eb varies from @xmath187 for @xmath188 monotonically to @xmath189 in the brownian limit @xmath27 . for the special case @xmath146 one finds @xmath190 , while for @xmath191 , @xmath192 . the scatter distribution is sensitive to few extreme events . when these are lacking , the distribution should be significantly different from the form discussed for ctrw subdiffusion with diverging characteristic time scale . indeed , as shown in fig . [ cutoff_scatter ] for the cutoff waiting time distribution defined through eq . ( [ wtd_cutoff ] ) , @xmath170 drops down to zero around @xmath182 and assumes an almost gaussian shape around the ergodic value @xmath179 . note that the simulations were performed with the same parameters as for fig . [ ctrw_cutoff_conf ] , in which the breaking of ergodicity still persists , despite the cutoff . this demonstrates that different quantities have different sensitivity to the cutoff .     for ctrw subdiffusion with cutoff , as defined in eq . ( [ wtd_cutoff ] ) . the observed form is clearly different from the much more asymmetric shape in fig . [ ctrw_scatter ] . the value of @xmath170 reaches zero for small values of @xmath178 . an almost gaussian shape centred around the ergodic value @xmath179 is observed . same parameters as in fig . [ ctrw_cutoff_conf].,width=332 ]      fbm is based on long - ranged correlations . however , we can obtain an approximate expression for the scatter distribution @xcite @xmath193 where @xmath194 is an intrinsic time scale . expression ( [ fbmscatter ] ) is valid for sufficiently small lag times @xmath1 , for which the correlations are neglected @xcite . note that this distribution is independent of @xmath42 and centred around the ergodic value @xmath179 . in particular , in the long measurement time limit @xmath174 , the distribution converges to the sharp form @xmath195 . also this behaviour is surprisingly well preserved for short trajectories , as demonstrated in fig . [ fbm_scatter ] .     of the time averaged mean squared displacement , with @xmath196 for fbm subdiffusion with @xmath97 and @xmath180 ( a.u . ) . the filled and open symbols , respectively , represent the cases for unconfined and confined motion . for small values of the lag time @xmath1 we see quite good agreement with the expected gaussian limiting distribution of the scatter , eq . ( [ fbmscatter ] ) , that is centred around @xmath179 . for larger values of @xmath1 deviations are observed , however , even for @xmath185 the value around @xmath182 is consistently zero . a similar behaviour is found for larger values of @xmath42 @xcite . note that when @xmath174 the process is ergodic and @xmath197 approaches a @xmath198 function.,width=332 ]    for fbm one can define a quantity similar to the ergodicity breaking parameter ( [ eb ] ) . namely , without taking the long measurement time limit , we obtain the normalised variance of the time averaged mean squared displacement @xmath199 it turns out that the convergence to ergodicity is algebraically slow , and for subdiffusion @xmath9 one obtains a decay , which is inversely proportional to @xmath22 @xcite : @xmath200 the coefficient @xmath201 here is defined as @xcite @xmath202 it increases continuously from zero at @xmath203 to @xmath204 .    the quantity @xmath205 is connected to the probability density ( [ fbmscatter ] ) by @xmath206 , obtained from approximating that the values of @xmath178 may range in the interval @xmath207 , which appears reasonable given the sharp decay of @xmath197 to 0 at @xmath182 for fbm . apart from the fact that @xmath205 explicitly depends on @xmath42 while the approximation leading to eq . ( [ fbmscatter ] ) loses the @xmath42 dependence , both quantities decay @xmath208 , and the internal time scale @xmath194 takes on a role similar to the lag time @xmath1 . a typical quantity accessible from experimental data is the velocity autocorrelation function , which is defined through @xmath209 here the velocity is defined as the difference quotient @xmath210 $ ] . the velocity autocorrelation ( [ autov00 ] ) was suggested as a tool to distinguish between different subdiffusion models @xcite . we show here that for confined processes the shape of the velocity autocorrelation function does not allow for a significant distinction between subdiffusive ctrw and fbm . note that the calculation of @xmath211 amounts to determine four two - point correlation functions of the type @xmath212 .      for unbounded ctrw process starting with initial condition @xmath213 , the position correlation function is given by the following expression @xcite , @xmath214^\alpha . \label{autov03}\ ] ] this result for free ctrw is due to the fact that the jump lengths in the interval @xmath215 for @xmath216 have zero mean . with eq . ( [ autov03 ] ) we find @xmath217 0 & \tau\geq\epsilon \end{array}\right . \label{autov04}\ ] ] the velocity autocorrelation function for an unbounded ctrw process does not yield negative values due to the absence of correlations for different jumps . in this case the velocity autocorrelations can be easily distinguished from the behaviour of fbm processes , see below . note that @xmath211 in eq . ( [ autov04 ] ) is non - analytic at @xmath218 , an observation that is confirmed in simulations . the behaviour of the velocity autocorrelations for ctrw subdiffusion are displayed in fig . [ velcorr1 ] .    for a confined ctrw process the situation is quite different . to explore the behaviour of the corresponding velocity autocorrelation function we use the general result for the correlation function of confined ctrw , eq . ( [ ctrw_conf_ea_auto ] ) . for simplicity we assume a symmetric potential such that @xmath219 . with the initial condition @xmath213 we obtain @xmath220 figs . [ velcor2 ] and [ velcor3 ] ( for the absolute value ) display excellent agreement of eq . ( [ autov05 ] ) with simulations of ctrw subdiffusion for a lattice of size 10 with reflecting boundary conditions , and @xmath146 . we observe that for confined motion the ctrw velocity autocorrelation function indeed attains negative values and has a minimum on @xmath218 , as the confinement effectively induces correlations . for long @xmath52 the velocity autocorrelation function decays to zero ( from the negative side ) as the power - law @xmath221 valid for @xmath222 .      for free fbm the position correlation function ( [ fbm_auto ] ) together with the definition ( [ autov00 ] ) produce the result @xmath223 this function yields negative values for sufficiently long @xmath52 , and its minimum value @xmath224 is assumed at @xmath218 . for long @xmath52 it decays toward zero from the negative side in the power - law form @xmath225 valid for @xmath222 .     for confined fbm with @xmath97 . the simulations were performed on the interval @xmath184 $ ] , and @xmath226 ( a.u . ) . for comparison , the velocity autocorrelation for free fbm is drawn based on eq . ( [ autov01]).,width=332 ]    we see that both the velocity autocorrelation function of the confined ctrw and the one for free fbm acquire negative values and decay as power - laws for large @xmath52 , see below . further both obtain a sharp minimum for @xmath218 . confined fbm behaves similarly to free fbm , see fig . [ fbm_box ] . from our analysis it becomes clear that the shape of the velocity autocorrelation function may not be a good diagnosis tool to distinguish between subdiffusive ctrw and fbm processes . we note that for ctrw subdiffusion the time averaged correlation functions will be random variables , exactly like the time averaged mean squared displacement . the physical mechanisms leading to subdiffusion in biological cells or other complex systems are varied . the anomalous diffusion of inert biopolymers larger than some 10kd in living cells is due to molecular crowding , these are excluded volume effects in the viscoelastic , superdense cellular environment @xcite . potentially similar effects occur in single file diffusion : a tracer particle diffusing in a one dimensional channel and interacting with other brownian particles will slow down its random motion , and subdiffusion with @xmath146 is observed @xcite . what is the resulting motion like ? consider a particle , that gets repeatedly trapped during its random walk . such trapping may give rise to a broad distribution of waiting times , as embodied in the ctrw model . such broad distribution of trapping times could be due to chemical binding of the tracer particle to its environment on varied time scales , or due to active gelation / de - gelation of the crowding particles . long - tailed trapping time distributions were observed for @xmath227m - sized particles due to interaction with a semiflexible actin mesh @xcite . there , the anomalous diffusion exponent @xmath42 depends on the size of the tracer particles versus the typical actin mesh size . alternatively , motion patterns of fbm type may be due to coupling to the viscoelastic crowding environment @xcite . yet other approaches are based on polymer dynamics . de gennes reptation model of a polymer diffusing in a tube formed by foreign chains in a melt yields subdiffusion with @xmath228 , and was used as a possible explanation of the short time dynamics of telomere motion @xcite .    one may speculate whether the widely observed subdiffusion of biomolecules and passive tracers is a mere coincidence of nature that should be attributed to the dense environment in the cell and/or to the wide distribution of obstacles and reactions in the cell . or , maybe , subdiffusion is by itself a goal which was obtained via evolution ? there exist claims that sub - diffusion is useful in certain cellular search strategies @xcite . one may also view subdiffusion to be a sufficiently slow process to help maintaining the organisation of the genome in the nucleus of the cell without the need for physical compartments @xcite . since reactions in many cases are controlled by diffusion , the emergence of slower - than - normal diffusion has far - reaching implications to signalling and regulatory processes in the cell . the standard theories of diffusion - controlled reactions under such circumstances must be replaced by subdiffusion - controlled reaction models . while these issues are clearly important , we here focused more on the characterisation of trajectories of single particles in the cell , in particular , as a quantitative diagnosis method to probe the nature of the stochastic motion in cells and other complex media . single particle tracking microscopy is a widely used technique . it allows one to locally probe complex systems in the liquid phase _ in situ_. in particular it has become one of the standard tools in biophysical , colloidal , polymeric , and gel - like environments . the motion of tracers in these systems is often anomalous . typically the experimentally recorded time series are evaluated in terms of the time averaged mean squared displacement @xmath13 . here we collected the behaviour of @xmath229 for the two most prominent anomalous stochastic processes , the continuous time random walk and fractional brownian motion . for ctrw subdiffusion , connected with ageing and weak ergodicity breaking , in an unconfined system the time averaged mean squared displacement scales linearly in time , @xmath13 [ eq . ( [ ctrwtamsd ] ) ] , renouncing the anomalous nature of the process . only the dependence on the overall measurement time @xmath22 pays tribute to the underlying subdiffusion . this behaviour contrasts the anomalous scaling @xmath230 of the ensemble averaged mean squared displacement . under confinement no plateau value is observed as in the ensemble average , instead , a power - law of the form @xmath43 , eq . ( [ conf_tamsd ] ) , is found . interestingly , this characteristic behaviour is approximately preserved when an appropriate cutoff in the waiting time is introduced . conversely the distribution of the time averaged mean squared displacement around its ensemble mean becomes almost gaussian in presence of the cutoff , while it has an exponential decay and a finite value at @xmath182 when the system is non - ergodic and exhibits ageing . fbm , in contrast , is ergodic , although ergodicity is reached algebraically slowly . free fbm subdiffusion shows @xmath231 , eq . ( [ fbm_free ] ) , while under confinement the plateau value of the ensemble average is reached , eq . ( [ fbm_cnf ] ) . for finite trajectories the distribution of the time averaged mean squared displacement is approximately gaussian for short lag times . our analysis demonstrates how different the two stochastic processes are , despite sharing the same form of the ensemble averaged mean squared displacement . at the same time the velocity autocorrelation of confined ctrw subdiffusion is hardly distinguishable from that of fbm subdiffusion . given a recorded time series of anomalous diffusion from experiment it is important to know more precisely which stochastic process is responsible for the observed behaviour , in particular , with respect to diffusion - limited reactions , general transport behaviour , and related processes such as gene regulation . the analysis presented here , along with complementary tools discussed in refs . @xcite , will be instrumental in the classification of anomalous diffusion behaviour . we thank c. selhuber - unkel and l. oddershede for providing the data used in fig . [ christine ] , and for many useful discussions . partial financial support from the deutsche forschungsgemeinschaft and the israel science foundation is gratefully acknowledged . using the theory of fox @xmath232-functions we obtain the exact form for the distribution @xmath170 of the dimensionless variable @xmath178 @xcite : @xmath233.\ ] ] this function has the series expansion @xmath234^n}{n!\gamma(-\alpha n)},\ ] ] and the asymptotic form @xmath235^{1/(1-\alpha)}\right)\end{aligned}\ ] ] valid at @xmath236 . s. b. yuste , g. oshanin , k. lindenberg , o. bnichou , and j. klafter , phys . e * 78 * , 021105 ( 2008 ) ; e. abad . s. b. yuste , and k. lindenberg , phys . e * 81 * , 031115 ( 2010 ) ; i. m. sokolov , s. b. yuste , j. j. ruiz - lorenzo , and k. lindenberg , phys . e * 79 * , 051113 ( 2009 ) ; d. froemberg and i. m. sokolov , phys . lett . * 100 * , 108304 ( 2008 ) . bouchaud , j. phys . ( paris ) i , * 2 * , 1705 ( 1992 ) ; g. bel and e. barkai , phys . rev . lett . * 94 * , 240602 ( 2005 ) ; a. rebenshtok and e. barkai , _ ibid . _ * 99 * , 210601 ( 2007 ) ; m. a. lomholt , i. m. zaid , and r. metzler , _ ibid . _ , 98 , 200603 ( 2007 ) ; i. m. zaid , m. a. lomholt , and r. metzler , biophys . j. * 97 * , 710 ( 2009 ) ; f. d. stefani , j. p. hoogenboom , and e. barkai , physics today * 62*(2 ) , 34 ( 2009 ) ; g. margolin and e. barkai , phys . 94 * , 080601 ( 2005 ) ; g. aquino , p. grigolini , and b. j. west , epl * 80 * , 10002 ( 2007 ) . r. metzler , e. barkai , and j. klafter , phys . lett . * 82 * , 3563 ( 1999 ) ; europhys . lett . * 46 * , 431 ( 1999 ) . e. barkai , r. metzler , and j. klafter , phys . e * 61 * , 132 ( 2000 ) ; r. metzler , j. klafter , and i. m. sokolov , _ ibid . _ * 58 * , 1621 ( 1998 ) . a. n. kolmogorov , dokl . 26 * , 115 ( 1940 ) ; b. b. mandelbrot and j. w. van ness , siam rev . * 1 * , 422 ( 1968 ) ; h. qian , fractional brownian motion and fractional gaussian noise . in g. rangarajan and m.z . ding ( eds ) , _ processes with long - range correlations _ ( springer , lecture notes in physics , vol.621 ) , pp.22 - 33 . t. e. harris , j. appl . * 2 * , 323 ( 1965 ) ; l. lizana and t. ambjrnsson , phys . . lett . * 100 * , 200601 ( 2008 ) ; l. lizana , t. ambjrnsson , a. taloni , e. barkai , and m. a. lomholt , phys . e * 81 * , 051118 ( 2010 ) ; e. barkai and r. silbey , phys . * 102 * , 050602 ( 2009 ) . m. magdziarz , a. weron , k. burnecki , and j. klafter , phys . * 103 * , 180602 ( 2009 ) . compare also m. magdziarz and j. klafter , phys . e , at press ; k. burnecki and a. weron , phys . e * 82 * , 021130 ( 2010 ) . h. c. fogedby , phys . e * 50 * , 1657 ( 1994 ) ; a. baule and r. friedrich , phys . e * 71 * , 026101 ( 2005 ) ; m. magdziarz , a. weron , and j. klafter , phys . rev . * 101 * , 210601 ( 2008 ) ; m. magdziarz , a. weron , and k. weron , phys . e * 75 * , 016708 ( 2007 ) ; e. heinsalu , m. patriarca , i. goychuk , g. schmid , and p. hnggi , phys . rev . e * 73 * , 046133 ( 2006 ) . a. m. mathai and r. k. saxena , the h - function with applications in statistics and other disciplines ( wiley eastern ltd . , new delhi , 1978 ) ; a. m. mathai , r. k. saxena , and h. j. haubold , the h - function , theory and applications ( springer , new york , 2010 ) .
Abstract:

Answer: anomalous diffusion has been widely observed by single particle tracking microscopy in complex systems such as biological cells . the resulting time series are usually evaluated in terms of time averages . often anomalous diffusion is connected with non - ergodic behaviour . in such cases the time averages remain random variables and hence irreproducible . here we present a detailed analysis of the time averaged mean squared displacement for systems governed by anomalous diffusion , considering both unconfined and restricted ( corralled ) motion . we discuss the behaviour of the time averaged mean squared displacement for two prominent stochastic processes , namely , continuous time random walks and fractional brownian motion . we also study the distribution of the time averaged mean squared displacement around its ensemble mean , and show that this distribution preserves typical process characteristic even for short time series . recently , velocity correlation functions were suggested to distinguish between these processes . we here present analytucal expressions for the velocity correlation functions . knowledge of the results presented here are expected to be relevant for the correct interpretation of single particle trajectory data in complex systems .


Instruction: Write an abstract for the following acadamic paper.

Input:
Paper: we have recorded the flight of a fly during take off and landing using digital high speed photography . it is shown that the dynamics of flexible wings are different for these procedures . during this observation fly flew freely in a big box and it was not tethered .
Abstract: in this fluid dynamics video , we demonstrated take off and landing of a fly . the deformation of wings is in focus in this video .

Paper: two important database operations are sorting and joins . these operations have three primary hardware - related costs : disk access , cpu operation and main memory access . the growth of main memory in current computers implies that main memory databases become more popular . for main memory databases , the bottleneck moves from disk to main memory and the cost for disk access is not an issue anymore . further , the growing cpu - memory gap implies that cpu costs represent an increasingly small portion of the total time . this has been borne out by several studies of dbms  @xcite . the impact of the cpu - memory gap was popularized by the paper of wulf and mckee  @xcite on the _ memory wall_.    the diminishing role of the cpu in the total running time is partially accounted for by increasing cpu speeds and greater on - chip functional parallelism . in part , it is also accounted for because most cpus today implement non - blocking caches and hardware prefetch in order to overlap cpu execution with memory access  @xcite . hence , memory access becomes the bottleneck . therefore , we follow the example of previous researchers  @xcite in concentrating on main memory as the bottleneck .    at the heart of this memory bottleneck lies the _ record retrieval problem _ : the problem of copying records from a source array into a destination array according to a new ordering . in a typical application , one will be given a source data file , a sequence of _ record ids _ ( rids ) for that data file , and a destination file . the task is to copy the source records into the destination file in the order specified by the sequence of rids . fast record retrieval is the key to faster sorting and faster joins .    a standard approach for data retrieval accesses the records of the source data file directly according to the sequence of the rids . this implies random access to the main memory . this , for example , is what was done in alphasort  @xcite and superscalarsort  @xcite , the current record holders for the datamation sorting challenge  @xcite . however , the cost of such random access has now become the dominant cost in main memory sorting , since the cpu - memory gap has widened still further since the original work on sorting . ( in fact , alphasort and superscalarsort sort data on disk , but the size of the data file , 100  mb , is small enough that the disk access consists solely of reading the source data file from disk , and writing to a destination data file . )    random access is harmful not just in disk resident databases , but also in main memory resident databases . current dram technologies , such as ddr ram and rambus ram ( rdram ) , extract a large latency penalty for any non - sequential access to ram . this is because the memory chips are divided into memory pages of several kilobytes , and there is a latency penalty for switching to a new memory page  @xcite . random access to ram also harms performance in a second manner . random access incurs a heavy penalty when large cache blocks are used . on a cache miss , the entire cache block is loaded into memory . if the record size is small compared to the cache block size , then there is a large overhead to load the entire cache block . for example , for a cache miss on a pentium  4 with ddr-266 ram , approximately 60  ns are spent loading the cache block , and approximately 60  ns are spent waiting on the latency of ddr ram . the trend is toward larger cache blocks . the 128  byte l2 cache blocks of the pentium  4 are four times larger than those of the pentium  iii . the ibm power4 processor goes still further using 512  byte l3 cache blocks . the solution to avoid these latency penalties in main memory is to access main memory sequentially . this is similar in spirit to the way in which traditional databases strongly prefer to access disk sequentially . in analogy with operation on disk , two - pass algorithms are a key for faster main memory performance . the dpg - based sorting algorithms immediately yield faster sorting algorithms . both alphasort and superscalarsort sort their data essentially in three phases : extraction of key - pointer pairs ; sorting of the key - pointer pairs ; and copying of the original records into the destination array according to the sorted key - pointer pairs . the last phase is essentially record retrieval . a re - implementation of alphasort and superscalarsort on a ibm p690 turbo shows that the record retrieval phase now dominates the running time . with the dpg record retrieval algorithm replacing the standard record retrieval algorithm , we immediately produce a faster sorting algorithm . in the re - implementation of superscalarsort , the versions with dpg as an accelerator are 27% faster . a direct consequence of faster record retrieval is faster main memory sort - merge joins . for example , in implementing sort - merge join using superscalarsort , we find that the use of dpg sort instead of superscalarsort results in a 27% faster join algorithm .. the dpg record retrieval phase in isolation is 48% faster than traditional record retrieval algorithm . finally , we apply dpg algorithm in the context of _ foreign key joins_. in a foreign key join the join key is the same as the foreign key . we assume that one relation has an secondary index on the join key . foreign key joins have the advantage that one need only rearrange the records of one of the files . this is in distinction to sort - merge join and hash join , which both require to rearrange each of the two files into a new file with join key values in sorted order . foreign key joins require less record retrieval because it is possible to first extract _ join triples _ , ( @xmath0 , rid@xmath1 , rid@xmath2 ) , where @xmath3 and @xmath4 are the two relations and @xmath0 is the join key value . assume that @xmath3 references a foreign key of  @xmath4 . to construct a join triple , do file scan of  @xmath3 , for each record of  @xmath3 , its key  @xmath0 is extracted . the secondary b - tree index of the join key on  @xmath4 is then used to derive the corresponding record i d , rid@xmath2 , with the key value  @xmath0 . the standard index lookup is very expensive , we propose a cache efficient b - tree batch lookup to generate the join triples . join triples reduce foreign key join to record pair retrieval . the join triples specify the record pairs , ( rid@xmath1 , rid@xmath2 ) , to be retrieved . one can re - order the records of  @xmath4 to match the ordering of rid@xmath1 in the sequence of record pairs . recall that , we do file scan for  @xmath3 , so rid@xmath1s in the join triples are in sorted order . alternatively , one can sort the join triples according to rid@xmath2 , and re - order the records of  @xmath3 to match the ordering of rid@xmath2 in the sorted rid pairs . in either situation , there are fewer record retrievals , and so foreign key join is faster than a general join . the ideas of faster record retrieval can also be applied to the general case of record pair retrieval . many algorithms for spatial join  @xcite and temporal join  @xcite produce record pairs . unlike equijoin , there is no single search key , and so record retrieval is more difficult . the ideas of this paper are described in terms of foreign key join . however , it is even simpler to translate the ideas into record pair retrieval , since an initial join triple extraction is not required . the _ distribute - probe - gather algorithm _ ( dpg ) is a record retrieval algorithm . given a data file of records and a sequence of record ids ( rids ) for the file , the goal is to copy the records into a destination file with the property that the ordering of records in the destination file corresponds to the ordering of the rids in the given sequence . for example , let the source file , # r # , be the array of records with a secondary b+ tree index on the attribute # a#. we want to place the records of # r # in sorted order according to the values of  # a#. the sequence of record ids , # s_rid # , at leaf nodes of the b+ tree is a list of record ids in sorted order according to the value of a. then , the destination file , # d # , below , will contain the corresponding records in sorted order according to the value of  # a#.    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ for each i , d[i ] = r[s_rid[i ] ] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _    in the case that the sequence of rids is a permutation of the rids for the data file , the sequence of rids acts as a permutation vector . the destination file is then a permutation of the records in the input file . note , however , that the dpg algorithm is not limited to permutations of data . in the case of join , one record from one relation may match more than one records from another relation . in such cases portion of the original records must be duplicated . the dpg algorithm also works for this case . the input for the algorithm is : a list of rids , # rid_list # , and a data file of records , # input#. the dpg algorithm partitions the rids , # rid_list # , into separate runs . it also partitions the data records , # input # , into separate runs . the algorithm makes two passes over the record ids , # rid_list # , and two passes over the records of the data file , # input#.    the ideas are presented in the context of main memory databases . we assume that neither the sequence of rids , # rid_list # , nor the data file , # input # , fit in cache . the dpg algorithm applies equally well as an external data retrieval algorithm between disk and main memory . the spirit of the dpg algorithm is : try to transform arbitrary memory access patterns into sequential memory access patterns ; where arbitrary memory access patterns are unavoidable , we try to divide the data into small partitions that fit into the cache .    for a sequential access pattern , it is easy to maintain a buffer in cache . in the dpg algorithm we need to read many streams simultaneously , and maintain a buffer for each stream . hence , we want to keep the buffer as small as possible while maintaining reasonable efficiency . we define a buffer in cache to consist of two cache blocks . when a cache block is full , it is written back to main memory and a new cache block is loaded from main memory into cache . on pentium  4 , this is done automatically by the hardware prefetch function unit if the access to main memory is sequential . on other architectures without the hardware prefetch function unit , the software instructions , @xmath5 and @xmath6 , are needed to maintain the buffers . [ [ constraints ] ] * constraints : * + + + + + + + + + + + + + +    the data records and rids are split into runs of length  @xmath7 . the run length  @xmath7 is chosen based on two constraints . first , the cache must be able to simultaneously hold both one run of data records of length  @xmath7 and one single buffer for the corresponding run of rids . ( this constraint applies in the second phase of dpg . ) second , the cache must simultaneously be able to hold a buffer for each run . ( this constraint applies in the first phase and in the third phase of dpg . ) these constraints are typical of the constraints for two - pass algorithms , such as external sorting .    if the data file has @xmath8  records , then the data file is partitioned into @xmath9 sets of consecutive records . assuming an rid consists of a page i d and offset on that page , the high order bits of the page number can be used to efficiently identify the particular partition to which the rid belongs . this assumes that the number of pages in a partition is a power of two , which can be satisfied by appropriate choice of  @xmath7 . for the sake of clarity , we assume the rids values are in the rage of @xmath10 and @xmath8 .    [ [ three - phases ] ] * three phases : * + + + + + + + + + + + + + + +    there are three phases in the dpg algorithm . the three phases are also illustrated by pseudo - code in figure  [ fig : dpgalgo ] and by the diagram of figure  [ fig : dpg ] . 1 .   * phase i. * the first phase is the _ distribute _ phase . one _ distributes _ the rids of # rid_list # into appropriate rid runs according to the values of the rids . the first rid run contains the rid values in the range from @xmath10 to @xmath7 , the second rid run contains the rid values in the range from @xmath7 to @xmath11 , and so on . both the access to every rid run and the access to # rid_list # are sequential . therefore , one only needs to maintains a buffer in cache for each rid run and a single buffer in cache for # rid_list # , at the end , we form @xmath9 rid runs and each rid run is a permutation vector . for example , the @xmath12-th rid run is a permutation vector in the range from @xmath13 and @xmath14 . * phase ii . * the second phase is the _ probe _ phase . in this phase , we allocate a second , temporary data file , # internal # , in main memory . the temporary data file has the same size as the original data file , # input # , and is organized into the same number of runs as the original data file . one then proceeds through each of the runs of rids and each of the runs of # input#. the rids from the @xmath12-th rid run are used to probe the @xmath12-th # input # run and the corresponding records are copied into the @xmath12-th # internal # run . + at the end , the @xmath12-th # internal # run contains the same records as the @xmath12-th # input # run , but the order of records in the # internal # run is organized according to the @xmath12-th rid run . both the @xmath12-th rid run and the @xmath12-th # internal # run are accessed sequentially . the @xmath12th # input # run is accessed randomly , but it can fit in cache . hence , every time , one loads the @xmath12-th # input # run entirely into cache and maintains two buffers in cache : one for the @xmath12th rid run and the other for the @xmath12-th # internal # run . * phase iii . * the third phase is the _ gather _ phase . this is an inverse of the _ distribute _ phase and is similar to the merge phase of external sorting . in the _ distribute _ phase , the rids are distributed into runs . as a result of the _ probe _ phase , the records in a given # internal # run are now in the same order as the rids in the corresponding rid run created in phase i. hence , it suffices to gather ( merge ) the records from the # internal # runs in exactly the same order as the order of the rids in # rid_list#. more precisely , if the @xmath12-th rid was distributed to the @xmath15-th rid run during the _ distribute _ phase , then at the @xmath12-th step of the _ gather _ phase , the next record from the @xmath15-th # internal # run is copied to the destination array . one maintains several buffers in cache : one single buffer for # rid_list # , one single buffer for the destination array , and a buffer for each # internal # run . the dpg algorithm is presented more formally in pseudo - code in figure  [ fig : dpgalgo ] . the input of the algorithm is an array , # rid # , of _ rids _ ( record ids ) and a data file , # input_rec # , of records . it is desired to retrieve the records from # input_rec # in the order corresponding to # rid#. the retrieved records are then written to # output_rec#.    the three phases of the dpg algorithm are illustrated by an example in figure  [ fig : dpg ] . the input in this example is the leaf nodes of a secondary b+-tree index . the input contains a sequence of key - rid pairs sorted according to the key values . the output is a sequence of records sorted according to the key values of the secondary index . the output will either be stored again on disk or else pipelined to the next stage . the ability of the dpg algorithm to take advantage of pipelining is an important feature for sorting and joins . the letters # a # , # b # , # c # , ... are used to indicate the sorted keys on the leaf nodes of the index . so ( # a # , 5 ) indicates that the record with the rid value of  5 has a key value of # a#. the first two rows are for phase  i , the next three rows for phase  ii , and the following three rows for phase  iii . the horizontal rectangle of the first row represents a sequence of key - rid pairs sorted according to the key values . the second row represents the runs of rids into which the first row is partitioned . similarly , the third row again represents the runs of rids , but now as part of phase  ii . the fourth row is the partitioned runs of input records , and so on . there are  12 elements and  3 runs in the example and each run contains  4 elements .    during * phase i * , one has a sequence of key - rid pairs sorted according to the key values and will distribute them into appropriate rid runs . the first rid run will contain rid values from  0 to  3 , the second rid run will contain rid values from  4 to  7 , and the third rid run will contain rid values from  8 to  11 . upon reading the sequence of pairs from the first row , one places the rids in their proper runs . for example , the rid  5 goes to the second rid run , the rid  7 goes to the second rid run , the rid  3 goes to the first rid run , the rid  8 goes to the third rid run , and so on . the third row in the figure presents the end of phase i. in this phase , we do sequential read on a single stream and sequential writes on multiple streams .    during * phase ii * , one copies the original data file , # input # , to a temporary data file , # internal#. the input to this phase is the third row . in the third row , each rid run has rid values that correspond to one contiguous range of the # input # file , an # input # run . for example , for the first rid run , we will load the first # input # run into the cache . the first # input # run consists of the first 4 contiguous records . the first rid in the first rid run is  3 and @xmath16 $ ] is in the cache , so one finds this record , # input[3 ] # , in cache and copies it to the buffer maintained in cache for the first # internal # run , and so on . at the end of phase  ii , the records in the first # input # run with key values in the sequence of i , l , f , c are reordered as records in the first # internal # run with key values in the sequence of c , f , i , l. the reordering is done according to a permutation specified by the first rid run , 3 , 2 , 0 , 1 .    during * phase iii * , we will use the original rid sequence in the list of key - rid pairs to _ gather _ ( merge ) records from all # internal # run . for example , upon reading  5 , we go to the second # internal # run to _ gather _ the record ; upon reading  7 , we go to the second # internal # run to _ gather _ the record ; upon reading  3 , we go to the first # internal # run to _ gather _ the record ; upon reading  8 we go to the third # internal # run to _ gather _ the record , and so on . for this phase , we maintain a buffer for the key - rid list , a buffer for # output # in cache and a buffer for each # internal # run . all buffers are in cache . 0    .... % $ ( for sake of emacs ) let $ l\gets$ ( cache_size/2 ) let $ n\gets$ num_records let num_runs $ \gets n / l$    \rm input :   integer \tt rid[num_rids ] ,      \rm record \tt input_rec[num_records ] ; \rm output : record \tt output_rec[num_rids ] ; \rm parameters : \rm integer \tt rid_run[num_runs][]\tt ,          \rm record \tt record_run[num_runs ] [ ] ;    //phase i :   { \it distribute } rid array into runs //           with each run of length $ l$ % for $ i$ = 1 , ... , num_runs do %    initialize write buffer of rid_run[$i$ ] for each rid , $ r$ , in rid do \ {   set { \it run\_num } = $ \lceil{}r / l\rceil$ ;   append $ r$ to rid_run[{\it run\_num } ] \ } //phase ii :   { \it probe } partitions of input_rec % ( conceptually partition input_rec into partitions of length $ l$ and %   for each partition , $ i$ , of input_rec , probe for each rid in rid_run[$i$ ] ) for $ i$ = 1 , ... , num_runs do \ { %    initialize read buffer for rid_run[$i$ ]    read into memory all records from     input_rec[$(i-1)\!*\!\!l\ ! + \ ! 1 $ ] to input_rec[$i\!*\!\!l$ ]    allocate memory for $ l$ records      to be stored in record_run[i ]    for each rid , $ r$ , in rid_run[$i$ ]      append input_rec[$r$ ] to record_run[$i$ ]    write out record_run[$i$ ] to disk \ } //phase iii : { \it gather } records from record_run [ ] //   into output_rec in same order as rid [ ] % for $ i$ = 1 , ... , num_runs do %    initialize read buffer of record_run[$i$ ] for each rid , $ r$ , in rid do \ {    set { \it run\_num } = $ \lceil{}r / l\rceil$ ;    read next record from record_run[{\it run\_num } ]    append record to output_rec \ } ....      implicit in the description of the dpg algorithm is that the input sequence of rids is distributed uniformly among the set of all rids of the input data file . this is always the case when dpg is applied to retrieve records after sorting key - pointer pairs . in that situation , the key - pointer pairs act as a permutation vector to permute the records in the input data file . if the input sequence of rids is not uniformly distributed , then some rid runs will be larger than other runs . as a consequence , in phase  ii , when the partition of the temporary data file ( the partition of # record_run # in figure  [ fig : dpgalgo ] ) may be larger than the size of the cache . if only a few of the partitions of the temporary data file are larger than cache then the overall running time is not greatly affected . if there is a great deal of data skew and many of the temporary partitions # record_run # are larger than cache , then the @xmath9 partitions of the input sequence of rids must be chosen on some other basis than the high order bits of the page number . in such cases , one can invoke the data skew handling techniques of dewitt et  al .  @xcite . their solution , reformulated in our context , is to sample the rids from the rid sequence . the sampled set of rids is then sorted , and partitions of the rids are chosen so as to evenly partition the sampled set . as discussed in the introduction , dpg acts as an accelerator for many main memory sorting algorithms . recall that main memory sorts typically proceed in three phases :    1 . extraction of key - pointer ; 2 . sorting of the key - pointer pairs ; and 3 . copying of the original records into the destination array according to the sorted key - pointer pairs . alphasort  @xcite and superscalarsort  @xcite are examples of this three - phase sorting paradigm . both sorting algorithms can be considered as main memory sorting algorithms .    in principle , the sorting algorithms are single - pass disk - based sorting algorithms . both sorting algorithms were introduced as an answer to the datamation sorting challenge  @xcite . the datamation challenge dictates that one is given one million records of 100  bytes . each record has a 10  byte key . the keys are uniformly distributed . at the time of the datamation challenge , external sorting algorithms were required . on today s computers , the data file of 100  mb easily fits in main memory . hence , the only disk - related portion of the datamation challenge is to overlap disk  i / o with cpu operation . disk striping has the potential to provide very fast disk  i / o . this occurs because the disks are accessed in parallel . in this situation , main memory data retrieval becomes the bottleneck . the dpg algorithm pushes back this main memory bottleneck . by using dpg for data movement , the largest cost of main memory sorting is reduced . we have reimplemented the main memory portion of superscalarsort , both with and without dpg . we use the ideas of dpg to present three new join algorithms : 1.c , 2 and  3 . algorithms 1.a , 1.b and  4 , will be included in the experimental section  [ sec : joinexperiments ] for completeness .    1 . sort - merge join 1 . sort - merge join with alphasort ( sort based on  @xcite ) 2 . sort - merge join with superscalarsort ( sort based on  @xcite ) 3 . sort - merge join with dpg sort ( sort based on dpg ; see  [ sec : sortmergejoin ] ) 2 . dpg - sort join ( see section  [ sec : foreignkeyjoin ] ) 3 . dpg - move join ( see section  [ sec : foreignkeyjoin ] ) 4 . radix join ( from  @xcite )      the well - known sort - merge join was introduced by blasgen and eswaran  @xcite . there are two steps in sort - merge joins : sort two relations on the join key and scan the sorted relations to do a merge on the join key . applying dpg sort at the first step provides a new faster sort - merge join method , sort - merge join with dpg sort . in sections  [ sec : sortmergejoinexperiments ] and [ sec : joinexperiments ] , we experimentally compare different versions of sort - merge join , according to the sorting methods used . specifically , we consider using dpg sort , alphasort and superscalarsort for the sorting step . we next consider joins in which the join key is a foreign key , and it has an index . we denote by  @xmath3 a non - indexed relation . we denote by  @xmath4 an indexed relation . the notation is motivated by the example of a foreign key join . in a foreign key join , the join key is the same as the foreign key . so , the join key is a set of attributes in the relation  @xmath3 that refers to a foreign key from relation  @xmath4 . a _ join triple _ is a triple ( @xmath0 , rid@xmath1 , rid@xmath2 ) , such that @xmath0 is a key value , rid@xmath1 is the rid of a record from  @xmath3 with key  @xmath0 , and rid@xmath2 the rid of a record from  @xmath4 with key  @xmath0 . there are three steps in a foreign key join algorithm with dpg . the first step is to construct join triples . the second step is to use the join triples to copy one of the two relations into a temporary file according to an order derived from the join triples . the third step is to join the temporary file with the remaining relation . we describe the second and third steps initially in section  [ sec : foreignkeyjoin ] . we then return to the more technical problem of efficiently constructing join triples in sections  [ sec : jointriples ] and  [ sec : indexlookup ] . this section describes two dpg join algorithms . it assumes that one has already constructed the join triples . some algorithms for constructing the join triples are described later in sections  [ sec : jointriples ] and  [ sec : indexlookup ] . assume that one has generated the join triples ( @xmath0 , rid@xmath1 , rid@xmath2 ) . it is now possible to ignore the key  @xmath0 , and deal directly with the rid pairs ( rid@xmath1 , rid@xmath2 ) . we wish to satisfy one of two goals :    1 . [ item : movef ] * dpg - move join : * move the records of  @xmath4 into a temporary file according to the ordering of the records of  @xmath3 . [ item : mover ] * dpg - sort join : * move the records of  @xmath3 into a temporary file according to the ordering of the records of  @xmath4 . so , the rid pairs ( rid@xmath1 , rid@xmath2 ) will be sorted according to rid@xmath2 . first consider dpg - move join . we will see how to generate the join triples in the order of rid@xmath1 . this is done by scanning the records of the relation  @xmath3 in file order ( in order of increasing rid@xmath1 ) . therefore the rid pairs can be used directly as part of a dpg algorithm . the second component of the pair , rid@xmath2 , is the sequence of rids according to which we want to move the records of  @xmath4 . this algorithm does not require any sorting or hashing . hence , we call it dpg - move join . next consider dpg - sort join . in this version , we first sort the rid pairs ( rid@xmath1 , rid@xmath2 ) according to the order of rid@xmath2 . this is done , for example , with superscalarsort and dpg . the algorithm then reduces to record movement in which we wish to move the records of  @xmath3 according to the ordering of  rid@xmath2 in the sorted sequence ( rid@xmath1 , rid@xmath2 ) . note that dpg - move join is preferred when the relation  @xmath4 is smaller . dpg - sort join is preferred when the relation  @xmath3 is smaller . the simplest solution for a foreign key join is to do a file scan of  @xmath3 , and for each record of  @xmath3 to extract the join key and do an index lookup in the index of  @xmath4 . one can then join the record of  @xmath3 with the corresponding record of  @xmath4 . this involves random access , and is economical only if the join produces very few records . in particular , this will be the case only if the number of records of  @xmath3 is small . a better solution is to do a file scan of  @xmath3 , and to use the index on  @xmath4 to create join triples . to construct the join triples , one scans the relation  @xmath3 . for each record of  @xmath3 , one extracts the corresponding rid and associated join key  @xmath0 . one then looks up the key  @xmath0 in the index of  @xmath4 . the index lookup yields the final element of the triple , rid@xmath2 . note that the join triple is constructed at the cost of a file scan of  @xmath3 and an index lookup in the index of  @xmath4 for each record of  @xmath3 . usually an index on a data file is much smaller than the full data file . if the index fits entirely in cache , then the index lookup will be significantly cheaper than the file scan . unfortunately , the indexes in many main memory databases generally do not fit in cache . in such cases , the index lookup in the index of  @xmath4 will dominate the costs . for example , in a b+  tree indexing @xmath8  records , if an internal node has @xmath17  children , then @xmath18 nodes of the b+  tree must be accessed . each such access will be a random access in main memory . most of the random accesses imply a cache miss . the cost of so many random accesses makes a naive index lookup uneconomical . even if the index is a hash index , at least one random access in memory will be required . luckily , it is possible to execute the index lookup faster than the above analysis would indicate . this is because the construction of the join triples requires many index lookups , with no intervening record accesses . for purposes of join triple construction , _ batch lookup _ of keys in an index suffices . by batch lookup , we assume that an array of join keys is first extracted by scanning the data file of  @xmath3 . the batch lookup then produces an array of rids for  @xmath4 through the use of the index on  @xmath4 . we show that the index lookups can be reorganized into a two - pass algorithm . two such two - pass algorithms are demonstrated : one for b+ tree indexes , and one for hash indexes . note that batch lookup of rids in an index can be substantially faster than individual lookup . rao and ross had previously discussed cache conscious indexes for main memory  @xcite . there they present css - trees , which have better cache behavior than either b+  trees or hash indexes . however , they only consider individual lookup of keys in an index , one at a time . in their scenario , a second key is not looked up until the index lookup of the first key has been resolved . [ [ batch - lookup - in - b - tree - indexes ] ] * batch lookup in b+ tree indexes * + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +    for simplicity , we describe the two - pass lookup for an enhanced b+  tree . the notion of _ enhanced b+  tree _ was introduced by rao and ross  @xcite . the idea is that all slots of a b+  tree node are used . this is similar to compact b - trees  @xcite or to the isam method introduced by ibm  @xcite . in the context of a general b+  tree this can be accomplished by maintaining updates to the b+  tree in a separate index , and then doing a batch update by reorganizing the b+  tree . for brevity , we will sometimes refer to b+  tree , although in all cases , an enhanced b+  tree is intended . assume that we are executing a file scan of  @xmath3 with the purpose of constructing the join triples . we collect a sequence of keys from the relation  @xmath3 , and we wish to carry out a batch lookup of rids in the enhanced b+-tree . the rids will be used to retrieve records from the foreign relation  @xmath4 . we assume that the b+  tree does not fit in cache . our goal is a two - pass algorithm which will efficiently accomplish batch lookup . assume that the b+  tree has @xmath17  entries per node . assume that the b+  tree indexes @xmath8  records . then there are @xmath18 levels . assume further that the first @xmath19 levels ( the top half of the b+  tree ) fit in cache . for @xmath20 , the top half of the tree has approximately @xmath21 slots . for example , if there are @xmath22 records , then there are 32,000 slots , which clearly fit inside cache . the strategy is a two - pass strategy . in the first pass , one performs a lookup of each key , but only using the top half of the b+  tree . as shown in figure  [ fig : btree ] , each leaf of the subtree comprising this upper half can be considered as the root of a second subtree comprising nodes in the lower half of the b+  tree . thus , at the end of this first pass , a key can be associated with a leaf of the upper subtree , which is also a root of one of the lower subtrees . after the first pass , one can associate each key with a subtree within the lower half of the b+  tree . so , in the second pass , one loads a subtree from the lower half . one then continues the lookup for all keys associated with the root of the subtree in the lower half . at the end , one then has a sequence of keys and rids . if one wishes to have the keys in the same order as the order of the original rids , then this can also be arranged . in this case , one extends the previous scenario to use the dpg algorithm . the first phase of the dpg algorithm is to _ distribute _ the keys into runs . in the initial lookup of a key in the first half , it was associated with a root of a subtree in the first half . the particular root of a subtree identifies the run into which the key is copied . the second phase of the dpg algorithm is to use the key to _ probe _ the index , in order to find the rid . one completes the lookup of all keys in a run associated with a particular subtree of the b+  tree , before proceeding to the next subtree in the lower half . the resulting rids are stored in a temporary partition or run . this is exactly what was described earlier . finally , the third phase of the dpg algorithm _ gathers _ the rids into a destination array in the same order as that of the original keys . hence , we have completed a batch lookup of the keys , and returned an array of rids in the same order as that of the original keys . [ [ batch - lookup - in - hash - indexes ] ] * batch lookup in hash indexes * + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +    batch lookup of hash indexes also proceeds in two passes . we assume that the hash array of the hash index stores at each hash entry one key - rid pair . a second hash array associated with the index stores pointers to overflow key - rid pairs that would have collided with an occupied slot in the first hash array . the two - pass lookup for hash indexes proceeds in a very simple manner . we extract the sequence of keys from  @xmath3 . as the keys are extracted , the hash values are computed . those key - hash value pairs are saved in an array in an order corresponding to the rid order of  @xmath3 . it now suffices to apply the dpg algorithm . the hash array is partitioned into sets of @xmath7  hash slots . in the distribute phase , the hash value acts as an index into the hash array . hence , this becomes the permutation vector of the dpg algorithm . the key and hash value are then written into separate runs according to the partition of the hash array . there is one run of key - hash values for each partition of the hash array .    in the probe phase , a run of hash values is loaded into cache along with the corresponding partition of the hash array . as part of the probe phase , the key and hash values from the run are used to look up the the corresponding rid in the partition of the hash array . the rids are then saved in a temporary partition , in the same order as the order in which the key - hash values of the original partition are stored . finally , in the gather phase , the ordering of the key - hash value pairs are used to gather the rids from the temporary partition . as in section [ sec : dpg ] , the rids are gathered into a destination array in an order corresponding the ordering of the original key - hash value pairs . figure  [ fig : sorting ] demonstrates the acceleration achieved by superscalarsort when dpg is used . the results are demonstrated on the ibm pseries 690 turbo . the ibm  p690 has an l3  cache of size 128  mb . hence , in order to realistically demonstrate dpg , we were forced to increase the size of the database . we chose to implement superscalarsort for a data file of size 512  mb . the record size was treated as a variable , to illustrate the influence of record size . as in the original datamation challenge , the key is 10  bytes . note that the additional speed of superscalarsort with dpg is most pronounced for smaller record sizes . for record sizes of 256  bytes and higher , dpg provides only a small advantage . this is because the ibm p690 has an l3  cache block of size 512  bytes . so , for record sizes below 256  bytes , a cache miss incurs significant overhead in loading a 512  byte cache block . there was some variability in the results because the data was taken on a time - shared , shared memory machine . on the ibm  p690 , the l2  cache is shared among two cpus , and the l3  cache is shared among eight cpus . our experiments were run on a single cpu . hence another process on a neighboring cpu could consume some of our cache , thereby affecting the timings . the reported experimental results are the averages of three runs each . we implement two sort - merge join methods : sort - merge join with dpg sort and sort - merge join with superscalarsort . as defined in section  [ sec : sortmergejoin ] , sort - merge join with dpg sort applies the dpg sort for the sorting phase and sort - merge join with superscalarsort applies superscalarsort for the sorting phase . as expected , the acceleration in the speed of sort - merge join in figure  [ fig : sortmerge ] follows the same pattern as that of figure  [ fig : sorting ] for sorting . in this case , we illustrate our results on a database of size 128  mb on the ibm p690 . the ibm p690 has an l2  cache of size 1.4  mb . so , in this case , the l3  cache is acting as the main memory , while l2  cache is acting as the `` cache '' .      in figures  [ fig : join_nonuniform_1 ] through  [ fig : join_dup_3 ] , we now experimentally compare the six join methods originally presented at the begining of section  [ sec : joinalgodpg ] . the three join algorithms labelled sort - merge join with dpg sort , dpg - sort join , and dpg - move join are all new . the remaining algorithms , sort - merge join with alphasort , sort - merge join with superscalarsort , and radix join , are all based on sorting or join algorithms from the literature . as explained earlier , we denote by  @xmath4 an indexed relation and  @xmath3 a relation that has foreign key on the indexed attribute of  @xmath4 . we consider both uniform and non - uniform distribution of foreign key values . [ [ the - non - uniform - distribution - of - join - key - values . ] ] * the non - uniform distribution of join key values . * + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +    sort - merge join with alphasort and dpg - sort are the only two of the previously discussed join methods that operate correctly for a non - uniform distribution of join key values .    in the following we use count bucket sort for bucket sort . the count bucket sort is as follows :    1 . count the number of elements destined for each bucket . 2 .   set bucket boundaries according to the statics computed and distribute elements to buckets . alphasort works for non - uniformly distributed data , because it uses quicksort to sort each run and uses replacement - selection to merge the runs . dpg - sort join works for non - uniformly distributed data too , because we use count bucket sort to sort rids . dpg - sort join sorts the rids according to the lowest   @xmath23 bits is enough , @xmath8 is the number of records . in the simulation , we assign the rid values in the range @xmath24 $ ] , the lowest @xmath23 bits is sufficient for sorting . for example , for @xmath25 sorting rids according to the lowest 20 bits is enough . this could be done in two steps with count bucket sort : first do count bucket sort according to the lower 10 bits , then do count bucket sort according to the higher 10 bits .    using the unix @xmath26 and @xmath27 functions we generate an exponential distribution of the data as @xmath28 ( @xmath29 is a constant and in our experiments we assign c with @xmath30 ) . a comparison of sort - merge join with alphasort and dpg - sort join on three different computer architectures is provided in the figures  [ fig : join_nonuniform_1 ] ,  [ fig : join_nonuniform_2 ] , and  [ fig : join_nonuniform_3 ] . the three different architectures are the ibm power4 pseries 690 turbo , the 3.06 ghz pentium 4 with rambus pc-1200 ram , and the 2.6 ghz pentium 4 with ddr-266 ram , from the comparison , we can see that dpg - sort join is much faster than sort - merge join with alphasort . [ [ the - uniform - distribution - of - join - key - values . ] ] * the uniform distribution of join key values . * + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +    first we will show how all algorithms work for uniformly distributed join key values . superscalarsort is a key - prefix - sort explained further in section  [ sec : intro ] . it assumes that data is uniformly distributed according to the highest  7 bits of the key . this kind of distribution can be applied to all dpg algorithms , because a unform distribution of the join key values is the only constraint of sort - merge join with dpg and dpg - move join . we also implement the radix join of manegold et al . they describe a variation of hash join for main memory . radix join also requires a uniform distribution of the join key values . otherwise , some of their partitions will be too large to fit into the l2 cache and how to set the boundaries is unknown . we produce uniformly distributed foreign key values using the unix @xmath26 function . a comparison of the different join methods on three different computer architectures as before is provided in the figures . figure  [ fig : join_no_dup ] reflects data with no duplicate join key values in the relation  @xmath3 . figures  [ fig : join_dup_1 ] ,  [ fig : join_dup_2 ] , and  [ fig : join_dup_3 ] , show the same information in which duplicate join key values are allowed . from the comparison , we can see that dpg - move join and radix join are the fastest . dpg - move is better for large records and radix join is better for small records . the use of dpg in the sorting provides an accelerator for existing sorting algorithms . especially for the smaller record sizes , such as 32  bytes and 64  bytes , the performance improvements are really impressive .    for the more common case of non - uniform distribution of join key values , dpg - sort join works better than sort - merge join with alphasort across all the tested platforms . for smaller records sizes , such as , 32 , 64 , dpg - sort join is much better than sort - merge join with alphasort . more impressively , on the newer platform , rambus pc-1200 ram , dpg - sort works better than on pc with ddr-266 ram and even works better for larger record sizes , for example 512  bytes .    for special case of uniform distribution of join key values , dpg - move join and radix join are the best . the remaining dpg algorithms are also competitive with older algorithms although with a smaller improvement . the dpg algorithm can be easily generalized to multiple passes . this can be useful when there is a very small cache in relation to the size of main memory . however , we do not encounter this scenario in our current experiments . we would like to thank betty salzberg and donghui zhang for extensive conversations and insights into additional situations where distribute - probe - gather can be beneficial . we also gratefully acknowledge the use of the support for the computations on the ibm p690 by the scientific computing and visualization ( scv ) group at boston university . a. ailamaki , d.j . dewitt , m.d . hill , d.a . wood , `` dbmss on a modern processor : where does time go ? '' , _ vldb99 , proceedings of 25th international conference on very large data bases _ , 1999 , pp .  266 - 277 . l.  arge , o.  procopiuc , s.  ramaswamy , t.  suel and j.s . vitter , `` scalable sweeping - based spatial join '' , _ vldb 1998 , proceedings of 24th international conference on very large databases _ , 1998 , pp . 570581 . dittrich , b.  seeger , d.s . taylor and p.  widmayer , `` progressive merge join : a generic and non - blocking sort - based join algorithm '' , _ vldb 2002 , proceedings of 28th international conference on very _ , 2002 .        web site : `` latency vs. bandwidth , a performance analysis '' , http://www.lostcircuits.com/memory/latency/2.shtml ; `` inside the eddr chip : combining dram storage and sram speed '' , http://www.lostcircuits.com/memory/eddr/ , nov . 27 , 2000 ; and `` high performance ddr dimms '' , http://www.lostcircuits.com/memory/ddr2/ , july  17 , 2001 . k.  keeton , d.a . patterson , y.q . raphael and w.e . baker `` performance characterization of a quad pentium pro smp using oltp workloads '' _ porc . of the int . syp . on computer architecture _ , p0p .  1526 , barcelona , spain , 1998 . s.  manegold , p.a . boncz and m.l . kersten , `` what happens during a join ? dissecting cpu and memory optimization effects '' , _ vldb 2000 , proceedings of 26th international conference on very large data bases , september 10 - 14 , 2000 , cairo , egypt _ , 2000 , pp . 339350 . p. trancoso j.l . larriba - pey , z.  zhang and j.  torellas , `` the memory performance of dss commercial workloads in shared - memory multiprocessors '' , _ int . symp . on high performance computer architecture _ , san antonio , tx , usa 1997 .
Abstract:

Answer: retrieval of records on disk is well - known to be at the heart of many database problems . we show that the corresponding movement of records in main memory has now become a severe bottleneck for many database operations . this is due to the stagnating latency of main memory , even while cpu speed , main memory bandwidth , and disk speed all continue to improve . as a result , record movement has become the dominant cost in main memory sorting . we present a new algorithm for fast record retrieval , _ distribute - probe - gather _ , or dpg . dpg has important applications both in sorting and in joins . current main memory sorting algorithms split their work into three phases : extraction of key - pointer pairs ; sorting of the key - pointer pairs ; and copying of the original records into the destination array according the sorted key - pointer pairs . the copying in the last phase dominates today s sorting time . hence , the use of dpg in the third phase provides an accelerator for existing sorting algorithms . dpg also provides two new join methods for foreign key joins : dpg - move join and dpg - sort join . the resulting join methods with dpg are faster because dpg join is cache - efficient and at the same time dpg join avoids the need for sorting or for hashing . the ideas presented for foreign key join can also be extended to faster record pair retrieval for spatial and temporal databases .