Instruction: Write an abstract for the following acadamic paper.

Input:
Paper: we have recorded the flight of a fly during take off and landing using digital high speed photography . it is shown that the dynamics of flexible wings are different for these procedures . during this observation fly flew freely in a big box and it was not tethered .
Abstract: in this fluid dynamics video , we demonstrated take off and landing of a fly . the deformation of wings is in focus in this video .

Paper: a central factor in the application of machine learning to a given task is the _ inductive bias _ , _ i.e. _  the choice of hypotheses space from which learned functions are taken . the restriction posed by the inductive bias is necessary for practical learning , and reflects prior knowledge regarding the task at hand . perhaps the most successful exemplar of inductive bias to date manifests itself in the use of convolutional networks ( @xcite ) for computer vision tasks . these hypotheses spaces are delivering unprecedented visual recognition results  ( _ e.g. _  @xcite ) , largely responsible for the resurgence of deep learning  ( @xcite ) . unfortunately , our formal understanding of the inductive bias behind convolutional networks is limited    the assumptions encoded into these models , which seem to form an excellent prior knowledge for imagery data , are for the most part a mystery . existing works studying the inductive bias of deep networks ( not necessarily convolutional ) do so in the context of _ depth efficiency _ , essentially arguing that for a given amount of resources , more layers result in higher expressiveness . more precisely , depth efficiency refers to a situation where a function realized by a deep network of polynomial size , requires super - polynomial size in order to be realized ( or approximated ) by a shallower network . in recent years , a large body of research was devoted to proving existence of depth efficiency under different types of architectures ( see for example  @xcite ) . nonetheless , despite the wide attention it is receiving , depth efficiency does not convey the complete story behind the inductive bias of deep networks . while it does suggest that depth brings forth functions that are otherwise unattainable , it does not explain why these functions are useful . loosely speaking , the hypotheses space of a polynomially sized deep network covers a small fraction of the space of all functions . we would like to understand why this small fraction is so successful in practice . a specific family of convolutional networks gaining increased attention is that of _ convolutional arithmetic circuits_. these models follow the standard paradigm of locality , weight sharing and pooling , yet differ from the most conventional convolutional networks in that their point - wise activations are linear , with non - linearity originating from product pooling . recently ,  analyzed the depth efficiency of convolutional arithmetic circuits , showing that besides a negligible ( zero measure ) set , all functions realizable by a deep network require exponential size in order to be realized ( or approximated ) by a shallow one . this result , termed _ complete depth efficiency _ , stands in contrast to previous depth efficiency results , which merely showed _ existence _ of functions efficiently realizable by deep networks but not by shallow ones . besides their analytic advantage , convolutional arithmetic circuits are also showing promising empirical performance . in particular , they are equivalent to simnets    a deep learning architecture that excels in computationally constrained settings  ( ) , and in addition , have recently been utilized for classification with missing data  ( ) . motivated by these theoretical and practical merits , we focus our analysis in this paper on convolutional arithmetic circuits , viewing them as representative of the class of convolutional networks . we empirically validate our conclusions with both convolutional arithmetic circuits and _ convolutional rectifier networks _   convolutional networks with rectified linear ( relu ,  @xcite ) activation and max or average pooling . adaptation of the formal analysis to networks of the latter type , similarly to the adaptation of the analysis in   carried out by  , is left for future work . our analysis approaches the study of inductive bias from the direction of function inputs . specifically , we study the ability of convolutional arithmetic circuits to model correlation between regions of their input . to analyze the correlations of a function , we consider different partitions of input regions into disjoint sets , and ask how far the function is from being separable w.r.t .  these partitions . distance from separability is measured through the notion of _ separation rank _  ( @xcite ) , which can be viewed as a surrogate of the @xmath0 distance from the closest separable function . for a given function and partition of its input , high separation rank implies that the function induces strong correlation between sides of the partition , and vice versa . we show that a deep network supports exponentially high separation ranks for certain input partitions , while being limited to polynomial or linear ( in network size ) separation ranks for others . the network s pooling geometry effectively determines which input partitions are favored in terms of separation rank , _ i.e. _  which partitions enjoy the possibility of exponentially high separation rank with polynomial network size , and which require network to be exponentially large . the standard choice of square contiguous pooling windows favors interleaved ( entangled ) partitions over coarse ones that divide the input into large distinct areas . other choices lead to different preferences , for example pooling windows that join together nodes with their spatial reflections lead to favoring partitions that split the input symmetrically . we conclude that in terms of modeled correlations , pooling geometry controls the inductive bias , and the particular design commonly employed in practice orients it towards the statistics of natural images ( nearby pixels more correlated than ones that are far apart ) . moreover , when processing data that departs from the usual domain of natural imagery , prior knowledge regarding its statistics can be used to derive respective pooling schemes , and accordingly tailor the inductive bias .    with regards to depth efficiency , we show that separation ranks under favored input partitions are exponentially high for all but a negligible set of the functions realizable by a deep network . shallow networks on the other hand , treat all partitions equally , and support only linear ( in network size ) separation ranks . therefore , almost all functions that may be realized by a deep network require a replicating shallow network to have exponential size . by this we return to the complete depth efficiency result of  , but with an added important insight into the benefit of functions brought forth by depth   they are able to efficiently model strong correlation under favored partitions of the input . the remainder of the paper is organized as follows . [ sec : prelim ] provides a brief presentation of necessary background material from the field of tensor analysis . [ sec : cac ] describes the convolutional arithmetic circuits we analyze , and their relation to tensor decompositions . in sec . [ sec : sep_rank ] we convey the concept of separation rank , on which we base our analyses in sec .  [ sec : analysis ] and  [ sec : inductive_bias ] . the conclusions from our analyses are empirically validated in sec . [ sec : exp ] . finally , sec . [ sec : discussion ] concludes . the analyses carried out in this paper rely on concepts and results from the field of tensor analysis . in this section we establish the minimal background required in order to follow our arguments , referring the interested reader to  @xcite for a broad and comprehensive introduction to the field . the core concept in tensor analysis is a _ tensor _ , which for our purposes may simply be thought of as a multi - dimensional array . the _ order _ of a tensor is defined to be the number of indexing entries in the array , which are referred to as _ modes_. the _ dimension _ of a tensor in a particular mode is defined as the number of values that may be taken by the index in that mode . for example , a @xmath1-by-@xmath2 matrix is a tensor of order  @xmath3 , _ i.e. _  it has two modes , with dimension  @xmath1 in mode  @xmath4 and dimension  @xmath2 in mode  @xmath3 . if @xmath5 is a tensor of order @xmath6 and dimension @xmath7 in each mode @xmath8:=\{1,\ldots , n\}$ ] , the space of all configurations it can take is denoted , quite naturally , by @xmath9 .    a fundamental operator in tensor analysis is the _ tensor product _ , which we denote by @xmath10 . it is an operator that intakes two tensors @xmath11 and @xmath12 ( orders @xmath13 and @xmath14 respectively ) , and returns a tensor @xmath15 ( order @xmath16 ) defined by : @xmath17 . notice that in the case @xmath18 , the tensor product reduces to the standard outer product between vectors , _ i.e. _  if @xmath19 and @xmath20 , then @xmath21 is no other than the rank-1 matrix @xmath22 . we now introduce the important concept of matricization , which is essentially the rearrangement of a tensor as a matrix . suppose @xmath5 is a tensor of order @xmath6 and dimension @xmath7 in each mode @xmath8 $ ] , and let @xmath23 be a partition of @xmath24 $ ] , _ i.e. _  @xmath25 and  @xmath26 are disjoint subsets of @xmath24 $ ] whose union gives  @xmath24 $ ] . we may write @xmath27 where @xmath28 , and similarly @xmath29 where @xmath30 . the _ matricization of @xmath5 w.r.t . the partition @xmath23 _ , denoted @xmath31 , is the @xmath32-by-@xmath33 matrix holding the entries of @xmath5 such that @xmath34 is placed in row index @xmath35 and column index @xmath36 . if @xmath37 or @xmath38 , then by definition @xmath31 is a row or column ( respectively ) vector of dimension @xmath39 holding @xmath34 in entry @xmath40 . a well known matrix operator is the _ kronecker product _ , which we denote by @xmath41 . for two matrices @xmath42 and @xmath43 , @xmath44 is the matrix in @xmath45 holding @xmath46 in row index @xmath47 and column index @xmath48 . let @xmath5 and @xmath49 be tensors of orders @xmath13 and @xmath14 respectively , and let @xmath23 be a partition of @xmath50 $ ] . the basic relation that binds together the tensor product , the matricization operator , and the kronecker product , is : _ i , j=_i , j_(i - p),(j - p ) [ eq : mat_tensor_prod ] where @xmath51 and @xmath52 are simply the sets obtained by subtracting @xmath13 from each of the elements in  @xmath25 and  @xmath26 respectively . in words , eq . [ eq : mat_tensor_prod ] implies that the matricization of the tensor product between @xmath5 and @xmath49 w.r.t . the partition @xmath23 of @xmath50 $ ] , is equal to the kronecker product between two matricizations : that of  @xmath5 w.r.t . the partition of  @xmath53 $ ] induced by the lower values of @xmath23 , and that of  @xmath49 w.r.t . the partition of  @xmath54 $ ] induced by the higher values of @xmath23 . the convolutional arithmetic circuit architecture on which we focus in this paper is the one considered in  , portrayed in fig .  [ fig : nets_patterns](a ) . instances processed by a network are represented as @xmath6-tuples of @xmath55-dimensional vectors . they are generally thought of as images , with the @xmath55-dimensional vectors corresponding to local patches . for example , instances could be @xmath56-by-@xmath56 rgb images , with local patches being @xmath57 regions crossing the three color bands . in this case , assuming a patch is taken around every pixel in an image ( boundaries padded ) , we have @xmath58 and @xmath59 . throughout the paper , we denote a general instance by @xmath60 , with @xmath61 standing for its patches .    the first layer in a network is referred to as _ representation_. it consists of applying  @xmath62 _ representation functions _ @xmath63 to all patches , thereby creating  @xmath62 feature maps . in the case where representation functions are chosen as @xmath64 , with parameters @xmath65 and some point - wise activation  @xmath66 , the representation layer reduces to a standard convolutional layer . more elaborate settings are also possible , for example modeling the representation as a cascade of convolutional layers with pooling in - between . following the representation , a network includes  @xmath67 hidden layers indexed by @xmath68 . each hidden layer  @xmath69 begins with a _ @xmath70 conv _ operator , which is simply a three - dimensional convolution with  @xmath71 channels and filters of spatial dimensions @xmath4-by-@xmath4 .  conv operator . the first , referred to as _ weight sharing _ , is the one described above , and corresponds to standard convolution . the second is more general , allowing filters that slide across the previous layer to have different weights at different spatial locations . it is shown in   that without weight sharing , a convolutional arithmetic circuit with one hidden layer ( or more ) is universal , _ i.e. _  can realize any function if its size ( width ) is unbounded . this property is imperative for the study of depth efficiency , as that requires shallow networks to ultimately be able to replicate any function realized by a deep network . in this paper we limit the presentation to networks with weight sharing , which are not universal . we do so because they are more conventional , and since our entire analysis is oblivious to whether or not weights are shared ( applies as is to both settings ) . the only exception is where we reproduce the depth efficiency result of  . there , we momentarily consider networks without weight sharing . ] this is followed by spatial pooling , that decimates feature maps by taking products of non - overlapping two - dimensional windows that cover the spatial extent . the last of the  @xmath67 hidden layers ( @xmath72 ) reduces feature maps to singletons ( its pooling operator is global ) , creating a vector of dimension  @xmath73 . this vector is mapped into @xmath74 network outputs through a final dense linear layer . altogether , the architectural parameters of a network are the type of representation functions ( @xmath75 ) , the pooling window shapes and sizes ( which in turn determine the number of hidden layers @xmath67 ) , and the number of channels in each layer ( @xmath62 for representation , @xmath76 for hidden layers , @xmath74 for output ) . given these architectural parameters , the learnable parameters of a network are the representation weights ( @xmath77 for channel @xmath78 ) , the conv weights ( @xmath79 for channel @xmath80 of hidden layer @xmath69 ) , and the output weights ( @xmath81 for output node @xmath82 ) .    for a particular setting of weights , every node ( neuron ) in a given network realizes a function from  @xmath83 to  @xmath84 . the _ receptive field _ of a node refers to the indexes of input patches on which its function may depend . for example , the receptive field of node  @xmath85 in channel  @xmath80 of conv operator at hidden layer  @xmath86 is @xmath87 , and that of an output node is @xmath24 $ ] , corresponding to the entire input . denote by  @xmath88 the function realized by node  @xmath85 of channel  @xmath80 in conv operator at hidden layer  @xmath69 , and let @xmath89 $ ] be its receptive field . by the structure of the network it is evident that @xmath90 does not depend on  @xmath80 , so we may write  @xmath91 instead . moreover , assuming pooling windows are uniform across channels ( as customary with convolutional networks ) , and taking into account the fact that they do not overlap , we conclude that @xmath92  and  @xmath93 are necessarily disjoint if  @xmath94 . a simple induction over @xmath68 then shows that  @xmath88 may be expressed as @xmath95 , where @xmath96 stands for the receptive field @xmath91 , and  @xmath97 is a tensor of order  @xmath98 and dimension  @xmath62 in each mode , with entries given by polynomials in the network s conv weights @xmath99 . taking the induction one step further ( from last hidden layer to network output ) , we obtain the following expression for functions realized by network outputs : h_y(x_1,  ,x_n)=_d_1  d_n=1^ma_d_1  d_n^y_i=1^n f__d_i(x_i ) [ eq : h_y ] @xmath100 $ ] here is an output node index , and  @xmath101 is the function realized by that node . @xmath102  is a tensor of order  @xmath6 and dimension  @xmath62 in each mode , with entries given by polynomials in the network s conv weights @xmath99 and output weights @xmath81 . hereafter , terms such as _ function realized by a network _ or _ coefficient tensor realized by a network _ , are to be understood as referring to  @xmath101 or  @xmath102 respectively . next , we present explicit expressions for  @xmath102 under two canonical networks    deep and shallow . [ [ deep - network . ] ] deep network . + + + + + + + + + + + + +    consider a network as in fig . [ fig : nets_patterns](a ) , with pooling windows set to cover four entries each , resulting in @xmath103 hidden layers . the linear weights of such a network are @xmath104}$ ] for conv operator in hidden layer  @xmath86 , @xmath105}$ ] for conv operator in hidden layer @xmath106 , and @xmath107}$ ] for dense output operator . they determine the coefficient tensor  @xmath102  ( eq .  [ eq : h_y ] ) through the following recursive decomposition : _ & = & _ = 1^r_0 a_^1 , ^4 a^0 , , + & & + _ & = & _ = 1^r_l-1 a_^l , ^4 ^l-1 ,  , l\{2  l-1 } , + & & + _ & = & _ = 1^r_l-1 a_^l , y ^4 ^l-1 , [ eq : hr_decomp ] @xmath108 and @xmath109 here are scalars representing entry  @xmath110 in the vectors @xmath79 and @xmath81 respectively , and the symbol  @xmath10 with a superscript stands for a repeated tensor product , _ e.g. _  @xmath111 . to verify that under pooling windows of size four  @xmath102 is indeed given by eq . [ eq : hr_decomp ] , simply plug the rows of the decomposition into eq . [ eq : h_y ] , starting from bottom and continuing upwards . for context , eq .  [ eq : hr_decomp ] describes what is known as a hierarchical tensor decomposition ( see chapter  11 in  @xcite ) , with underlying tree over modes being a full quad - tree ( corresponding to the fact that the network s pooling windows cover four entries each ) . [ [ shallow - network . ] ] shallow network . + + + + + + + + + + + + + + + +    the second network we pay special attention to is shallow , comprising a single hidden layer with global pooling    see illustration in fig . [ fig : nets_patterns](b ) . the linear weights of such a network are @xmath104}$ ] for hidden conv operator and @xmath112}$ ] for dense output operator . they determine the coefficient tensor  @xmath102  ( eq .  [ eq : h_y ] ) as follows : a^y=_=1^r_0 a^1,y _ ^na^0 , [ eq : cp_decomp ] where @xmath113 stands for entry  @xmath80 of @xmath114 , and again , the symbol @xmath10 with a superscript represents a repeated tensor product . the tensor decomposition in eq . [ eq : cp_decomp ] is an instance of the classic cp decomposition , also known as rank-1 decomposition ( see  @xcite for a historic survey ) .    to conclude this section , we relate the background material above , as well as our contribution described in the upcoming sections , to the work of  . the latter shows that with arbitrary coefficient tensors  @xmath102 , functions  @xmath101 as in eq . [ eq : h_y ] form a universal hypotheses space . it is then shown that convolutional arithmetic circuits as in fig . [ fig : nets_patterns](a ) realize such functions by applying tensor decompositions to  @xmath102 , with the type of decomposition determined by the structure of a network ( number of layers , number of channels in each layer  _ etc . _ ) . the deep network ( fig . [ fig : nets_patterns](a ) with size-@xmath1 pooling windows and @xmath103 hidden layers ) and the shallow network ( fig . [ fig : nets_patterns](b ) ) presented hereinabove are two special cases , whose corresponding tensor decompositions are given in eq . [ eq : hr_decomp ] and  [ eq : cp_decomp ] respectively . the central result in   relates to inductive bias through the notion of depth efficiency    it is shown that in the parameter space of a deep network , all weight settings but a set of ( lebesgue ) measure zero give rise to functions that can only be realized ( or approximated ) by a shallow network if the latter has exponential size . this result does not relate to the characteristics of instances @xmath60 , it only treats the ability of shallow networks to replicate functions realized by deep networks .    in this paper we draw a line connecting the inductive bias to the nature of  @xmath115 , by studying the relation between a network s architecture and its ability to model correlation among patches  @xmath116 . specifically , in sec . [ sec : sep_rank ] we consider partitions  @xmath23 of  @xmath24 $ ] ( @xmath117 $ ] , where  @xmath118 stands for disjoint union ) , and present the notion of separation rank as a measure of the correlation modeled between the patches indexed by  @xmath25 and those indexed by  @xmath26 . in sec . [ sec : analysis : sep2mat ] the separation rank of a network s function  @xmath101 w.r.t . a partition  @xmath23 is proven to be equal to the rank of @xmath119    the matricization of the coefficient tensor  @xmath102 w.r.t . [ sec : analysis : deep ] derives lower and upper bounds on this rank for a deep network , showing that it supports exponential separation ranks with polynomial size for certain partitions , whereas for others it is required to be exponentially large . subsequently , sec . [ sec : analysis : shallow ] establishes an upper bound on @xmath120 for shallow networks , implying that these must be exponentially large in order to model exponential separation rank under any partition , and thus can not efficiently replicate a deep network s correlations . our analysis concludes in sec . [ sec : inductive_bias ] , where we discuss the pooling geometry of a deep network as a means for controlling the inductive bias by determining a correspondence between partitions  @xmath23 and spatial partitions of the input . finally , we demonstrate experimentally in sec . [ sec : exp ] how different pooling geometries lead to superior performance in different tasks . our experiments include not only convolutional arithmetic circuits , but also convolutional rectifier networks , _ i.e. _  convolutional networks with relu activation and max or average pooling . in this section we define the concept of separation rank for functions realized by convolutional arithmetic circuits ( sec . [ sec : cac ] ) , _ i.e. _  real functions that take as input @xmath121 . the separation rank serves as a measure of the correlations such functions induce between different sets of input patches , _ i.e. _  different subsets of the variable set @xmath122 . let @xmath23 be a partition of input indexes , _ @xmath25 and  @xmath26 are disjoint subsets of  @xmath24 $ ] whose union gives  @xmath24 $ ] . we may write @xmath27 where @xmath28 , and similarly @xmath29 where @xmath30 . for a function @xmath123 , the _ separation rank w.r.t . the partition @xmath23 _ is defined as follows : or @xmath38 then by definition @xmath124 ( unless @xmath125 , in which case @xmath126 ) . ] sep(h;i , j):=\{rn\{0 } :  s.t ..  [ eq : sep_rank ] + .h(x_1,  ,x_n)=_=1^rg_(x_i_1,  ,x_i_)g_(x_j_1,  ,x_j _ ) } in words , it is the minimal number of summands that together give @xmath127 , where each summand is _ separable w.r.t . @xmath23 _ , _ i.e. _  is equal to a product of two functions   one that intakes only patches indexed by @xmath25 , and another that intakes only patches indexed by @xmath26 . one may wonder if it is at all possible to express @xmath127 through such summands , _ i.e. _  if the separation rank of @xmath127 is finite . from the theory of tensor products between @xmath0 spaces ( see  @xcite for a comprehensive coverage ) , we know that any @xmath128 , _ i.e. _  any @xmath127 that is measurable and square - integrable , may be approximated arbitrarily well by summations of the form @xmath129 . exact realization however is only guaranteed at the limit @xmath130 , thus in general the separation rank of @xmath127 need not be finite . nonetheless , as we show in sec . [ sec : analysis ] , for the class of functions we are interested in , namely functions realizable by convolutional arithmetic circuits , separation ranks are always finite . the concept of separation rank was introduced in  @xcite for numerical treatment of high - dimensional functions , and has since been employed for various applications , _ e.g. _  quantum chemistry  ( @xcite ) , particle engineering  ( @xcite ) and machine learning  ( @xcite ) . if the separation rank of a function w.r.t . a partition of its input is equal to  @xmath4 , the function is separable , meaning it does not model any interaction between the sets of variables . specifically , if @xmath124 then there exist @xmath131 and @xmath132 such that @xmath133 , and the function  @xmath127 can not take into account consistency between the values of @xmath134 and those of @xmath135 . in a statistical setting , if  @xmath127 is a probability density function , this would mean that @xmath134 and @xmath135 are statistically independent . the higher @xmath136 is , the farther  @xmath127 is from this situation , _ i.e. _  the more it models dependency between @xmath134 and @xmath135 , or equivalently , the stronger the correlation it induces between the patches indexed by  @xmath25 and those indexed by  @xmath26 .    the interpretation of separation rank as a measure of deviation from separability is formalized in app . [ app : sep_rank_l2 ] , where it is shown that  @xmath136 is closely related to the @xmath0  distance of  @xmath127 from the set of separable functions w.r.t . specifically , we define  @xmath137 as the latter distance divided by the @xmath0  norm of  @xmath127 would accordingly rescale  @xmath137 , rendering the latter uninformative in terms of deviation from separability . ] , and show that @xmath136 provides an upper bound on  @xmath137 . while it is not possible to lay out a general lower bound on  @xmath137 in terms of  @xmath136 , we show that the specific lower bounds on  @xmath136 underlying our analyses can be translated into lower bounds on  @xmath137 . this implies that our results , facilitated by upper and lower bounds on separation ranks of convolutional arithmetic circuits , may equivalently be framed in terms of @xmath0  distances from separable functions . in this section we analyze convolutional arithmetic circuits ( sec .  [ sec : cac ] ) in terms of the correlations they can model between sides of different input partitions , _ i.e. _  in terms of the separation ranks ( sec . [ sec : sep_rank ] ) they support under different partitions  @xmath23 of  @xmath24 $ ] . we begin in sec . [ sec : analysis : sep2mat ] , establishing a correspondence between separation ranks and coefficient tensor matricization ranks . this correspondence is then used in sec . [ sec : analysis : deep ] and  [ sec : analysis : shallow ] to analyze the deep and shallow networks ( respectively ) presented in sec . [ sec : cac ] . we note that we focus on these particular networks merely for simplicity of presentation    the analysis can easily be adapted to account for alternative networks with different depths and pooling schemes . let  @xmath101 be a function realized by a convolutional arithmetic circuit , with corresponding coefficient tensor  @xmath102 ( eq .  [ eq : h_y ] ) . denote by  @xmath23 an arbitrary partition of @xmath24 $ ] , _ i.e. _  @xmath117 $ ] . we are interested in studying @xmath138    the separation rank of  @xmath101 w.r.t . @xmath23 ( eq .  [ eq : sep_rank ] ) . as claim  [ claim : sep_mat_ranks_equal ] below states , assuming representation functions @xmath139}$ ] are linearly independent ( if they are not , we drop dependent functions and modify  @xmath102 accordingly is dependent , _ i.e. _  there exist @xmath140 such that @xmath141 . we may then plug this into eq . [ eq : h_y ] , and obtain an expression for  @xmath101 that has @xmath142 as representation functions , and a coefficient tensor with dimension  @xmath143 in each mode . continuing in this fashion , one arrives at an expression for  @xmath101 whose representation functions are linearly independent . ] ) , this separation rank is equal to the rank of  @xmath119    the matricization of the coefficient tensor  @xmath102 w.r.t . the partition @xmath23 . our problem thus translates to studying ranks of matricized coefficient tensors . [ claim : sep_mat_ranks_equal ] let @xmath101 be a function realized by a convolutional arithmetic circuit ( fig . [ fig : nets_patterns](a ) ) , with corresponding coefficient tensor @xmath102 ( eq .  [ eq : h_y ] ) . assume that the network s representation functions  @xmath75 are linearly independent , and that they , as well as the functions  @xmath144 in the definition of separation rank ( eq .  [ eq : sep_rank ] ) , are measurable and square - integrable . may seem as a limitation at first glance , as for example neurons @xmath64 , with parameters @xmath65 and sigmoid or relu activation @xmath66 , do not meet this condition . however , since in practice our inputs are bounded ( _ e.g. _  they represent image pixels by holding intensity values ) , we may view functions as having compact support , which , as long as they are continuous ( holds in all cases of interest ) , ensures square - integrability . ] then , for any partition @xmath23 of @xmath24 $ ] , it holds that @xmath145 . see app . [ app : proofs : sep_mat_ranks_equal ] .    as the linear weights of a network vary , so do the coefficient tensors ( @xmath102 ) it gives rise to . accordingly , for a particular partition @xmath23 , a network does not correspond to a single value of @xmath120 , but rather supports a range of values . we analyze this range by quantifying its maximum , which reflects the strongest correlation that the network can model between the input patches indexed by  @xmath25 and those indexed by  @xmath26 . one may wonder if the maximal value of @xmath120 is the appropriate statistic to measure , as a - priori , it may be that @xmath120 is maximal for very few of the network s weight settings , and much lower for all the rest . apparently , as claim  [ claim : mat_rank_max_everywhere ] below states , this is not the case , and in fact @xmath120 is maximal under almost all of the network s weight settings . [ claim : mat_rank_max_everywhere ] consider a convolutional arithmetic circuit ( fig . [ fig : nets_patterns](a ) ) with corresponding coefficient tensor @xmath102 ( eq .  [ eq : h_y ] ) . @xmath102 depends on the network s linear weights   @xmath99 and @xmath81 , thus for a given partition @xmath23 of @xmath24 $ ] , @xmath120 is a function of these weights . this function obtains its maximum almost everywhere ( w.r.t . lebesgue measure ) . see app . [ app : proofs : mat_rank_max_everywhere ] .      in this subsection we study correlations modeled by the deep network presented in sec .  [ sec : cac ] ( fig . [ fig : nets_patterns](a ) with size-@xmath1 pooling windows and @xmath103 hidden layers ) . in accordance with sec . [ sec : analysis : sep2mat ] , we do so by characterizing the maximal ranks of coefficient tensor matricizations under different partitions .    recall from eq . [ eq : hr_decomp ] the hierarchical decomposition expressing a coefficient tensor  @xmath102 realized by the deep network . we are interested in matricizations of this tensor under different partitions of @xmath24 $ ] . let @xmath23 be an arbitrary partition , _ i.e. _  @xmath117 $ ] . matricizing the last level of eq . [ eq : hr_decomp ] w.r.t . @xmath23 , while applying the relation in eq . [ eq : mat_tensor_prod ] , gives : ^y_i , j & = & _ = 1^r_l-1a_^l , y^l-1,^l-1,^l-1,^l-1,_i , j + & = & _ = 1^r_l-1a_^l , y^l-1,^l-1,_i , j + & & ^l-1,^l-1,_(i-24^l-1),(j-24^l-1 ) + applying eq .  [ eq : mat_tensor_prod ] again , this time to matricizations of the tensor @xmath146 , we obtain : ^y_i , j & = & _ = 1^r_l-1a_^l , y^l-1,_i , j + & & ^l-1,_(i-4^l-1),(j-4^l-1 ) + & & ^l-1,_(i-24^l-1),(j-24^l-1 ) + & & ^l-1,_(i-34^l-1),(j-34^l-1 ) for every @xmath147 $ ] define @xmath148 $ ] and @xmath149 $ ] . in words , @xmath150 represents the partition induced by  @xmath23 on the  @xmath151th quadrant of  @xmath24 $ ] , _ i.e. _  on the @xmath151th size-@xmath152 group of input patches . we now have the following matricized version of the last level in eq . [ eq : hr_decomp ] : @xmath153 where the symbol  @xmath41 with a running index stands for an iterative kronecker product . to derive analogous matricized versions for the upper levels of eq . [ eq : hr_decomp ] , we define for @xmath154 $ ] : i_l , k:=(i-(k-1)4^l ) j_l , k:=(j-(k-1)4^l)[eq:_i_lk_j_lk ] that is to say , @xmath155 represents the partition induced by  @xmath23 on the set of indexes @xmath156 , _ i.e. _  on the @xmath151th size-@xmath157 group of input patches . with this notation in hand , traversing upwards through the levels of eq . [ eq : hr_decomp ] , with repeated application of the relation in eq . [ eq : mat_tensor_prod ] , one arrives at the following matrix decomposition for  @xmath119 : _ & = & _ = 1^r_0 a_^1 , ^0,_i_0,4(k-1)+t , j_0,4(k-1)+t  , + & & + _ & = & _ = 1^r_l-1 a_^l , ^l-1,_i_l-1,4(k-1)+t , j_l-1,4(k-1)+t  , l\{2  l-1 } , + & & + _ & = & _ = 1^r_l-1 a_^l , y ^l-1,_i_l-1,t , j_l-1,t [ eq : mat_hr_decomp ]    eq .  [ eq : mat_hr_decomp ] expresses @xmath119    the matricization w.r.t . the partition @xmath23 of a coefficient tensor @xmath102 realized by the deep network , in terms of the network s conv weights  @xmath99 and output weights  @xmath81 . as discussed above , our interest lies in the maximal rank that this matricization can take . theorem  [ theorem : deep_mat_rank_bounds ] below provides lower and upper bounds on this maximal rank , by making use of eq . [ eq : mat_hr_decomp ] , and of the rank - multiplicative property of the kronecker product ( @xmath158 ) . [ theorem : deep_mat_rank_bounds ] let @xmath23 be a partition of @xmath24 $ ] , and @xmath119 be the matricization w.r.t . @xmath23 of a coefficient tensor @xmath102 ( eq .  [ eq : h_y ] ) realized by the deep network ( fig . [ fig : nets_patterns](a ) with size-@xmath1 pooling windows ) . for every @xmath159 and @xmath160 $ ] , define  @xmath161 and  @xmath162 as in eq . [ eq:_i_lk_j_lk ] . then , the maximal rank that @xmath119 can take ( when network weights vary ) is :    * no smaller than @xmath163 , where @xmath164 : i_{1,k}\neq\emptyset \wedge j_{1,k}\neq\emptyset\ } \right\rvert}$ ] . * no greater than @xmath165 , where @xmath166 for @xmath167 $ ] , and @xmath168 for @xmath169,k\in[n/4^l]$ ] . see app . [ app : proofs : deep_mat_rank_bounds ] . the lower bound in theorem  [ theorem : deep_mat_rank_bounds ] is exponential in @xmath170 , the latter defined to be the number of size-@xmath1 patch groups that are split by the partition @xmath23 , _ i.e. _  whose indexes are divided between  @xmath25 and  @xmath26 . partitions that split many of the size-@xmath1 patch groups will thus lead to a large lower bound . for example , consider the partition @xmath171 defined as follows : i^odd=\{1,3,  ,n-1 } j^even=\{2,4,  ,n } [ eq : part_odd_even ] this partition splits all size-@xmath1 patch groups ( @xmath172 ) , leading to a lower bound that is exponential in the number of patches ( @xmath6 ) .    the upper bound in theorem  [ theorem : deep_mat_rank_bounds ] is expressed via constants  @xmath173 , defined recursively over levels @xmath68 , with  @xmath151 ranging over @xmath174 for each level  @xmath69 . what prevents @xmath173 from growing double - exponentially fast ( w.r.t . @xmath69 ) is the minimization with @xmath175 . specifically , if @xmath176 is small , _ i.e. _  if the partition induced by @xmath23 on the @xmath151th size-@xmath157 group of patches is unbalanced ( most of the patches belong to one side of the partition , and only a few belong to the other ) , @xmath173 will be of reasonable size . the higher this takes place in the hierarchy ( _ i.e. _  the larger  @xmath69 is ) , the lower our eventual upper bound will be . in other words , if partitions induced by @xmath23 on size-@xmath157 patch groups are unbalanced for large values of @xmath69 , the upper bound in theorem  [ theorem : deep_mat_rank_bounds ] will be small . for example , consider the partition @xmath177 defined by : i^low=\{1,  ,n/2 } j^high=\{n/2 + 1,  ,n } [ eq : part_low_high ] under @xmath177 , all partitions induced on size-@xmath152 patch groups ( quadrants of  @xmath24 $ ] ) are completely one - sided ( @xmath178 for all @xmath147 $ ] ) , resulting in the upper bound being no greater than @xmath73    linear in network size .    to summarize this discussion , theorem  [ theorem : deep_mat_rank_bounds ] states that with the deep network , the maximal rank of a coefficient tensor matricization w.r.t . @xmath23 , highly depends on the nature of the partition  @xmath23    it will be exponentially high for partitions such as @xmath171 , that split many size-@xmath1 patch groups , while being only polynomial ( or linear ) for partitions like @xmath177 , under which size-@xmath157 patch groups are unevenly divided for large values of  @xmath69 . since the rank of a coefficient tensor matricization w.r.t . @xmath23 corresponds to the strength of correlation modeled between input patches indexed by  @xmath25 and those indexed by  @xmath26 ( sec . [ sec : analysis : sep2mat ] ) , we conclude that the ability of a polynomially sized deep network to model correlation between sets of input patches highly depends on the nature of these sets . we now turn to study correlations modeled by the shallow network presented in sec . [ sec : cac ] ( fig . [ fig : nets_patterns](b ) ) . in line with sec . [ sec : analysis : sep2mat ] , this is achieved by characterizing the maximal ranks of coefficient tensor matricizations under different partitions .    recall from eq . [ eq : cp_decomp ] the cp decomposition expressing a coefficient tensor  @xmath102 realized by the shallow network . for an arbitrary partition @xmath23 of @xmath24 $ ] , _ i.e. _  @xmath117 $ ] , matricizing this decomposition with repeated application of the relation in eq . [ eq : mat_tensor_prod ] , gives the following expression for @xmath119    the matricization w.r.t . @xmath23 of a coefficient tensor realized by the shallow network : ^y_i , j=_=1^r_0 a^1,y _ ( ^a^0 , ) ( ^a^0,)^[eq : mat_cp_decomp ] @xmath179 and @xmath180 here are column vectors of dimensions  @xmath181 and  @xmath182 respectively , standing for the kronecker products of @xmath183 with itself  @xmath184 and  @xmath185 times ( respectively ) . [ eq : mat_cp_decomp ] immediately leads to two observations regarding the ranks that may be taken by @xmath119 . first , they depend on the partition @xmath23 only through its division size , _ i.e. _  through @xmath184 and @xmath185 . second , they are no greater than @xmath186 , meaning that the maximal rank is linear ( or less ) in network size . in light of sec . [ sec : analysis : sep2mat ] and  [ sec : analysis : deep ] , these findings imply that in contrast to the deep network , which with polynomial size supports exponential separation ranks under favored partitions , the shallow network treats all partitions ( of a given division size ) equally , and can only give rise to an exponential separation rank if its size is exponential . suppose now that we would like to use the shallow network to replicate a function realized by a polynomially sized deep network . so long as the deep network s function admits an exponential separation rank under at least one of the favored partitions ( _ e.g. _  @xmath171    eq .  [ eq : part_odd_even ] ) , the shallow network would have to be exponentially large in order to replicate it , _ i.e. _  depth efficiency takes place .  ) are not universal . in particular , it may very well be that a function realized by a polynomially sized deep network can not be replicated by the shallow network , no matter how large ( wide ) we allow it to be . in such scenarios depth efficiency does not provide insight into the complexity of functions brought forth by depth . to obtain a shallow network that is universal , thus an appropriate gauge for depth efficiency , we may remove the constraint of weight sharing , _ i.e. _  allow the filters in the hidden conv operator to hold different weights at different spatial locations ( see   for proof that this indeed leads to universality ) . all results we have established for the original shallow network remain valid when weight sharing is removed . in particular , the separation ranks of the network are still linear in its size . this implies that as suggested , depth efficiency indeed holds . ] since all but a negligible set of the functions realizable by the deep network give rise to maximal separation ranks ( sec  [ sec : analysis : sep2mat ] ) , we obtain the complete depth efficiency result of  . however , unlike  , which did not provide any explanation for the usefulness of functions brought forth by depth , we obtain an insight into their utility   they are able to efficiently model strong correlation under favored partitions of the input . the deep network presented in sec . [ sec : cac ] , whose correlations we analyzed in sec . [ sec : analysis : deep ] , was defined as having size-@xmath1 pooling windows , _ i.e. _  pooling windows covering four entries each . we have yet to specify the shapes of these windows , or equivalently , the spatial ( two - dimensional ) locations of nodes grouped together in the process of pooling . in compliance with standard convolutional network design , we now assume that the network s ( size-@xmath1 ) pooling windows are contiguous square blocks , _ i.e. _  have shape  @xmath187 . under this configuration , the network s functional description ( eq .  [ eq : h_y ] with  @xmath102 given by eq . [ eq : hr_decomp ] ) induces a spatial ordering of input patches ) , the nodes corresponding to patches @xmath188 are pooled into one group , those corresponding to @xmath189 are pooled into another , and so forth . similar assumptions hold for the deeper layers . for example , in the second pooling operation ( hidden layer  @xmath4 ) , the node with receptive field @xmath190 , _ i.e. _  the one corresponding to the quadruple of patches @xmath191 , is assumed to be pooled together with the nodes whose receptive fields are @xmath192 , @xmath193 and @xmath194 . ] , which may be described by the following recursive process :    * set the index of the top - left patch to  @xmath4 . * for @xmath195 : replicate the already - assigned top - left @xmath196-by-@xmath196 block of indexes , and place copies on its right , bottom - right and bottom . then , add a  @xmath197 offset to all indexes in the right copy , a  @xmath198 offset to all indexes in the bottom - right copy , and a  @xmath199 offset to all indexes in the bottom copy .    with this spatial ordering ( illustrated in fig . [ fig : nets_patterns](c ) ) , partitions  @xmath23 of  @xmath24 $ ] convey a spatial pattern . for example , the partition  @xmath171 ( eq .  [ eq : part_odd_even ] ) corresponds to the pattern illustrated on the left of fig . [ fig : nets_patterns](c ) , whereas  @xmath177 ( eq .  [ eq : part_low_high ] ) corresponds to the pattern illustrated on the right . our analysis ( sec . [ sec : analysis : deep ] ) shows that the deep network is able to model strong correlation under  @xmath171 , while being inefficient for modeling correlation under  @xmath177 . more generally , partitions for which  @xmath170 , defined in theorem  [ theorem : deep_mat_rank_bounds ] , is high , convey patterns that split many @xmath187 patch blocks , _ i.e. _  are highly entangled . these partitions enjoy the possibility of strong correlation . on the other hand , partitions for which  @xmath176 is small for large values of  @xmath69 ( see eq . [ eq:_i_lk_j_lk ] for definition of @xmath161 and @xmath162 ) convey patterns that divide large @xmath200 patch blocks unevenly , _ i.e. _  separate the input to distinct contiguous regions . these partitions , as we have seen , suffer from limited low correlations . we conclude that with @xmath187 pooling , the deep network is able to model strong correlation between input regions that are highly entangled , at the expense of being inefficient for modeling correlation between input regions that are far apart . had we selected a different pooling regime , the preference of input partition patterns in terms of modeled correlation would change . for example , if pooling windows were set to group nodes with their spatial reflections ( horizontal , vertical and horizontal - vertical ) , coarse patterns that divide the input symmetrically , such as the one illustrated on the right of fig . [ fig : nets_patterns](c ) , would enjoy the possibility of strong correlation , whereas many entangled patterns would now suffer from limited low correlation . the choice of pooling shapes thus serves as a means for controlling the inductive bias in terms of correlations modeled between input regions . square contiguous windows , as commonly employed in practice , lead to a preference that complies with our intuition regarding the statistics of natural images ( nearby pixels more correlated than distant ones ) . other pooling schemes lead to different preferences , and this allows tailoring a network to data that departs from the usual domain of natural imagery . we demonstrate this experimentally in the next section , where it is shown how different pooling geometries lead to superior performance in different tasks . the main conclusion from our analyses ( sec .  [ sec : analysis ] and  [ sec : inductive_bias ] ) is that the pooling geometry of a deep convolutional network controls its inductive bias by determining which correlations between input regions can be modeled efficiently . we have also seen that shallow networks can not model correlations efficiently , regardless of the considered input regions . in this section we validate these assertions empirically , not only with convolutional arithmetic circuits ( subject of our analyses ) , but also with convolutional rectifier networks    convolutional networks with relu activation and max or average pooling . for conciseness , we defer to app . [ app : impl ] some details regarding our implementation . the latter is fully available online at https://github.com/huji-deep/inductive-pooling . our experiments are based on a synthetic classification benchmark inspired by medical imaging tasks . instances to be classified are @xmath56-by-@xmath56 binary images , each displaying a random distorted oval shape ( _ blob _ ) with missing pixels in its interior ( _ holes _ ) . for each image , two continuous scores in range  @xmath201 $ ] are computed . the first , referred to as _ closedness _ , reflects how morphologically closed a blob is , and is defined to be the ratio between the number of pixels in the blob , and the number of pixels in its closure ( see app . [ app : morph_closure ] for exact definition of the latter ) . the second score , named _ symmetry _ , reflects the degree to which a blob is left - right symmetric about its center . it is measured by cropping the bounding box around a blob , applying a left - right flip to the latter , and computing the ratio between the number of pixels in the intersection of the blob and its reflection , and the number of pixels in the blob . to generate labeled sets for classification ( train and test ) , we render multiple images , sort them according to their closedness and symmetry , and for each of the two scores , assign the label `` high '' to the top 40% and the label `` low '' to the bottom 40% ( the mid 20% are considered ill - defined ) . this creates two binary ( two - class ) classification tasks   one for closedness and one for symmetry ( see fig .  [ fig : exp_data ] for a sample of images participating in both tasks ) . given that closedness is a property of a local nature , we expect its classification task to require a predictor to be able to model strong correlations between neighboring pixels . symmetry on the other hand is a property that relates pixels to their reflections , thus we expect its classification task to demand that a predictor be able to model correlations across distances . we evaluated the deep convolutional arithmetic circuit considered throughout the paper ( fig . [ fig : nets_patterns](a ) with size-@xmath1 pooling windows ) under two different pooling geometries . the first , referred to as _ , comprises standard @xmath187 pooling windows . the second , dubbed _ mirror _ , pools together nodes with their horizontal , vertical and horizontal - vertical reflections . in both cases , input patches ( @xmath116 ) were set as individual pixels , resulting in @xmath58 patches and @xmath202 hidden layers . @xmath203 representation functions ( @xmath75 ) were fixed , the first realizing the identity on binary inputs ( @xmath204 for @xmath205 ) , and the second realizing negation ( @xmath206 for @xmath205 ) . classification was realized through @xmath207 network outputs , with prediction following the stronger activation . the number of channels across all hidden layers was uniform , and varied between  @xmath208 and  @xmath209 . [ fig : exp_results_cac_pool4 ] shows the results of applying the deep network with both square and mirror pooling , to both closedness and symmetry tasks , where each of the latter has @xmath210 images for training and @xmath211 images for testing . as can be seen in the figure , square pooling significantly outperforms mirror pooling in closedness classification , whereas the opposite occurs in symmetry classification . this complies with our discussion in sec . [ sec : inductive_bias ] , according to which square pooling supports modeling correlations between entangled ( neighboring ) regions of the input , whereas mirror pooling puts focus on correlations between input regions that are symmetric w.r.t . one another . we thus obtain a demonstration of how prior knowledge regarding a task at hand may be used to tailor the inductive bias of a deep convolutional network by designing an appropriate pooling geometry . in addition to the deep network , we also evaluated the shallow convolutional arithmetic circuit analyzed in the paper ( fig . [ fig : nets_patterns](b ) ) . the architectural choices for this network were the same as those described above for the deep network besides the number of hidden channels , which in this case applied to the network s single hidden layer , and varied between  @xmath212 and  @xmath213 . the highest train and test accuracies delivered by this network ( with @xmath213 hidden channels ) were roughly  @xmath214 on closedness task , and  @xmath215 on symmetry task . the fact that these accuracies are inferior to those of the deep network , even when the latter s pooling geometry is not optimal for the task at hand , complies with our analysis in sec . [ sec : analysis ] . namely , it complies with the observation that separation ranks ( correlations ) are sometimes exponential and sometimes polynomial with the deep network , whereas with the shallow one they are never more than linear in network size . finally , to assess the validity of our findings for convolutional networks in general , not just convolutional arithmetic circuits , we repeated the above experiments with convolutional rectifier networks . namely , we placed relu activations after every conv operator , switched the pooling operation from product to average , and re - evaluated the deep ( square and mirror pooling geometries ) and shallow networks . we then reiterated this process once more , with pooling operation set to max instead of average . the results obtained by the deep networks are presented in fig . [ fig : exp_results_crn_pool4 ] . the shallow network with average pooling reached train / test accuracies of roughly  @xmath216 on closedness task , and  @xmath217 on symmetry task . with max pooling , performance of the shallow network did not exceed chance . altogether , convolutional rectifier networks exhibit the same phenomena observed with convolutional arithmetic circuits , indicating that the conclusions from our analyses likely apply to such networks as well . formal adaptation of the analyses to convolutional rectifier networks , similarly to the adaptation of   carried out in  , is left for future work . through the notion of separation rank , we studied the relation between the architecture of a convolutional network , and its ability to model correlations among input regions . for a given input partition , the separation rank quantifies how far a function is from separability , which in a probabilistic setting , corresponds to statistical independence between sides of the partition . our analysis shows that a polynomially sized deep convolutional arithmetic circuit supports exponentially high separation ranks for certain input partitions , while being limited to polynomial or linear ( in network size ) separation ranks for others . the network s pooling window shapes effectively determine which input partitions are favored in terms of separation rank , _ i.e. _  which partitions enjoy the possibility of exponentially high separation ranks with polynomial network size , and which require network to be exponentially large . pooling geometry thus serves as a means for controlling the inductive bias . the particular pooling scheme commonly employed in practice   square contiguous windows , favors interleaved partitions over ones that divide the input to distinct areas , thus orients the inductive bias towards the statistics of natural images ( nearby pixels more correlated than distant ones ) . other pooling schemes lead to different preferences , and this allows tailoring the network to data that departs from the usual domain of natural imagery . as opposed to deep convolutional arithmetic circuits , shallow ones support only linear ( in network size ) separation ranks . therefore , in order to replicate a function realized by a deep network ( exponential separation rank ) , a shallow network must be exponentially large . by this we derive the depth efficiency result of  , but in addition , provide an insight into the benefit of functions brought forth by depth   they are able to efficiently model strong correlation under favored partitions of the input . we validated our conclusions empirically , with convolutional arithmetic circuits as well as convolutional rectifier networks    convolutional networks with relu activation and max or average pooling . our experiments demonstrate how different pooling geometries lead to superior performance in different tasks . specifically , we evaluate deep networks in the measurement of shape continuity , a task of a local nature , and show that standard square pooling windows outperform ones that join together nodes with their spatial reflections . in contrast , when measuring shape symmetry , modeling correlations across distances is of vital importance , and the latter pooling geometry is superior to the conventional one . shallow networks are inefficient at modeling correlations of any kind , and indeed lead to poor performance on both tasks .    finally , our analyses and results bring forth the possibility of expanding the coverage of correlations efficiently modeled by a deep convolutional network . specifically , by blending together multiple pooling geometries in the hidden layers of a network , it is possible to facilitate simultaneous support for a wide variety of correlations suiting data of different types . investigation of this direction , from both theoretical and empirical perspectives , is viewed as a promising avenue for future research . this work is supported by intel grant icri - ci # 9 - 2012 - 6133 , by isf center grant 1790/12 and by the european research council ( theorydl project ) . nadav cohen is supported by a google doctoral fellowship in machine learning . we prove the equality in two steps , first showing that @xmath218 , and then establishing the converse . the first step is elementary , and does not make use of the representation functions ( @xmath75 ) linear independence , or of measurability / square - integrability . the second step does rely on these assumptions , and employs slightly more advanced mathematical machinery . throughout the proof , we assume without loss of generality that the partition @xmath23 of  @xmath24 $ ] is such that  @xmath25 takes on lower values , while  @xmath26 takes on higher ones . that is to say , we assume that @xmath219 and @xmath220 . and @xmath29 , and define an auxiliary function @xmath221 by permuting the entries of @xmath101 such that those indexed by @xmath25 are on the left and those indexed by @xmath26 on the right , _ i.e. _  @xmath222 . obviously @xmath223 , where the partition @xmath224 is defined by @xmath225 and @xmath226 . analogously to the definition of  @xmath221 , let  @xmath227 be the tensor obtained by permuting the modes of  @xmath102 such that those indexed by @xmath25 are on the left and those indexed by @xmath26 on the right , _ it is not difficult to see that matricizing  @xmath229 w.r.t . @xmath224 is equivalent to matricizing  @xmath230 w.r.t . @xmath23 , _ i.e. _  @xmath231 , and in particular @xmath232 . moreover , since by definition @xmath230 is a coefficient tensor corresponding to @xmath101 ( eq .  [ eq : h_y ] ) , @xmath227 will be a coefficient tensor that corresponds to @xmath221 . now , our proof will show that @xmath233 , which , in light of the equalities above , implies @xmath145 , as required . ]    to prove that @xmath218 , denote by @xmath234 the rank of @xmath119 . the latter is an @xmath181-by-@xmath182 matrix , thus there exist vectors @xmath235 and @xmath236 such that @xmath237 . for every @xmath238 $ ] , let @xmath239 be the tensor of order @xmath184 and dimension @xmath62 in each mode whose arrangement as a column vector gives @xmath240 , _ i.e. _  whose matricization w.r.t . the partition @xmath241,\emptyset)$ ] is equal to @xmath240 . similarly , let  @xmath242 , @xmath238 $ ] , be the tensor of order @xmath243 and dimension @xmath62 in each mode whose matricization w.r.t . the partition @xmath244)$ ] ( arrangement as a row vector ) is equal to @xmath245 . it holds that : ^y_i , j & = & _ = 1^ru_v_^ + & = & _ = 1^r^_[],^ _ , [ ] + & = & _ = 1^r^_i , j^_(i-),(j- ) + & = & _ = 1^r^^_i , j + & = & _ = 1^rb^^_i , j where the third equality relies on the assumption @xmath246 , the fourth equality makes use of the relation in eq . [ eq : mat_tensor_prod ] , and the last equality is based on the linearity of the matricization operator . since matricizations are merely rearrangements of tensors , the fact that @xmath247 implies @xmath248 , or equivalently , @xmath249 for every @xmath250 $ ] . plugging this into eq . [ eq : h_y ] gives : h_y(x_1,  ,x_n ) & = & _ d_1  d_n=1^ma_d_1  d_n^y_i=1^n f__d_i(x_i ) + & = & _ d_1  d_n=1^m_=1^rb^_d_1  d_c^_d_+1  d_n_i=1^n f__d_i(x_i ) + & = & _ = 1^r(_d_1  d_=1^mb^_d_1  d__i=1^ f__d_i(x_i ) ) + & & ( _ d_+1  d_n=1^mc^_d_+1  d_n_i=+1^n f__d_i(x_i ) ) [ eq : mat_rank_over_sep_rank ] for every @xmath238 $ ] , define the functions @xmath251 and @xmath252 as follows : g_(x_1,  ,x _ ) & : = & _ d_1  d_=1^mb^_d_1  d__i=1^ f__d_i(x_i ) + g_(x_1,  ,x _ ) & : = & _ d_1  d_=1^mc^_d_1  d__i=1^ f__d_i(x_i ) substituting these into eq . [ eq : mat_rank_over_sep_rank ] leads to : @xmath253 which by definition of the separation rank ( eq .  [ eq : sep_rank ] ) , implies @xmath254 . by this we have shown that @xmath218 , as required .    for proving the converse inequality , _ i.e. _  @xmath255 , we rely on basic concepts and results from functional analysis , or more specifically , from the topic of @xmath0 spaces . while a full introduction to this topic is beyond our scope ( the interested reader is referred to  @xcite ) , we briefly lay out here the minimal background required in order to follow our proof . for any @xmath256 , @xmath257 is formally defined as the hilbert space of lebesgue measurable square - integrable real functions over @xmath258 on which they differ has measure zero . ] , equipped with standard ( point - wise ) addition and scalar multiplication , as well as the inner product defined by integration over point - wise multiplication . for our purposes , @xmath257 may simply be thought of as the ( infinite - dimensional ) vector space of functions @xmath259 satisfying @xmath260 , with inner product defined by @xmath261 . our proof will make use of the following basic facts related to @xmath0 spaces :    [ fact : orth_decomp ] if @xmath262 is a finite - dimensional subspace of @xmath257 , then any @xmath263 may be expressed as @xmath264 , with @xmath265 and @xmath266 ( _ i.e. _  @xmath267 is orthogonal to all elements in @xmath262 ) . moreover , such a representation is unique , so in the case where @xmath268 , we necessarily have @xmath269 and @xmath270 . [ fact : prod ] if @xmath271 , then the function @xmath272 belongs to @xmath273 . [ fact : prod_orth ] let @xmath262 and @xmath274 be finite - dimensional subspaces of @xmath275 and @xmath276 respectively , and define @xmath277 to be the subspace spanned by @xmath278 . given @xmath271 , consider the function @xmath272 in @xmath273 . this function belongs to  @xmath279 if @xmath280 or @xmath281 . [ fact : prod_ind ] if @xmath282 are linearly independent , then for any @xmath283 , the set of functions @xmath284}$ ] is linearly independent in @xmath285 .    to facilitate application of the theory of @xmath0 spaces , we now make use of the assumption that the network s representation functions  @xmath75 , as well as the functions  @xmath144 in the definition of separation rank ( eq .  [ eq : sep_rank ] ) , are measurable and square - integrable . taking into account the expression given in eq . [ eq : h_y ] for  @xmath101 , as well as fact  [ fact : prod ] above , one readily sees that @xmath286 implies  @xmath287 . the separation rank @xmath138 will be the minimal non - negative integer  @xmath234 such that there exist @xmath288 and @xmath289 for which : h_y(x_1,  ,x_n)=_=1^rg_(x_1,  ,x_)g_(x_+1,  ,x_n ) [ eq : h_y_sep ]    we would like to show that @xmath255 . our strategy for achieving this will be to start from eq . [ eq : h_y_sep ] , and derive an expression for  @xmath119 comprising a sum of  @xmath234 rank-@xmath4 matrices . as an initial step along this path , define the following finite - dimensional subspaces : v&:=&span\{(x_1,  ,x_)_i=1^f__d_i(x_i)}_d_1  d _  l^2((r^s)^ ) [ eq : v ] + v&:=&span\{(x_1,  ,x_)_i=1^f__d_i(x_i)}_d_1  d _  l^2((r^s)^ ) [ eq : vpr ] + u&:=&span\{(x_1,  ,x_n)_i=1^nf__d_i(x_i)}_d_1  d_n  l^2((r^s)^n ) [ eq : u ] notice that @xmath290 ( eq .  [ eq : h_y ] ) , and that  @xmath291 is the span of products from  @xmath262 and  @xmath274 ,  _ i.e. _ : u = span\{(x_1,  ,x_n)p(x_1,  ,x_)p(x_+1,  ,x_n):pv , pv } [ eq : u_prod_v_vpr ] returning to eq . [ eq : h_y_sep ] , we apply fact  [ fact : orth_decomp ] to obtain orthogonal decompositions of @xmath292 w.r.t . @xmath262 , and of @xmath293 w.r.t . this gives @xmath294 , @xmath295 , @xmath296 and @xmath297 , such that @xmath298 and @xmath299 for every @xmath238 $ ] . plug this into eq . [ eq : h_y_sep ] : h_y(x_1,  ,x_n ) & = & _ = 1^rg_(x_1,  ,x_)g_(x_+1,  ,x_n ) + & = & _ = 1^r(p_(x_1,  ,x_)+_(x_1,  ,x _ ) ) + & & ( p_(x_+1,  ,x_n)+_(x_+1,  ,x_n ) ) + & = & _ = 1^rp_(x_1,  ,x_)p_(x_+1,  ,x_n ) + & & + _ = 1^rp_(x_1,  ,x_)_(x_+1,  ,x_n ) + & & + _ = 1^r_(x_1,  ,x_)p_(x_+1,  ,x_n ) + & & + _ = 1^r_(x_1,  ,x_)_(x_+1,  ,x_n ) given that  @xmath291 is the span of products from  @xmath262 and  @xmath274 ( eq .  [ eq : u_prod_v_vpr ] ) , and that @xmath300 , one readily sees that the first term in the latter expression belongs to  @xmath291 , while , according to fact  [ fact : prod_orth ] , the second , third and fourth terms are orthogonal to  @xmath291 . we thus obtained an orthogonal decomposition of  @xmath101 w.r.t .  @xmath291 . since  @xmath101 is contained in  @xmath291 , the orthogonal component must vanish ( fact  [ fact : orth_decomp ] ) , and we amount at : h_y(x_1,  ,x_n)=_=1^rp_(x_1,  ,x_)p_(x_+1,  ,x_n ) [ eq : h_y_sep_subspace ] for every @xmath238 $ ] , let  @xmath239 and  @xmath242 be coefficient tensors of  @xmath301 and  @xmath302 w.r.t . the functions that span  @xmath262 and  @xmath274 ( eq .  [ eq : v ] and  [ eq : vpr ] ) , respectively . put formally , @xmath239  and  @xmath242 are tensors of orders  @xmath184 and  @xmath185 ( respectively ) , with dimension  @xmath62 in each mode , meeting : p_(x_1,  ,x _ ) & = & _ d_1  d_=1^mb^_d_1  d__i=1^ f__d_i(x_i ) + p_(x_1,  ,x _ ) & = & _ d_1  d_=1^mc^_d_1  d__i=1^ f__d_i(x_i ) substitute into eq . [ eq : h_y_sep_subspace ] : h_y(x_1,  ,x_n ) & = & _ = 1^r(_d_1  d_=1^mb^_d_1  d__i=1^ f__d_i(x_i ) ) + & & ( _ d_+1  d_n=1^mc^_d_+1  d_n_i=+1^n f__d_i(x_i ) ) + & = & _ = 1^r_d_1  d_n=1^mb^_d_1  d_n_i=1^n f__d_i(x_i ) + & = & _ d_1  d_n=1^m(_=1^rb^_d_1  d_c^_d_+1  d_n)_i=1^n f__d_i(x_i ) compare this expression for  @xmath101 to that given in eq . [ eq : h_y ] : _ d_1  d_n=1^m(_=1^rb^_d_1  d_c^_d_+1  d_n)_i=1^n f__d_i(x_i ) = _ d_1  d_n=1^ma_d_1  d_n^y_i=1^n f__d_i(x_i ) [ eq : h_y_compare ] at this point we utilize the given linear independence of @xmath286 , from which it follows ( fact  [ fact : prod_ind ] ) that the functions spanning  @xmath291 ( eq .  [ eq : u ] ) are linearly independent in  @xmath303 . both sides of eq . [ eq : h_y_compare ] are linear combinations of these functions , thus their coefficients must coincide : @xmath304 }   ~~\longleftrightarrow~~ { { \mathcal a}}^y=\sum\nolimits_{\nu=1}^r{{\mathcal b}}^{\nu}\otimes{{\mathcal c}}^{\nu}\ ] ] matricizing the tensor equation on the right w.r.t . @xmath23 gives : ^y_i , j & = & _ = 1^rb^^_i , j + & = & _ = 1^r^^_i , j + & = & _ = 1^r^_i , j^_(i-),(j- ) + & = & _ = 1^r^_[],^ _ , [ ] where the second equality is based on the linearity of the matricization operator , the third equality relies on the relation in eq . [ eq : mat_tensor_prod ] , and the last equality makes use of the assumption @xmath246 . for every @xmath238 $ ] , @xmath305,\emptyset}$ ]  is a column vector of dimension  @xmath181 and  @xmath306}$ ] is a row vector of dimension  @xmath182 . denoting these by  @xmath240 and  @xmath245 respectively , we may write : @xmath307 this shows that @xmath308 . since  @xmath234 is a general non - negative integer that admits eq . [ eq : h_y_sep ] , we may take it to be minimal , _ i.e. _  to be equal to @xmath138    the separation rank of  @xmath101 w.r.t .  @xmath23 . by this we obtain @xmath309 , which is what we set out to prove . the claim is framed in measure theoretical terms , and in accordance , so will its proof be . while a complete introduction to measure theory is beyond our scope ( the interested reader is referred to  @xcite ) , we briefly convey here the intuition behind the concepts we will be using , as well as facts we rely upon . the _ lebesgue measure _ is defined over sets in a euclidean space , and may be interpreted as quantifying their `` volume '' . for example , the lebesgue measure of a unit hypercube is one , of the entire space is infinity , and of a finite set of points is zero . in this context , when a phenomenon is said to occur _ almost everywhere _ , it means that the set of points in which it does not occur has lebesgue measure zero , _ i.e. _  is negligible . an important result we will make use of ( proven in  @xcite for example ) is the following . given a polynomial defined over @xmath310 real variables , the set of points in @xmath258 on which it vanishes is either the entire space ( when the polynomial in question is the zero polynomial ) , or it must have lebesgue measure zero . in other words , if a polynomial is not identically zero , it must be different from zero almost everywhere .    heading on to the proof , we recall from sec .  [ sec : cac ] that the entries of the coefficient tensor  @xmath102 ( eq .  [ eq : h_y ] ) are given by polynomials in the network s conv weights  @xmath99 and output weights  @xmath81 . since @xmath119    the matricization of  @xmath102 w.r.t . the partition  @xmath23 , is merely a rearrangement of the tensor as a matrix , this matrix too has entries given by polynomials in the network s linear weights . now , denote by  @xmath311 the maximal rank taken by  @xmath119 as network weights vary , and consider a specific setting of weights for which this rank is attained . we may assume without loss of generality that under this setting , the top - left @xmath311-by-@xmath311 block of  @xmath119 is non - singular . the corresponding minor , _ i.e. _  the determinant of the sub - matrix  @xmath312 , is thus a polynomial defined over  @xmath99 and  @xmath81 which is not identically zero . in light of the above , this polynomial is different from zero almost everywhere , implying that @xmath313 almost everywhere . since @xmath314 , and since by definition @xmath311 is the maximal rank that  @xmath119 can take , we have that @xmath120 is maximal almost everywhere .      the matrix decomposition in eq . [ eq : mat_hr_decomp ] expresses  @xmath31 in terms of the network s linear weights   @xmath104}$ ] for conv operator in hidden layer  @xmath86 , @xmath105}$ ] for conv operator in hidden layer @xmath106 , and @xmath315 for node  @xmath82 of dense output operator . we prove lower and upper bounds on the maximal rank that  @xmath31 can take as these weights vary . our proof relies on the rank - multiplicative property of the kronecker product ( @xmath158 for any real matrices @xmath316 and @xmath317    see  @xcite for proof ) , but is otherwise elementary . beginning with the lower bound , consider the following weight setting ( @xmath318 here stands for a vector holding  @xmath4 in entry  @xmath80 and  @xmath86 at all other entries , @xmath319  stands for a vector holding  @xmath86 at all entries , and  @xmath320 stands for a vector holding  @xmath4 at all entries , with the dimension of a vector to be understood by context ) : a^0 , & = & \ {    ll e _ & , \{r_0,m } + 0 & ,    . [ eq : weight_setting ] + a^1 , & = & \ {    ll 1 & , = 1 + 0 & ,    . + a^l , & = & \ {    ll e_1 & , = 1 + 0 & ,    . + a^l , y & = & e_1 let @xmath321 $ ] . recalling the definition of  @xmath161 and  @xmath162 from eq . [ eq:_i_lk_j_lk ] , consider the sets  @xmath322 and  @xmath323 , as well as  @xmath324 and  @xmath325 for @xmath326 $ ] . @xmath327  is a partition of  @xmath328 $ ] , _ i.e. _  @xmath329 $ ] , and for every @xmath326 $ ] we have @xmath330 and @xmath331 if  @xmath332 belongs to  @xmath322 , and otherwise @xmath333 and @xmath334 if  @xmath332 belongs to  @xmath323 . this implies that for an arbitrary vector  @xmath335 , the matricization @xmath336 is equal to  @xmath335 if @xmath337 , and to  @xmath338 if @xmath339 . accordingly , for any  @xmath340 $ ] : @xmath341 assume that @xmath342 . by our setting @xmath343 , so the above matrix holds  @xmath4 in a single entry and  @xmath86 in all the rest . moreover , if the matrix is not a row or column vector , _ i.e. _  if both  @xmath322 and  @xmath323 are non - empty , the column index and row index of the entry holding  @xmath4 are both unique w.r.t . @xmath80 , _ i.e. _  they do not repeat as @xmath80 ranges over @xmath344 . we thus have : @xmath345 since we set @xmath346 and @xmath347 for @xmath348 , we may write : @xmath349 the latter matrix is by definition equal to @xmath350 ( see top row of eq . [ eq : mat_hr_decomp ] ) , and so for every @xmath321 $ ] : rank^1,1_i_1,n , j_1,n=\ {    ll \{r_0,m } & , i_1,n  j_1,n + 1 & , i_1,n=  j_1,n=    . [ eq : phi_1_1_rank ] now , the fact that we set @xmath351 and @xmath352 for @xmath353 , implies that the second to last levels of the decomposition in eq . [ eq : mat_hr_decomp ] collapse to : @xmath354 applying the rank - multiplicative property of the kronecker product , and plugging in eq . [ eq : phi_1_1_rank ] , we obtain : @xmath355 where @xmath356 : i_{1,t}\neq\emptyset \wedge j_{1,t}\neq\emptyset\ } \right\rvert}$ ] . this equality holds for the specific weight setting we defined in eq . [ eq : weight_setting ] . maximizing over all weight settings gives the sought after lower bound : @xmath357    moving on to the upper bound , we show by induction over @xmath106 that for any @xmath160 $ ] and @xmath358 $ ] , the rank of  @xmath359 is no greater than  @xmath173 , regardless of the chosen weight setting . for the base case @xmath360 we have : @xmath361 the  @xmath362-by-@xmath363 matrix  @xmath364 is given here as a sum of  @xmath365 rank-1 terms , thus obviously its rank is no greater than @xmath366 . since by definition @xmath367 for all @xmath368 $ ] , we may write : @xmath369 @xmath370  is defined by the right hand side of this inequality , so our inductive hypotheses holds for @xmath360 . for @xmath371 : @xmath372 taking ranks : rank^l,_i_l , k , j_l , k & = & rank(_=1^r_l-1 a_^l , ^l-1,_i_l-1,4(k-1)+t , j_l-1,4(k-1)+t ) + & & _ = 1^r_l-1 rank ( ^l-1,_i_l-1,4(k-1)+t , j_l-1,4(k-1)+t ) + & = & _ = 1^r_l-1_t=1 ^ 4 rank^l-1,_i_l-1,4(k-1)+t , j_l-1,4(k-1)+t + & & _ = 1^r_l-1_t=1 ^ 4 c^l-1,4(k-1)+t + & = & r_l-1_t=1 ^ 4 c^l-1,4(k-1)+t where we used rank sub - additivity in the second line , the rank - multiplicative property of the kronecker product in the third line , and our inductive hypotheses for  @xmath373 in the fourth line . since the number rows and columns in @xmath359 is  @xmath374 and  @xmath375 respectively , we may incorporate these terms into the inequality , obtaining : @xmath376 the right hand side here is equal to  @xmath173 by definition , so our inductive hypotheses indeed holds for all @xmath106 . to establish the sought after upper bound on the rank of @xmath119 , we recall that the latter is given by : @xmath153 carry out a series of steps similar to before , while making use of our inductive hypotheses for @xmath72 : rank^y_i , j & = & rank(_=1^r_l-1 a_^l , y ^l-1,_i_l-1,t , j_l-1,t ) + & & _ = 1^r_l-1 rank ( ^l-1,_i_l-1,t , j_l-1,t ) + & = & _ = 1^r_l-1_t=1 ^ 4 rank^l-1,_i_l-1,t , j_l-1,t + & & _ = 1^r_l-1_t=1 ^ 4 c^l-1,t + & = & r_l-1_t=1 ^ 4 c^l-1,t since  @xmath119 has  @xmath377 rows and  @xmath378 columns , we may include these terms in the inequality , thus reaching the upper bound we set out to prove . our analysis of correlations modeled by convolutional networks is based on the concept of separation rank , conveyed in sec . [ sec : sep_rank ] . when the separation rank of a function w.r.t . a partition of its input is equal to  @xmath4 , the function is separable , meaning it does not model any interaction between sides of the partition . we argued that the higher the separation rank , the farther the function is from this situation , _ i.e. _  the stronger the correlation it induces between sides of the partition . in the current appendix we formalize this argument , by relating separation rank to the @xmath0  distance from the set of separable functions . we begin by defining and characterizing a normalized ( scale invariant ) version of this distance ( app .  [ app : sep_rank_l2:normalized_dist ] ) . it is then shown ( app . [ app : sep_rank_l2:ub_sep_rank ] ) that separation rank provides an upper bound on the normalized distance . finally , a lower bound that applies to deep convolutional arithmetic circuits is derived ( app . [ app : sep_rank_l2:lb_cac ] ) , based on the lower bound for their separation ranks established in sec . [ sec : analysis : deep ] . together , these steps imply that our entire analysis , facilitated by upper and lower bounds on separation ranks of convolutional arithmetic circuits , can be interpreted as based on upper and lower bounds on ( normalized ) @xmath0  distances from separable functions .    in the text hereafter , we assume familiarity of the reader with the contents of sec .  [ sec : prelim ] ,  [ sec : cac ] ,  [ sec : sep_rank ] ,  [ sec : analysis ] and the proofs given in app . [ app : proofs ] . we also rely on basic knowledge in the topic of @xmath0  spaces ( see discussion in app . [ app : proofs : sep_mat_ranks_equal ] for minimal background required in order to follow our arguments ) , as well as several results concerning singular values of matrices . in line with sec . [ sec : analysis ] , an assumption throughout this appendix is that all functions in question are measurable and square - integrable ( _ i.e. _  belong to  @xmath0 over the respective euclidean space ) , and in app . [ app : sep_rank_l2:lb_cac ] , we also make use of the fact that representation functions  ( @xmath75 ) of a convolutional arithmetic circuit can be regarded as linearly independent ( see sec .  [ sec : analysis : sep2mat ] ) . finally , for convenience , we now fix  @xmath23    an arbitrary partition of  @xmath24 $ ] . specifically , @xmath25  and  @xmath26 are disjoint subsets of  @xmath24 $ ] whose union gives  @xmath24 $ ] , denoted by @xmath27 with @xmath28 , and @xmath29 with @xmath30 .      for a function @xmath128 ( which is not identically zero ) , the _ normalized @xmath0 distance from the set of separable functions w.r.t . @xmath23 _ , is defined as follows : d(h;i , j):=_h(x_1,  ,x_n)-g(x_i_1,  ,x_i_)g(x_j_1,  ,x_j _ ) [ eq : normalized_dist ] where  @xmath379 refers to the norm of  @xmath0 space , _ e.g. _  @xmath380 . in words , @xmath137 is defined as the minimal @xmath0  distance between  @xmath127 and a function that is separable w.r.t . @xmath23 , divided by the norm of  @xmath127 . the normalization ( division by  @xmath381 ) admits scale invariance to  @xmath137 , and is of critical importance    without it , rescaling  @xmath127 would accordingly rescale the distance measure , rendering the latter uninformative in terms of deviation from separability . it is worthwhile noting the resemblance between  @xmath137 and the concept of mutual information ( see  @xcite for a comprehensive introduction ) . both measures quantify the interaction that a normalized function is the minimal @xmath0 distance between  @xmath382 and a function separable w.r.t . accordingly , we may view  @xmath137 as operating on normalized functions . ] induces between input variables , by measuring distance from separable functions . the difference between the measures is threefold . first , mutual information considers probability density functions ( non - negative and in  @xmath383 ) , while  @xmath137 applies to functions in  @xmath0 . second , the notion of distance in mutual information is quantified through the kullback - leibler divergence , whereas in  @xmath137 it is simply the @xmath0  metric . third , while mutual information evaluates the distance from a specific separable function    product of marginal distributions , @xmath137  evaluates the minimal distance across all separable functions . we now turn to establish a spectral characterization of  @xmath137 , which will be used in app . [ app : sep_rank_l2:ub_sep_rank ] and  [ app : sep_rank_l2:lb_cac ] for deriving upper and lower bounds ( respectively ) . assume we have the following expression for  @xmath127 : h(x_1,  ,x_n)=_=1^m_=1^ma_,_(x_i_1,  ,x_i_)_(x_j_1,  ,x_j _ ) [ eq : orth_sep_decomp ] where  @xmath384 and  @xmath385 are positive integers , @xmath316  is an @xmath384-by-@xmath385 real matrix , and @xmath386 are orthonormal sets of functions in  @xmath387 respectively . we refer to such expression as an _ orthonormal separable decomposition _ of  @xmath127 , with  @xmath316 being its _ coefficient matrix_. we will show that for any orthonormal separable decomposition , @xmath137  is given by the following formula : d(h;i , j)= [ eq : normalized_dist_formula ] where @xmath388 are the singular values of the coefficient matrix  @xmath316 . this implies that if the largest singular value of  @xmath316 accounts for a significant portion of the spectral energy , the normalized @xmath0  distance of  @xmath127 from separable functions is small . on the other hand , if all but a fraction of the spectral energy is attributed to trailing singular values , @xmath127  is far from being separable ( @xmath137 is close to  @xmath4 ) .    as a first step in deriving eq . [ eq : normalized_dist_formula ] , we show that @xmath389 : h ^2 & & + & & ( _ = 1^m_=1^ma_,_(x_i_1,  ,x_i_)_(x_j_1,  ,x_j_))^2dx_1dx_n + & & _ , |=1^m_,|=1^ma_,a_|,|_(x_i_1,  ,x_i_)_(x_j_1,  ,x_j _ ) + & & _ |(x_i_1,  ,x_i_)_|(x_j_1,  ,x_j_)dx_1dx_n + & & _ , |=1^m_,|=1^ma_,a_|,|_(x_i_1,  ,x_i_)_(x_j_1,  ,x_j _ ) + & & _ |(x_i_1,  ,x_i_)_|(x_j_1,  ,x_j_)dx_1dx_n + & & _ , |=1^m_,|=1^ma_,a_|,|_(x_i_1,  ,x_i_)_|(x_i_1,  ,x_i_)dx_i_1dx_i _ + & & _(x_j_1,  ,x_j_)_|(x_j_1,  ,x_j_)dx_j_1dx_j _ + & & _ , |=1^m_,|=1^ma_,a_|,| \ {    ll 1 & , =| + 0 & ,    } \ {    ll 1 & , =| + 0 & ,    } + & & _ = 1^m_=1^ma_,^2 + & & _ 1 ^ 2(a)++_\{m , m}^2(a ) [ eq : orth_sep_decomp_norm ] equality  @xmath390 here originates from the definition of @xmath0  norm . @xmath391  is obtained by plugging in the expression in eq . [ eq : orth_sep_decomp ] . @xmath392  is merely an arithmetic manipulation . @xmath393  follows from the linearity of integration . @xmath394  makes use of fubini s theorem ( see  @xcite ) . @xmath395  results from the orthonormality of @xmath396 and @xmath397 . @xmath398  is a trivial computation . finally , @xmath399  is an outcome of the fact that the squared frobenius norm of a matrix , _ i.e. _  the sum of squares over its entries , is equal to the sum of squares over its singular values ( see  @xcite for proof ) . let @xmath400 . by fact  [ fact : orth_decomp ] in app . [ app : proofs : sep_mat_ranks_equal ] , there exist scalars @xmath401 , and a function @xmath402 orthogonal to @xmath403 , such that @xmath404 . similarly , for any @xmath405 there exist @xmath406 and @xmath407 such that @xmath408 . fact  [ fact : prod ] in app . [ app : proofs : sep_mat_ranks_equal ] indicates that the function given by @xmath409 belongs to  @xmath303 . we may express it as follows : g(x_i_1,  ,x_i_)g(x_j_1,  ,x_j _ ) & = & _ = 1^m_=1^m___(x_i_1,  ,x_i_)_(x_j_1,  ,x_j _ ) + & & + ( _ = 1^m__(x_i_1,  ,x_i_))(x_j_1,  ,x_j _ ) + & & + ( x_i_1,  ,x_i_)(_=1^m__(x_j_1,  ,x_j _ ) ) + & & + ( x_i_1,  ,x_i_)(x_j_1,  ,x_j _ ) according to fact  [ fact : prod_orth ] in app . [ app : proofs : sep_mat_ranks_equal ] , the second , third and fourth terms on the right hand side of the above are orthogonal to @xmath410,\mu'\in[m']}$ ] . denote their summation by  @xmath411 , and subtract the overall function from  @xmath127 ( given by eq . [ eq : orth_sep_decomp ] ) : & & h(x_1,  ,x_n)-g(x_i_1,  ,x_i_)g(x_j_1,  ,x_j _ ) + & & = _ = 1^m_=1^ma_,_(x_i_1,  ,x_i_)_(x_j_1,  ,x_j _ ) + & & -_=1^m_=1^m___(x_i_1,  ,x_i_)_(x_j_1,  ,x_j_)-e(x_1,  ,x_n ) + & & = _ = 1^m_=1^m(a_,-__)_(x_i_1,  ,x_i_)_(x_j_1,  ,x_j_)-e(x_1,  ,x_n ) since the two terms in the latter expression are orthogonal to one another , we have : h(x_1,  ,x_n)-g(x_i_1,  ,x_i_)g(x_j_1,  ,x_j _ ) ^2 + = _ = 1^m_=1^m(a_,-__)_(x_i_1,  ,x_i_)_(x_j_1,  ,x_j _ ) ^2+(x_1,  ,x_n ) ^2 applying a sequence of steps as in eq . [ eq : orth_sep_decomp_norm ] to the first term in the second line of the above , we obtain : @xmath412 @xmath413 if  @xmath267 and  @xmath414 are the zero functions , implying that : @xmath415 with equality holding if  @xmath416 and  @xmath417 . now , @xmath418 is the squared frobenius distance between the matrix  @xmath316 and the rank-@xmath4 matrix  @xmath419 , where  @xmath420 and  @xmath421 are column vectors holding @xmath422 and @xmath423 respectively . this squared distance is greater than or equal to the sum of squares over the second to last singular values of  @xmath316 , and moreover , the inequality holds with equality for proper choices of  @xmath420 and  @xmath421 ( @xcite ) . from this we conclude that : @xmath424 with equality holding if  @xmath425 and  @xmath426 are set to  @xmath427 and  @xmath428 ( respectively ) for proper choices of @xmath422 and @xmath423 . we thus have the infimum over all possible  @xmath429 : _ h(x_1,  ,x_n)-g(x_i_1,  ,x_i_)g(x_j_1,  ,x_j _ ) ^2=_2 ^ 2(a)++_\{m , m}^2(a ) [ eq : unnormalized_dist_sq_formula ]    recall that we would like to derive the formula in eq . [ eq : normalized_dist_formula ] for @xmath137 , assuming  @xmath127 is given by the orthonormal separable decomposition in eq . [ eq : orth_sep_decomp ] . taking square root of the equalities established in eq . [ eq : orth_sep_decomp_norm ] and  [ eq : unnormalized_dist_sq_formula ] , and plugging them into the definition of  @xmath137 ( eq .  [ eq : normalized_dist ] ) , we obtain the sought after result . we now relate @xmath137    the normalized @xmath0 distance of @xmath128 from the set of separable functions w.r.t . @xmath23 ( eq .  [ eq : normalized_dist ] ) , to @xmath136    the separation rank of  @xmath127 w.r.t . @xmath23 ( eq .  [ eq : sep_rank ] ) . specifically , we make use of the formula in eq . [ eq : normalized_dist_formula ] to derive an upper bound on  @xmath137 in terms of  @xmath136 . assuming  @xmath127 has finite separation rank ( otherwise the bound we derive is trivial ) , we may express it as : h(x_1,  ,x_n)=_=1^rg_(x_i_1,  ,x_i_)g_(x_j_1,  ,x_j _ ) [ eq : sep_decomp ] where  @xmath234 is some positive integer ( necessarily greater than or equal to  @xmath136 ) , and @xmath288 , @xmath289 . let @xmath430 and @xmath431 be two sets of orthonormal functions spanning @xmath432 and @xmath433 respectively . by definition , for every @xmath434 there exist @xmath435 and @xmath436 such that @xmath437 and @xmath438 . plugging this into eq . [ eq : sep_decomp ] , we obtain : @xmath439 this is an orthonormal separable decomposition of  @xmath127 ( eq .  [ eq : orth_sep_decomp ] ) , with coefficient matrix  @xmath440 , where @xmath441^{\top}$ ] and @xmath442^{\top}$ ] for every  @xmath434 . obviously the rank of  @xmath316 is no greater than  @xmath234 , implying : @xmath443 where as in app . [ app : sep_rank_l2:normalized_dist ] , @xmath388 stand for the singular values of  @xmath316 . introducing this inequality into eq . [ eq : normalized_dist_formula ] gives : @xmath444 the latter holds for any @xmath445 that admits eq . [ eq : sep_decomp ] , so in particular we may take it to be minimal , _ i.e. _  to be equal to  @xmath136 ( @xmath127  is identically zero ) . ] , bringing forth the sought after upper bound : d(h;i , j ) [ eq : normalized_dist_ub ]    by eq .  [ eq : normalized_dist_ub ] , low separation rank implies proximity ( in normalized @xmath0 sense ) to a separable function . we may use the inequality to translate the upper bounds on separation ranks established for deep and shallow convolutional arithmetic circuits ( sec . [ sec : analysis : deep ] and  [ sec : analysis : shallow ] respectively ) , into upper bounds on normalized @xmath0 distances from separable functions . to completely frame our analysis in terms of the latter measure , a translation of the lower bound on separation ranks of deep convolutional arithmetic circuits ( sec . [ sec : analysis : deep ] ) is also required . eq .  [ eq : normalized_dist_ub ] does not facilitate such translation , and in fact , it is easy to construct functions  @xmath127 whose separation ranks are high yet are very close ( in normalized @xmath0 sense ) to separable functions . is given by an orthonormal separable decomposition ( eq .  [ eq : orth_sep_decomp ] ) , with coefficient matrix  @xmath316 that has high rank but whose spectral energy is highly concentrated on one singular value . ] however , as we show in app . [ app : sep_rank_l2:lb_cac ] below , the specific lower bound of interest can indeed be translated , and our analysis may entirely be framed in terms of normalized @xmath0 distance from separable functions . let @xmath287 be a function realized by a deep convolutional arithmetic circuit ( fig . [ fig : nets_patterns](a ) with size-@xmath1 pooling windows and @xmath103 hidden layers ) , _ i.e. _  @xmath101 is given by eq . [ eq : h_y ] , where @xmath286 are linearly independent representation functions , and  @xmath102 is a coefficient tensor of order  @xmath6 and dimension  @xmath62 in each mode , determined by the linear weights of the network ( @xmath446 ) through the hierarchical decomposition in eq . [ eq : hr_decomp ] . rearrange eq . [ eq : h_y ] by grouping indexes @xmath447 in accordance with the partition  @xmath23 : h_y(x_1,  ,x_n)= _ d_i_1  d_i_=1^m _ d_j_1  d_j_=1^m a_d_1  d_n^y(_t=1^ f__d_i_t(x_i_t ) ) ( _ t=1^ f__d_j_t(x_j_t ) ) [ eq : h_y_i_j ] let  @xmath448 , and define the following mapping : @xmath449^{{\left\lverti \right\rvert}}\to[m]\quad,\quad \mu(d_{i_1},\ldots , d_{i_{{\left\lverti \right\rvert}}})=1+\sum\nolimits_{t=1}^{{\left\lverti \right\rvert}}(d_{i_t}-1){\cdot}m^{{\left\lverti \right\rvert}-t}\ ] ] @xmath450  is a one - to - one correspondence between the index sets  @xmath451^{{\left\lverti \right\rvert}}$ ] and  @xmath452 $ ] . we slightly abuse notation , and denote by @xmath453 the tuple in  @xmath451^{{\left\lverti \right\rvert}}$ ] that maps to  @xmath454 $ ] . additionally , we denote the function @xmath455 , which according to fact  [ fact : prod ] in app . [ app : proofs : sep_mat_ranks_equal ] belongs to  @xmath456 , by  @xmath457 . in the exact same manner , we let  @xmath458 , and define the bijective mapping : @xmath459^{{\left\lvertj \right\rvert}}\to[m']\quad,\quad \mu'(d_{j_1},\ldots , d_{j_{{\left\lvertj \right\rvert}}})=1+\sum\nolimits_{t=1}^{{\left\lvertj \right\rvert}}(d_{j_t}-1){\cdot}m^{{\left\lvertj \right\rvert}-t}\ ] ] as before , @xmath460 stands for the tuple in  @xmath451^{{\left\lvertj \right\rvert}}$ ] that maps to  @xmath461 $ ] , and the function @xmath462 is denoted by  @xmath463 . now , recall the definition of matricization given in sec . [ sec : prelim ] , and consider @xmath119    the matricization of the coefficient tensor  @xmath102 w.r.t . this is a matrix of size @xmath384-by-@xmath385 , holding @xmath34 in row index @xmath464 and column index @xmath465 . rewriting eq . [ eq : h_y_i_j ] with the indexes  @xmath450 and  @xmath466 instead of @xmath467 and @xmath468 , we obtain : h_y(x_1,  ,x_n)= _ = 1^m _ =1^m ( ^y_i , j)_,_(x_i_1,  ,x_i _ ) _(x_j_1,  ,x_j _ ) [ eq : h_y_i_j_mat ] this equation has the form of eq . [ eq : orth_sep_decomp ] . however , for it to qualify as an orthonormal separable decomposition , the sets of functions @xmath430 and @xmath431 must be orthonormal . if the latter holds eq . [ eq : normalized_dist_formula ] may be applied , giving an expression for @xmath469    the normalized @xmath0 distance of  @xmath101 from the set of separable functions w.r.t . @xmath23 , in terms of the singular values of  @xmath119 . we now direct our attention to the special case where @xmath286    the network s representation functions , are known to be orthonormal . the general setting , in which only linear independence is known , will be treated thereafter . orthonormality of representation functions implies that @xmath470 are orthonormal as well : , _ | & & _ ( x_i_1,  ,x_i_)_|(x_i_1,  ,x_i_)dx_i_1dx_i _ + & & _ t=1^f__d_i_t()(x_i_t)_t=1^f__d_i_t(|)(x_i_t)dx_i_1dx_i _ + & & _ t=1^f__d_i_t()(x_i_t)f__d_i_t(|)(x_i_t)dx_i_t + & & _ t=1^,f__d_i_t(| ) + & & _ t=1^ \ {    ll 1 & , d_i_t()=d_i_t(| ) + 0 & ,    } + & & \ {    ll 1 & , d_i_t()=d_i_t(| )   + 0 & ,    . + & & \ {    ll 1 & , =| + 0 & ,    . @xmath390  and  @xmath393 here follow from the definition of inner product in @xmath0 space , @xmath391  replaces  @xmath471 and  @xmath472 by their definitions , @xmath392  makes use of fubini s theorem ( see  @xcite ) , @xmath394  relies on the ( temporary ) assumption that representation functions are orthonormal , @xmath395  is a trivial step , and @xmath398  owes to the fact that @xmath473 is an injective mapping . a similar sequence of steps ( applied to @xmath474 ) shows that in addition to @xmath475 , the functions @xmath476 will also be orthonormal if @xmath477 are . we conclude that if representation functions are orthonormal , eq .  [ eq : h_y_i_j_mat ] indeed provides an orthonormal separable decomposition of  @xmath101 , and the formula in eq . [ eq : normalized_dist_formula ] may be applied : d(h_y;i , j)= [ eq : normalized_dist_formula_cac ] where @xmath478 are the singular values of the coefficient tensor matricization  @xmath119 .    in sec . [ sec : analysis : deep ] we showed that the maximal separation rank realizable by a deep network is greater than or equal to  @xmath163 , where  @xmath479 are the number of channels in the representation and first hidden layers ( respectively ) , and  @xmath170 stands for the number of index quadruplets ( sets of the form @xmath480 for some  @xmath481 $ ] ) that are split by the partition  @xmath23 . to prove this lower bound , we presented in app . [ app : proofs : deep_mat_rank_bounds ] a specific setting for the linear weights of the network ( @xmath446 ) under which @xmath482 . careful examination of the proof shows that with this particular weight setting , not only is the rank of  @xmath119 equal to  @xmath163 , but also , all of its non - zero singular values are equal to one another.$ ] , @xmath350  has one of two forms : it is either a non - zero ( row / column ) vector , or it is a matrix holding  @xmath4 in several entries and  @xmath86 in all the rest , where any two entries holding  @xmath4 reside in different rows and different columns . the first of the two forms admits a single non - zero singular value . the second brings forth several singular values equal to  @xmath4 , possibly accompanied by null singular values . in both cases , all non - zero singular values of  @xmath350 are equal to one another . now , since @xmath483 , and since the kronecker product multiplies singular values ( see  @xcite ) , we have that all non - zero singular values of  @xmath119 are equal , as required . ] this implies that @xmath484 , and since we currently assume that @xmath477 are orthonormal , eq .  [ eq : normalized_dist_formula_cac ] applies and we obtain @xmath485 . maximizing over all possible weight settings , we arrive at the following lower bound for the normalized @xmath0 distance from separable functions brought forth by a deep convolutional arithmetic circuit : _ \{a^l,}_l ,  ,  a^l , yd(h_y|_\{a^l,}_l,,a^l , y  ; i , j ) [ eq : normalized_dist_max_lb ]    turning to the general case , we omit the assumption that representation functions @xmath286 are orthonormal , and merely rely on their linear independence . the latter implies that the dimension of @xmath486 is  @xmath62 , thus there exist orthonormal functions @xmath487 that span it . let  @xmath488 be a transition matrix between the bases    the matrix defined by @xmath489}$ ] . suppose now that we replace the original representation functions @xmath477 by the orthonormal ones @xmath490 . using the latter , the lower bound in eq . [ eq : normalized_dist_max_lb ] applies , and there exists a setting for the linear weights of the network   @xmath446 , such that @xmath491 . recalling the structure of convolutional arithmetic circuits ( fig . [ fig : nets_patterns](a ) ) , one readily sees that if we return to the original representation functions @xmath477 , while multiplying conv weights in hidden layer  @xmath86 by  @xmath492 ( _ i.e. _  mapping @xmath493 ) , the overall function  @xmath101 remains unchanged , and in particular @xmath491 still holds . we conclude that the lower bound in eq . [ eq : normalized_dist_max_lb ] applies , even if representation functions are not orthonormal .    to summarize , we translated the lower bound from sec . [ sec : analysis : deep ] on the maximal separation rank realized by a deep convolutional arithmetic circuit , into a lower bound on the maximal normalized @xmath0 distance from separable functions ( eq .  [ eq : normalized_dist_max_lb ] ) . this , along with the translation of upper bounds facilitated in app . [ app : sep_rank_l2:ub_sep_rank ] , implies that the analysis carried out in the paper , which studies correlations modeled by convolutional networks through the notion of separation rank , may equivalently be framed in terms of normalized @xmath0 distance from separable functions . we note however that there is one particular aspect in our original analysis that does not carry through the translation . namely , in sec .  [ sec : analysis : sep2mat ] it was shown that separation ranks realized by convolutional arithmetic circuits are maximal almost always , _ i.e. _  for all linear weight settings but a set of ( lebesgue ) measure zero . put differently , for a given partition  @xmath23 , the maximal separation rank brought forth by a network characterizes almost all functions realized by it . an equivalent statement does not hold with the continuous measure of normalized @xmath0 distance from separable functions . the behavior of this measure across the hypotheses space of a network is non - trivial , and forms a subject for future research . in this appendix we provide implementation details omitted from the description of our experiments in sec . [ sec : exp ] . our implementation , available online at https://github.com/huji-deep/inductive-pooling , is based on the simnets branch ( ) of caffe toolbox ( @xcite ) . the latter realizes convolutional arithmetic circuits in log - space for numerical stability . when training convolutional arithmetic circuits , we followed the hyper - parameter choices made by  . in particular , our objective function was the cross - entropy loss with no @xmath0  regularization ( _ i.e. _  with weight decay set to  @xmath86 ) , optimized using adam ( @xcite ) with step - size  @xmath494 and moment decay rates  @xmath495 . @xmath496 iterations with batch size  @xmath212 ( @xmath497  epochs ) were run , with the step - size  @xmath110 decreasing by a factor of  @xmath498 after @xmath499 iterations ( @xmath500  epochs ) . we did not use dropout  ( @xcite ) , as the limiting factor in terms of accuracies was the difficulty of fitting training data ( as opposed to overfitting )    see fig . [ fig : exp_results_cac_pool4 ] . for training the conventional convolutional rectifier networks , we merely switched the hyper - parameters of adam to the recommended settings specified in  @xcite ( @xmath501 ) , and set weight decay to the standard value of  @xmath502 . the synthetic dataset used in our experiments ( sec . [ sec : exp ] ) consists of binary images displaying different shapes ( blobs ) . one of the tasks facilitated by this dataset is the detection of morphologically closed blobs , _ i.e. _  of images that are relatively similar to their morphological closure . the procedure we followed for computing the morphological closure of a binary image is :    1 . pad the given image with background ( @xmath86  value ) pixels 2 . morphological dilation : simultaneously turn on ( set to  @xmath4 ) all pixels that have a ( left , right , top or bottom ) neighbor originally active ( holding  @xmath4 ) 3 . morphological erosion : simultaneously turn off ( set to  @xmath86 ) all pixels that have a ( left , right , top or bottom ) neighbor currently inactive ( holding  @xmath86 ) 4 .   remove pixels introduced in padding    it is not difficult to see that any pixel active in the original image is necessarily active in its closure . moreover , pixels that are originally inactive yet are surrounded by active ones will also be turned on in the closure , hence the effect of `` gap filling '' . finally , we note that the particular sequence of steps described above represents the most basic form of morphological closure . the interested reader is referred to  @xcite for a much more comprehensive introduction .
Abstract:

Answer: our formal understanding of the inductive bias that drives the success of convolutional networks on computer vision tasks is limited . in particular , it is unclear what makes hypotheses spaces born from convolution and pooling operations so suitable for natural images . in this paper we study the ability of convolutional networks to model correlations among regions of their input . we theoretically analyze convolutional arithmetic circuits , and empirically validate our findings on other types of convolutional networks as well . correlations are formalized through the notion of separation rank , which for a given partition of the input , measures how far a function is from being separable . we show that a polynomially sized deep network supports exponentially high separation ranks for certain input partitions , while being limited to polynomial separation ranks for others . the network s pooling geometry effectively determines which input partitions are favored , thus serves as a means for controlling the inductive bias . contiguous pooling windows as commonly employed in practice favor interleaved partitions over coarse ones , orienting the inductive bias towards the statistics of natural images . other pooling schemes lead to different preferences , and this allows tailoring the network to data that departs from the usual domain of natural imagery . in addition to analyzing deep networks , we show that shallow ones support only linear separation ranks , and by this gain insight into the benefit of functions brought forth by depth they are able to efficiently model strong correlation under favored partitions of the input .