Backward convolution

[ducoffeM]

• previous method get_affine_components did not scale to large convolution filters
• replace this method by deriving the affine weights with the get_toeplitz method
• get_toeplitz use extract_patches and im2col which is the recommandation of tensorflow for computing the affine versions of convolution
• two versions are implemented in backward_layers/utils_conv: data_format = [channels_first, channels_last]
• associated tests are available in test_backward_utils_conv

[nhuet]

Beware:

• The first commit about notebooks belongs to another PR (Cleaning existing notebooks and add a new header #55).
• The new test file mentioned in the description is missing (test_backward_utils_conv.py).

[nhuet]

Great description. But if we are to follow PEP-257, "Multi-line docstrings consist of a summary line just like a one-line docstring, followed by a blank line, followed by a more elaborate description."

So maybe something like

"""Express the implicit affine matrix of the convolution layer.

Conv is a linear operator but its affine component is implicit
we use im2col and extract_patches to express the affine matrix
Note that this matrix is Toeplitz

Args:
     x: list of input tensors
Returns:
     the affine operators W, b : conv(x)= Wx + b
"""

would be better?

[nhuet]

I think you can actually squash the last commits into first one (as they are only corrections for the first one). LGTM once the commented test is removed.

[nhuet]

OOops, some commented code slipped out ^^.
It should be removed for better readibility.

[nhuet]

It is okay now. Actually my comment was quite general as other functions in the new code was presenting the same kind of welll detailed but misformatted docstring. Maybe we could cope with all those docstrings in another PR because there is work throughout the library to make all docstrings be compliant with pep 257. Or you can make at least the new docstings respect it as you wish.

[nhuet]

When running the tests locally some are failing.
It concerns only "channels_first" tests that are not skipped on my laptop as i used a GPU, but must be skipped on github runner.

Example of failed test:

____________________________________________________________________________________________________________ test_toeplitz_from_Decomon[11-3-2-False-channels_first-valid-64] ____________________________________________________________________________________________________________

channels = 11, filter_size = 3, strides = 2, flatten = False, data_format = 'channels_first', padding = 'valid', floatx = 64, helpers = <class 'tests.conftest.Helpers'>

def test_toeplitz_from_Decomon(channels, filter_size, strides, flatten, data_format, padding, floatx, helpers):

    # filter_size, strides, flatten,
    K.set_floatx("float{}".format(floatx))
    eps = K.epsilon()
    decimal = 5
    if floatx == 16:
        K.set_epsilon(1e-2)
        decimal = 0

    if data_format == "channels_first" and not len(K._get_available_gpus()):
        return

    # filter_size=3
    # strides=1
    # flatten = True
    odd, m_0, m_1 = 0, 0, 1

    # should be working either with convolution of conv2D
    layer = Conv2D(
        channels, (filter_size, filter_size), strides=strides, use_bias=False, padding=padding, dtype=K.floatx()
    )

    inputs = helpers.get_tensor_decomposition_images_box(data_format, odd)
    inputs_ = helpers.get_standard_values_images_box(data_format, odd, m0=m_0, m1=m_1)
    y = inputs[1]

  result_ref = layer(y)

tests/test_backward_utils_conv.py:94:

---

../../Progs/miniconda3/envs/deco-gpu/lib/python3.9/site-packages/keras/utils/traceback_utils.py:70: in error_handler
raise e.with_traceback(filtered_tb) from None

---

graph = <tensorflow.python.framework.func_graph.FuncGraph object at 0x7f347584ea30>
node_def = name: "conv2d_862/Conv2D"
op: "Conv2D"
attr {
key: "T"
value {
type: DT_DOUBLE
}
}
attr {
key: "data_forma...{
i: 1
i: 2
i: 2
i: 1
}
}
}
attr {
key: "use_cudnn_on_gpu"
value {
b: true
}
}

inputs = [<tf.Tensor 'Placeholder:0' shape=(None, 2, 6, 6) dtype=float64>, <tf.Tensor 'conv2d_862/Conv2D/ReadVariableOp:0' shape=(3, 3, 6, 11) dtype=float64>], control_inputs = []
op_def = name: "Conv2D"
input_arg {
name: "input"
type_attr: "T"
}
input_arg {
name: "filter"
type_attr: "T"
}
output_a...: "dilations"
type: "list(int)"
default_value {
list {
i: 1
i: 1
i: 1
i: 1
}
}
}
, extract_traceback = True

@tf_export("__internal__.create_c_op", v1=[])
@traceback_utils.filter_traceback
def _create_c_op(graph,
                 node_def,
                 inputs,
                 control_inputs,
                 op_def=None,
                 extract_traceback=True):
  """Creates a TF_Operation.

  Args:
    graph: a `Graph`.
    node_def: `node_def_pb2.NodeDef` for the operation to create.
    inputs: A flattened list of `Tensor`s. This function handles grouping
      tensors into lists as per attributes in the `node_def`.
    control_inputs: A list of `Operation`s to set as control dependencies.
    op_def: Optional. `op_def_pb2.OpDef` for the operation to create. If not
      specified, is looked up from the `graph` using `node_def.op`.
    extract_traceback: if True, extract the current Python traceback to the
      TF_Operation.

  Returns:
    A wrapped TF_Operation*.
  """
  if op_def is None:
    op_def = graph._get_op_def(node_def.op)  # pylint: disable=protected-access
  # TODO(skyewm): op_def_library.apply_op() flattens the incoming inputs.
  # Refactor so we don't have to do this here.
  inputs = _reconstruct_sequence_inputs(op_def, inputs, node_def.attr)
  # pylint: disable=protected-access
  with graph._c_graph.get() as c_graph:
    op_desc = pywrap_tf_session.TF_NewOperation(c_graph,
                                                compat.as_str(node_def.op),
                                                compat.as_str(node_def.name))
  if node_def.device:
    pywrap_tf_session.TF_SetDevice(op_desc, compat.as_str(node_def.device))
  # Add inputs
  for op_input in inputs:
    if isinstance(op_input, (list, tuple)):
      pywrap_tf_session.TF_AddInputList(op_desc,
                                        [t._as_tf_output() for t in op_input])
    else:
      pywrap_tf_session.TF_AddInput(op_desc, op_input._as_tf_output())

  # Add control inputs
  for control_input in control_inputs:
    pywrap_tf_session.TF_AddControlInput(op_desc, control_input._c_op)
  # pylint: enable=protected-access

  # Add attrs
  for name, attr_value in node_def.attr.items():
    serialized = attr_value.SerializeToString()
    # TODO(skyewm): this creates and deletes a new TF_Status for every attr.
    # It might be worth creating a convenient way to re-use the same status.
    pywrap_tf_session.TF_SetAttrValueProto(op_desc, compat.as_str(name),
                                           serialized)

  try:
    c_op = pywrap_tf_session.TF_FinishOperation(op_desc)
  except errors.InvalidArgumentError as e:
    # Convert to ValueError for backwards compatibility.

  raise ValueError(e.message)

E ValueError: Exception encountered when calling layer "conv2d_862" (type Conv2D).
E
E Negative dimension size caused by subtracting 3 from 2 for '{{node conv2d_862/Conv2D}} = Conv2D[T=DT_DOUBLE, data_format="NHWC", dilations=[1, 1, 1, 1], explicit_paddings=[], padding="VALID", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true](Placeholder, conv2d_862/Conv2D/ReadVariableOp)' with input shapes: [?,2,6,6], [3,3,6,11].
E
E Call arguments received by layer "conv2d_862" (type Conv2D):
E • inputs=tf.Tensor(shape=(None, 2, 6, 6), dtype=float64)

../../Progs/miniconda3/envs/deco-gpu/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:1967: ValueError

List of failed tests:

FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-1-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_105" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-1-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_106" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-1-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_107" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-1-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_117" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-1-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_118" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-1-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_119" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-2-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_129" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-2-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_130" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-2-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_131" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-2-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_141" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-2-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_142" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[1-3-2-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_143" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-1-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_249" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-1-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_250" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-1-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_251" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-1-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_261" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-1-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_262" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-1-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_263" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-2-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_273" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-2-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_274" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-2-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_275" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-2-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_285" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-2-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_286" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[2-3-2-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_287" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-1-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_393" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-1-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_394" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-1-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_395" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-1-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_405" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-1-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_406" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-1-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_407" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-2-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_417" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-2-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_418" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-2-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_419" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-2-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_429" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-2-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_430" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Keras[11-3-2-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_431" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-1-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_537" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-1-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_538" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-1-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_539" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-1-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_549" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-1-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_550" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-1-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_551" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-2-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_561" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-2-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_562" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-2-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_563" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-2-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_573" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-2-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_574" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[1-3-2-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_575" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-1-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_681" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-1-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_682" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-1-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_683" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-1-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_693" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-1-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_694" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-1-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_695" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-2-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_705" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-2-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_706" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-2-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_707" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-2-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_717" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-2-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_718" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[2-3-2-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_719" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-1-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_825" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-1-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_826" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-1-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_827" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-1-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_837" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-1-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_838" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-1-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_839" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-2-True-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_849" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-2-True-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_850" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-2-True-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_851" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-2-False-channels_first-valid-32] - ValueError: Exception encountered when calling layer "conv2d_861" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-2-False-channels_first-valid-64] - ValueError: Exception encountered when calling layer "conv2d_862" (type Conv2D).
FAILED tests/test_backward_utils_conv.py::test_toeplitz_from_Decomon[11-3-2-False-channels_first-valid-16] - ValueError: Exception encountered when calling layer "conv2d_863" (type Conv2D).