Hello, I tested the code in the Basic Usage section of the README.md and result = model.forward(video, instructions)
results in the following error.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/robocat/model.py", line 762, in forward
logits = self.model(video, text)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<@beartype(robocat.model.Gato.forward) at 0x7f4bb9d9cf40>", line 46, in forward
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/classifier_free_guidance_pytorch/classifier_free_guidance_pytorch.py", line 137, in inner
return fn_maybe_with_text(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/classifier_free_guidance_pytorch/classifier_free_guidance_pytorch.py", line 130, in fn_maybe_with_text
return fn(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/robocat/model.py", line 691, in forward
learned_tokens = self.token_learner(tokens)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/robocat/model.py", line 509, in forward
attn = self.net(x)
^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/container.py", line 215, in forward
input = module(input)
^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 460, in forward
return self._conv_forward(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/robocat/miniforge3/envs/robocat/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Given groups=8, weight of size [12288, 768, 1, 1], expected input[12, 512, 112, 112] to have 6144 channels, but got 512 channels instead
Also there are a bunch of robotcat instead of robocat in the readme.
Pay now to fund the work behind this issue.
Get updates on progress being made.
Maintainer is rewarded once the issue is completed.
You're funding impactful open source efforts
You want to contribute to this effort
You want to get funding like this too