Model's quantizer bits may have a CPU/GPU mismatch due to the construction logic in quantizers.py. We see differences between "normal" (non-HPO) training and HPO training using hyperparameters_optimization.py.
In normal training whether the model is moved to CPU/GPU or not for the initial forward pass (to construct the full model). In HPO, this is taken care of by the objective function in hyperparameters_optimization.py:
Normal training:
Device | PQ Quantizer | HGQ Quantizer
CPU | Works | Works
GPU | Fails | Fails
HPO:
Device | PQ Quantizer | HGQ Quantizer
CPU | fails at second trial (reinitialization) | fails directly
GPU | fails directly | fails directly
I suspect this is caused by the following line, which does not place the parameter on the CUDA device at initialization. This makes PyTorch store the parameter as a CPU-only tensor, despite the move of the (entire) model to GPU afterwards so it is fine for normal, non-HPO training. See also #64 .
self.k = torch.nn.Parameter(torch.full(param_shape, float(k)), requires_grad=False)
For normal training, doing the forward pass on CPU should not be a problem. For HPO, I have not discovered and therefore cannot propose any permanent workaround yet that fixes this issue but by hard-coding device=device in the quantizer initialization..
Model's quantizer bits may have a CPU/GPU mismatch due to the construction logic in quantizers.py. We see differences between "normal" (non-HPO) training and HPO training using
hyperparameters_optimization.py.In normal training whether the model is moved to CPU/GPU or not for the initial forward pass (to construct the full model). In HPO, this is taken care of by the objective function in
hyperparameters_optimization.py:Normal training:
Device | PQ Quantizer | HGQ Quantizer
CPU | Works | Works
GPU | Fails | Fails
HPO:
Device | PQ Quantizer | HGQ Quantizer
CPU | fails at second trial (reinitialization) | fails directly
GPU | fails directly | fails directly
I suspect this is caused by the following line, which does not place the parameter on the CUDA device at initialization. This makes PyTorch store the parameter as a CPU-only tensor, despite the move of the (entire) model to GPU afterwards so it is fine for normal, non-HPO training. See also #64 .
self.k = torch.nn.Parameter(torch.full(param_shape, float(k)), requires_grad=False)For normal training, doing the forward pass on CPU should not be a problem. For HPO, I have not discovered and therefore cannot propose any permanent workaround yet that fixes this issue but by hard-coding
device=devicein the quantizer initialization..