+
&j^ v ^ RI t ^ RI Ht ^ RIHtHt ^ RIHt ^RIH t RR.t
! R R] 4 t ! R R] 4 t R# )
NTensor)
functionalinit) Parameter)Module Embedding EmbeddingBagc a a ] tR t^t oRt. ROtRV3R lV 3R llltV3R lR ltV3R lR ltV3R lR lt V3R
lR lt
] RR l4 t V3R
lt
RtVtV ;t# )r as A simple lookup table that stores embeddings of a fixed dictionary and size.
This module is often used to store word embeddings and retrieve them using indices.
The input to the module is a list of indices, and the output is the corresponding
word embeddings.
Args:
num_embeddings (int): size of the dictionary of embeddings
embedding_dim (int): the size of each embedding vector
padding_idx (int, optional): If specified, the entries at :attr:`padding_idx` do not contribute to the gradient;
therefore, the embedding vector at :attr:`padding_idx` is not updated during training,
i.e. it remains as a fixed "pad". For a newly constructed Embedding,
the embedding vector at :attr:`padding_idx` will default to all zeros,
but can be updated to another value to be used as the padding vector.
max_norm (float, optional): If given, each embedding vector with norm larger than :attr:`max_norm`
is renormalized to have norm :attr:`max_norm`.
norm_type (float, optional): The p of the p-norm to compute for the :attr:`max_norm` option. Default ``2``.
scale_grad_by_freq (bool, optional): If given, this will scale gradients by the inverse of frequency of
the words in the mini-batch. Default ``False``.
sparse (bool, optional): If ``True``, gradient w.r.t. :attr:`weight` matrix will be a sparse tensor.
See Notes for more details regarding sparse gradients.
Attributes:
weight (Tensor): the learnable weights of the module of shape (num_embeddings, embedding_dim)
initialized from :math:`\mathcal{N}(0, 1)`
Shape:
- Input: :math:`(*)`, IntTensor or LongTensor of arbitrary shape containing the indices to extract
- Output: :math:`(*, H)`, where `*` is the input shape and :math:`H=\text{embedding\_dim}`
.. note::
Keep in mind that only a limited number of optimizers support
sparse gradients: currently it's :class:`optim.SGD` (`CUDA` and `CPU`),
:class:`optim.SparseAdam` (`CUDA` and `CPU`) and :class:`optim.Adagrad` (`CPU`)
.. note::
When :attr:`max_norm` is not ``None``, :class:`Embedding`'s forward method will modify the
:attr:`weight` tensor in-place. Since tensors needed for gradient computations cannot be
modified in-place, performing a differentiable operation on ``Embedding.weight`` before
calling :class:`Embedding`'s forward method requires cloning ``Embedding.weight`` when
:attr:`max_norm` is not ``None``. For example::
n, d, m = 3, 5, 7
embedding = nn.Embedding(n, d, max_norm=1.0)
W = torch.randn((m, d), requires_grad=True)
idx = torch.tensor([1, 2])
a = (
embedding.weight.clone() @ W.t()
) # weight must be cloned for this to be differentiable
b = embedding(idx) @ W.t() # modifies weight in-place
out = a.unsqueeze(0) + b.unsqueeze(1)
loss = out.sigmoid().prod()
loss.backward()
Examples::
>>> # an Embedding module containing 10 tensors of size 3
>>> embedding = nn.Embedding(10, 3)
>>> # a batch of 2 samples of 4 indices each
>>> input = torch.LongTensor([[1, 2, 4, 5], [4, 3, 2, 9]])
>>> # xdoctest: +IGNORE_WANT("non-deterministic")
>>> embedding(input)
tensor([[[-0.0251, -1.6902, 0.7172],
[-0.6431, 0.0748, 0.6969],
[ 1.4970, 1.3448, -0.9685],
[-0.3677, -2.7265, -0.1685]],
[[ 1.4970, 1.3448, -0.9685],
[ 0.4362, -0.4004, 0.9400],
[-0.6431, 0.0748, 0.6969],
[ 0.9124, -2.3616, 1.1151]]])
>>> # example with padding_idx
>>> embedding = nn.Embedding(10, 3, padding_idx=0)
>>> input = torch.LongTensor([[0, 2, 0, 5]])
>>> embedding(input)
tensor([[[ 0.0000, 0.0000, 0.0000],
[ 0.1535, -2.0309, 0.9315],
[ 0.0000, 0.0000, 0.0000],
[-0.1655, 0.9897, 0.0635]]])
>>> # example of changing `pad` vector
>>> padding_idx = 0
>>> embedding = nn.Embedding(3, 3, padding_idx=padding_idx)
>>> embedding.weight
Parameter containing:
tensor([[ 0.0000, 0.0000, 0.0000],
[-0.7895, -0.7089, -0.0364],
[ 0.6778, 0.5803, 0.2678]], requires_grad=True)
>>> with torch.no_grad():
... embedding.weight[padding_idx] = torch.ones(3)
>>> embedding.weight
Parameter containing:
tensor([[ 1.0000, 1.0000, 1.0000],
[-0.7895, -0.7089, -0.0364],
[ 0.6778, 0.5803, 0.2678]], requires_grad=True)
c ~ < V ^8 d QhRS[ RS[ RS[ R, RS[R, RS[RS[RS[R S[R, R
S[R R/
# ) num_embeddings
embedding_dim padding_idxNmax_norm norm_typescale_grad_by_freqsparse_weight_freezereturnintfloatboolr )format
__classdict__s "o/Users/jameslopez/projects/CWCArchive/cwc-podcast/.venv/lib/python3.14/site-packages/torch/nn/modules/sparse.py __annotate__Embedding.__annotate__ s{ + + + + 4Z +
$, +
+ !+ + $ + +
+ c : < R V
RV /p \ S
V ` 4 Wn W n VeZ V^ 8 d W0P 8 d \ R4 hM6V^ 8 d0 W0P ) 8 d \ R4 hV P V, pW0n W@n WPn W`n Vf? \ \ P ! W33/ V B V ' * R7 V n V P 4 M<\ VP 4 W.8w d \ R4 h\ W' * R7 V n Wpn R# )devicedtypeNz)Padding_idx must be within num_embeddings)
requires_grad?Shape of weight does not match num_embeddings and embedding_dim)super__init__r r AssertionErrorr r r r r torchemptyweightreset_parameterslistshaper )selfr r r r r r r r r r# r$ factory_kwargs __class__s &&&&&&&&&&&& r r( Embedding.__init__ s
#FGU;
, * "Q "5"55()TUU 6 q "5"5!55()TUU " 1 1K ? &
""4 ?# ^ ;N~N")kDK
! ! #GMM"~&EE$U $G;GDK r! c < V ^8 d QhRR/# r
r N )r r s "r r r + +$ +r! c f \ P ! V P 4 V P 4 R # Nr normal_r, _fill_padding_idx_with_zeror0 s &r r- Embedding.reset_parameters T[[! ((*r! c < V ^8 d QhRR/# r5 r6 )r r s "r r r 7 7T 7r! c V P eU \ P ! 4 ;_uu_ 4 V P V P , P ^ 4 R R R 4 R # R # + ' g i R # ; ir9 r r* no_gradr, fill_r= s &r r< %Embedding._fill_padding_idx_with_zero J ' D , ,-33A6 ! ( -A%% A6 c & < V ^8 d QhRS[ RS[ /# )r
inputr r )r r s "r r r s
V
r! c \ P ! VV P V P V P V P
V P V P 4 # r9 )F embeddingr, r r r r r )r0 rJ s &&r forwardEmbedding.forward sD {{ KK MM NN # # KK
r! c < V ^8 d QhRS[ /# r
r str)r r s "r r r s ) )C )r! c 2 R pV P e
VR,
pV P e
VR,
pV P ^8w d
VR,
pV P RJd
VR,
pV P RJd
VR,
pVP
! R/ V P B # )!{num_embeddings}, {embedding_dim}, padding_idx={padding_idx}, max_norm={max_norm}, norm_type={norm_type}F), scale_grad_by_freq={scale_grad_by_freq}z
, sparse=Truer6 )r r r r r r __dict__)r0 ss & r
extra_reprEmbedding.extra_repr s / '
. .A == $
( (A >>Q
* *A " "% /
< >> # FloatTensor containing pretrained weights
>>> weight = torch.FloatTensor([[1, 2.3, 3], [4, 5.1, 6.3]])
>>> embedding = nn.Embedding.from_pretrained(weight)
>>> # Get embeddings for index 1
>>> input = torch.LongTensor([1])
>>> # xdoctest: +IGNORE_WANT("non-deterministic")
>>> embedding(input)
tensor([[ 4.0000, 5.1000, 6.3000]])
4Embeddings parameter is expected to be 2-dimensional) r r r r r r r r r )dimr) r/ ) cls
embeddingsfreezer r r r r rowscolsrM s &&&&&&&& r from_pretrainedEmbedding.from_pretrained sZ J >> q !WX X %%
# 1
r! c < V ^8 d Qh/ S[ ;R& S[ ;R& S[ R, ;R& S[R, ;R& S[;R& S[;R& S[;R& S[;R & S[;R
& # ) r
r r Nr r r r r, rc r r )r r s "r r r s \ ] ^ _ ` ta b dlc d e f g h
Ni j
Lk l
Lm r! )r r r r r r r r, )r r r r r r r ) NN @FFNFNN)TNNri FF__name__
__module__ __qualname____firstlineno____doc__
__constants__r( r- r< rN r\ classmethodrf __annotate_func____static_attributes____classdictcell__
__classcell__r2 r s @@r r r sb aFM(+ + Z+ +7 7
) ) 2 2Q r! c a a ] tR tRt oRt. ROtRV3R lV 3R llltV3R lR ltV3R lR ltRV3R lR
llt V3R lR lt
] RV3R
lR ll4 t V3R lt
RtVtV ;t# )r
i a Compute sums or means of 'bags' of embeddings, without instantiating the intermediate embeddings.
For bags of constant length, no :attr:`per_sample_weights`, no indices equal to :attr:`padding_idx`,
and with 2D inputs, this class
* with ``mode="sum"`` is equivalent to :class:`~torch.nn.Embedding` followed by ``torch.sum(dim=1)``,
* with ``mode="mean"`` is equivalent to :class:`~torch.nn.Embedding` followed by ``torch.mean(dim=1)``,
* with ``mode="max"`` is equivalent to :class:`~torch.nn.Embedding` followed by ``torch.max(dim=1)``.
However, :class:`~torch.nn.EmbeddingBag` is much more time and memory efficient than using a chain of these
operations.
EmbeddingBag also supports per-sample weights as an argument to the forward
pass. This scales the output of the Embedding before performing a weighted
reduction as specified by ``mode``. If :attr:`per_sample_weights` is passed, the
only supported ``mode`` is ``"sum"``, which computes a weighted sum according to
:attr:`per_sample_weights`.
Args:
num_embeddings (int): size of the dictionary of embeddings
embedding_dim (int): the size of each embedding vector
max_norm (float, optional): If given, each embedding vector with norm larger than :attr:`max_norm`
is renormalized to have norm :attr:`max_norm`.
norm_type (float, optional): The p of the p-norm to compute for the :attr:`max_norm` option. Default ``2``.
scale_grad_by_freq (bool, optional): if given, this will scale gradients by the inverse of frequency of
the words in the mini-batch. Default ``False``.
Note: this option is not supported when ``mode="max"``.
mode (str, optional): ``"sum"``, ``"mean"`` or ``"max"``. Specifies the way to reduce the bag.
``"sum"`` computes the weighted sum, taking :attr:`per_sample_weights`
into consideration. ``"mean"`` computes the average of the values
in the bag, ``"max"`` computes the max value over each bag.
Default: ``"mean"``
sparse (bool, optional): if ``True``, gradient w.r.t. :attr:`weight` matrix will be a sparse tensor. See
Notes for more details regarding sparse gradients. Note: this option is not
supported when ``mode="max"``.
include_last_offset (bool, optional): if ``True``, the size of offsets is equal to the number of bags + 1.
The last element is the size of the input, or the ending index position
of the last bag (sequence). This matches the CSR format. Ignored when
input is 2D. Default ``False``.
padding_idx (int, optional): If specified, the entries at :attr:`padding_idx` do not contribute to the
gradient; therefore, the embedding vector at :attr:`padding_idx` is not updated
during training, i.e. it remains as a fixed "pad". For a newly constructed
EmbeddingBag, the embedding vector at :attr:`padding_idx` will default to all
zeros, but can be updated to another value to be used as the padding vector.
Note that the embedding vector at :attr:`padding_idx` is excluded from the
reduction.
Attributes:
weight (Tensor): the learnable weights of the module of shape `(num_embeddings, embedding_dim)`
initialized from :math:`\mathcal{N}(0, 1)`.
Examples::
>>> # an EmbeddingBag module containing 10 tensors of size 3
>>> embedding_sum = nn.EmbeddingBag(10, 3, mode='sum')
>>> # a batch of 2 samples of 4 indices each
>>> input = torch.tensor([1, 2, 4, 5, 4, 3, 2, 9], dtype=torch.long)
>>> offsets = torch.tensor([0, 4], dtype=torch.long)
>>> # xdoctest: +IGNORE_WANT("non-deterministic")
>>> embedding_sum(input, offsets)
tensor([[-0.8861, -5.4350, -0.0523],
[ 1.1306, -2.5798, -1.0044]])
>>> # Example with padding_idx
>>> embedding_sum = nn.EmbeddingBag(10, 3, mode='sum', padding_idx=2)
>>> input = torch.tensor([2, 2, 2, 2, 4, 3, 2, 9], dtype=torch.long)
>>> offsets = torch.tensor([0, 4], dtype=torch.long)
>>> embedding_sum(input, offsets)
tensor([[ 0.0000, 0.0000, 0.0000],
[-0.7082, 3.2145, -2.6251]])
>>> # An EmbeddingBag can be loaded from an Embedding like so
>>> embedding = nn.Embedding(10, 3, padding_idx=2)
>>> embedding_sum = nn.EmbeddingBag.from_pretrained(
embedding.weight,
padding_idx=embedding.padding_idx,
mode='sum')
c < V ^8 d QhRS[ RS[ RS[R, RS[RS[RS[RS[R S[R, R
S[R S[ R, R R/ # )
r
r r r Nr r moder r include_last_offsetr r )r r r rS r )r r s "r r EmbeddingBag.__annotate__r s ,7 ,7,7 ,7 $, ,7
,7 !
,7 ,7 ,7 $ ,7 ",7 4Z,7
,7r! c
8 < R V RV /p
\ SV ` 4 Wn W n W0n W@n WPn V
eZ V
^ 8 d WP 8 d \ R4 hM6V
^ 8 d0 WP ) 8 d \ R4 hV P V
, p
Wn Vf8 \ \ P ! W33/ V
B 4 V n V P 4 M6\ VP 4 W.8w d \ R4 h\ V4 V n W`n Wpn Wn R# )r# r$ Nz)padding_idx must be within num_embeddingsr& )r' r( r r r r r r) r r r* r+ r, r- r. r/ ry r rz )r0 r r r r r ry r r rz r r# r$ r1 r2 s &&&&&&&&&&&&& r r( EmbeddingBag.__init__r s #FGU;
, *
""4 "Q "5"55()TUU 6 q "5"5!55()TUU " 1 1K ? & ?# ^ ;N~NDK
! ! #GMM"~&EE$U $G,DK #6 r! c < V ^8 d QhRR/# r5 r6 )r r s "r r r{ r7 r! c f \ P ! V P 4 V P 4 R # r9 r: r= s &r r- EmbeddingBag.reset_parameters r? r! c < V ^8 d QhRR/# r5 r6 )r r s "r r r{ rA r! c V P eU \ P ! 4 ;_uu_ 4 V P V P , P ^ 4 R R R 4 R # R # + ' g i R # ; ir9 rC r= s &r r< (EmbeddingBag._fill_padding_idx_with_zero rG rH c N < V ^8 d QhRS[ RS[ R, RS[ R, RS[ /# )r
rJ offsetsNper_sample_weightsr r )r r s "r r r{ s; 0
0
0
$ 0
#TM 0
0
r! c
\ P ! VV P VV P V P V P
V P V P VV P V P 4 # )a+ Forward pass of EmbeddingBag.
Args:
input (Tensor): Tensor containing bags of indices into the embedding matrix.
offsets (Tensor, optional): Only used when :attr:`input` is 1D. :attr:`offsets` determines
the starting index position of each bag (sequence) in :attr:`input`.
per_sample_weights (Tensor, optional): a tensor of float / double weights, or None
to indicate all weights should be taken to be ``1``. If specified, :attr:`per_sample_weights`
must have exactly the same shape as input and is treated as having the same
:attr:`offsets`, if those are not ``None``. Only supported for ``mode='sum'``.
Returns:
Tensor output shape of `(B, embedding_dim)`.
.. note::
A few notes about ``input`` and ``offsets``:
- :attr:`input` and :attr:`offsets` have to be of the same type, either int or long
- If :attr:`input` is 2D of shape `(B, N)`, it will be treated as ``B`` bags (sequences)
each of fixed length ``N``, and this will return ``B`` values aggregated in a way
depending on the :attr:`mode`. :attr:`offsets` is ignored and required to be ``None`` in this case.
- If :attr:`input` is 1D of shape `(N)`, it will be treated as a concatenation of
multiple bags (sequences). :attr:`offsets` is required to be a 1D tensor containing the
starting index positions of each bag in :attr:`input`. Therefore, for :attr:`offsets` of shape `(B)`,
:attr:`input` will be viewed as having ``B`` bags. Empty bags (i.e., having 0-length) will have
returned vectors filled by zeros.
)
rL
embedding_bagr, r r r ry r rz r )r0 rJ r r s &&&&r rN EmbeddingBag.forward s] H KK MM NN # # II KK $ $
r! c < V ^8 d QhRS[ /# rQ rR )r r s "r r r{ s J JC Jr! c v R pV P e
VR,
pV P ^8w d
VR,
pV P RJd
VR,
pVR,
pV P e
VR,
pVP ! R/ V P
P
4 UUu/ uF w r#V\ V4 bK uppB # u uppi )rU rW rX FrY z
, mode={mode}rV r6 )r r r r r rZ itemsrepr)r0 r[ kvs & r r\ EmbeddingBag.extra_repr s / == $
( (A >>Q
* *A " "% /
< >> # FloatTensor containing pretrained weights
>>> weight = torch.FloatTensor([[1, 2.3, 3], [4, 5.1, 6.3]])
>>> embeddingbag = nn.EmbeddingBag.from_pretrained(weight)
>>> # Get embeddings for index 1
>>> input = torch.LongTensor([[1, 0]])
>>> # xdoctest: +IGNORE_WANT("non-deterministic")
>>> embeddingbag(input)
tensor([[ 2.5000, 3.7000, 4.6500]])
r_ )
r r r r r r ry r rz r )r` r) r/ r, r% )
ra rb rc r r r ry r rz r rd re embeddingbags
&&&&&&&&&& r rf EmbeddingBag.from_pretrained sm N >> q !WX X %%
1 3#
17J )r! c < V ^8 d Qh/ S[ ;R& S[ ;R& S[R, ;R& S[;R& S[;R& S[;R& S[;R& S[;R & S[;R
& S[ R, ;R & # ) r
r r Nr r r r, ry r rz r )r r r r rS )r r s "r r r{ s x y z { | dl} ~ @ A B
NC D IE F
LG H I J tK r! )
r rz r ry r r r r r r, ) r r r r r ry r rz r )
Nri FmeanFNFNNN)NN)TNri Fr FFNrj rv s @@r r
r
sj M^
M.,7 ,7\+ +7 7
0
0
d J J 6 6 6 } r! )
r* r torch.nnr rL r torch.nn.parameterr moduler __all__r r
r6 r! r r s?
* (
'z zzT 6 T r!