pytorch - CybAI news

02 Oct

Transformer in PyTorch

stp2y0 CommentsNewscoding, community, development, engineering, inclusive, layer, model, pytorch, software, transformer

Transformer() can get the 2D or 3D tensor of the one or more elements computed by Transformer from the 2D or 3D tensor of one or more elements as shown below: import torch from torch import nn tensor1 = torch.tensor([[8., -3., 0., 1.]]) tensor2 = torch.tensor([[5., 9., -4., 8.], [-2., 7., 3., 6.]]) tensor1.requires_grad tensor2.requires_grad # False torch.manual_seed(42) tran1 = nn.Transformer(d_model=4, nhead=2) tensor3 = tran1(src=tensor1, tgt=tensor2) tensor3 # tensor([[1.5608, 0.1450, -0.6434, -1.0624], # [0.8815, 1.0994, -1.1523, -0.8286]], # grad_fn=<NativeLayerNormBackward0>) tensor3.requires_grad # True tran1 # Transformer( # (encoder): TransformerEncoder( # (layers): ModuleList( # (0-5): 6 x TransformerEncoderLayer( # (self_attn): MultiheadAttention( #…

18 Aug

HuberLoss() in PyTorch

stp2y0 CommentsNewscoding, community, development, engineering, function, huberloss, inclusive, lossfunction, pytorch, software

Buy Me a Coffee☕ *Memos: HuberLoss() can get the 0D or more D tensor of the zero or more values(float) computed by Huber Loss from the 0D or more D tensor of zero or more elements as shown below: *Memos: The 1st argument for initialization is reduction(Optional-Default:'mean'-Type:str). *'none', 'mean' or 'sum' can be selected. The 2nd argument for initialization is delta(Optional-Default:1.0-Type:float). *It must be 0<delta. The 1st argument is input(Required-Type:tensor of float). The 2nd argument is target(Required-Type:tensor of float). input and target should be the same size otherwise there is a warning. The empty 1D or more D input and…

16 Aug

The activation functions in PyTorch (5)

stp2y0 CommentsNewsactivationfunction, coding, community, deeplearning, development, engineering, inclusive, machinelearning, pytorch, software

Buy Me a Coffee☕ *Memos: My post explains Step function, Identity and ReLU. My post explains Leaky ReLU, PReLU and FReLU. My post explains ELU, SELU and CELU. My post explains GELU, Mish, SiLU and Softplus. My post explains Vanishing Gradient Problem, Exploding Gradient Problem and Dying ReLU Problem. (1) Tanh: can convert an input value(x) to the output value between -1 and 1. *0 and 1 are exclusive. 's formula is y = (ex - e-x) / (ex + e-x). is also called Hyperbolic Tangent Function. is Tanh() in PyTorch. is used in: RNN(Recurrent Neural Network). *RNN in PyTorch.…

02 Aug

The loss functions in PyTorch

stp2y0 CommentsNewscoding, community, development, engineering, function, inclusive, lossfunction, neuralnetwork, pytorch, software

Buy Me a Coffee☕ *My post explains optimizers in PyTorch. A loss function is the function which can get the mean(average) of the sum of the losses(differences) between a model's predictions and true values(train or test data) to optimize a model during training or to evaluate how good a model is during testing. *Loss function is also called Cost Function or Error Function. There are popular loss functions as shown below: (1) L1 Loss: can compute the mean(average) of the sum of the absolute losses(differences) between a model's predictions and true values(train and test data). 's formula: is used for…

25 Jun

Set dtype with dtype argument functions and get it in PyTorch

stp2y0 CommentsNewsargument, coding, community, development, dtype, engineering, function, inclusive, pytorch, software

You can set dtype with the functions which have dtype arguments and get it with dtype and type() as shown below: *Memos: tensor(). *My post explains tensor(): import torch my_tensor = torch.tensor([0, 1, 2]) my_tensor = torch.tensor([0, 1, 2], dtype=torch.int64) my_tensor = torch.tensor([0, 1, 2], dtype=int) my_tensor, my_tensor.dtype, my_tensor.type() # (tensor([0, 1, 2]), torch.int64, 'torch.LongTensor') my_tensor = torch.tensor([0., 1., 2.], dtype=torch.float64) my_tensor = torch.tensor([0., 1., 2.], dtype=float) my_tensor, my_tensor.dtype, my_tensor.type() # (tensor([0., 1., 2.], dtype=torch.float64), # torch.float64, # 'torch.DoubleTensor') my_tensor = torch.tensor([0.+7.j, 1.+4.j, 2.+5.j], dtype=torch.complex32) my_tensor, my_tensor.dtype, my_tensor.type() # (tensor([0.+7.j, 1.+4.j, 2.+5.j], dtype=torch.complex32), # torch.complex32, # 'torch.ComplexHalfTensor') my_tensor = torch.tensor([True,…

17 Jun

The loss functions for Neural Network in PyTorch

stp2y0 CommentsNewscoding, community, development, engineering, function, inclusive, lossfunction, neuralnetwork, pytorch, software

A loss function is the function which can get the difference(gap) between a model's predictions and true values to evaluate how good a model is. *Loss function is also called Cost Function or Error Function. There are popular loss function as shown below: (1) L1 Loss: can compute the average of the sum of the absolute differences between a model's predictions and true values. 's formula is as shown below: is also called Mean Absolute Error(MAE). is L1Loss() in PyTorch. (2) L2 Loss: can compute the average of the sum of the squared differences between a model's predictions and true…