Skip to content

Latest commit

 

History

History
 
 

autograd

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
functions
functions
 
 
utils
utils
 
 
FunctionsManual.cpp
FunctionsManual.cpp
 
 
FunctionsManual.h
FunctionsManual.h
 
 
InferenceMode.h
InferenceMode.h
 
 
README.md
README.md
 
 
TraceTypeManual.cpp
TraceTypeManual.cpp
 
 
VariableTypeManual.cpp
VariableTypeManual.cpp
 
 
VariableTypeUtils.h
VariableTypeUtils.h
 
 
anomaly_mode.cpp
anomaly_mode.cpp
 
 
anomaly_mode.h
anomaly_mode.h
 
 
autograd.cpp
autograd.cpp
 
 
autograd.h
autograd.h
 
 
autograd_meta.cpp
autograd_meta.cpp
 
 
autograd_not_implemented_fallback.cpp
autograd_not_implemented_fallback.cpp
 
 
autograd_not_implemented_fallback.h
autograd_not_implemented_fallback.h
 
 
cpp_hook.cpp
cpp_hook.cpp
 
 
cpp_hook.h
cpp_hook.h
 
 
custom_function.cpp
custom_function.cpp
 
 
custom_function.h
custom_function.h
 
 
edge.h
edge.h
 
 
engine.cpp
engine.cpp
 
 
engine.h
engine.h
 
 
forward_grad.cpp
forward_grad.cpp
 
 
forward_grad.h
forward_grad.h
 
 
function.cpp
function.cpp
 
 
function.h
function.h
 
 
function_hook.h
function_hook.h
 
 
grad_mode.h
grad_mode.h
 
 
graph_task.h
graph_task.h
 
 
init.cpp
init.cpp
 
 
input_buffer.cpp
input_buffer.cpp
 
 
input_buffer.h
input_buffer.h
 
 
input_metadata.cpp
input_metadata.cpp
 
 
input_metadata.h
input_metadata.h
 
 
jit_decomp_interface.cpp
jit_decomp_interface.cpp
 
 
jit_decomp_interface.h
jit_decomp_interface.h
 
 
profiler.h
profiler.h
 
 
profiler_kineto.cpp
profiler_kineto.cpp
 
 
profiler_kineto.h
profiler_kineto.h
 
 
profiler_legacy.cpp
profiler_legacy.cpp
 
 
profiler_legacy.h
profiler_legacy.h
 
 
profiler_python.cpp
profiler_python.cpp
 
 
profiler_python.h
profiler_python.h
 
 
python_anomaly_mode.cpp
python_anomaly_mode.cpp
 
 
python_anomaly_mode.h
python_anomaly_mode.h
 
 
python_autograd.h
python_autograd.h
 
 
python_cpp_function.cpp
python_cpp_function.cpp
 
 
python_cpp_function.h
python_cpp_function.h
 
 
python_engine.cpp
python_engine.cpp
 
 
python_engine.h
python_engine.h
 
 
python_enum_tag.h
python_enum_tag.h
 
 
python_fft_functions.h
python_fft_functions.h
 
 
python_function.cpp
python_function.cpp
 
 
python_function.h
python_function.h
 
 
python_hook.cpp
python_hook.cpp
 
 
python_hook.h
python_hook.h
 
 
python_legacy_variable.cpp
python_legacy_variable.cpp
 
 
python_legacy_variable.h
python_legacy_variable.h
 
 
python_linalg_functions.h
python_linalg_functions.h
 
 
python_nested_functions.h
python_nested_functions.h
 
 
python_nested_functions_manual.cpp
python_nested_functions_manual.cpp
 
 
python_nn_functions.h
python_nn_functions.h
 
 
python_saved_variable_hooks.cpp
python_saved_variable_hooks.cpp
 
 
python_saved_variable_hooks.h
python_saved_variable_hooks.h
 
 
python_sparse_functions.h
python_sparse_functions.h
 
 
python_special_functions.h
python_special_functions.h
 
 
python_torch_functions.h
python_torch_functions.h
 
 
python_torch_functions_manual.cpp
python_torch_functions_manual.cpp
 
 
python_variable.cpp
python_variable.cpp
 
 
python_variable.h
python_variable.h
 
 
python_variable_indexing.cpp
python_variable_indexing.cpp
 
 
python_variable_indexing.h
python_variable_indexing.h
 
 
record_function_ops.cpp
record_function_ops.cpp
 
 
record_function_ops.h
record_function_ops.h
 
 
saved_variable.cpp
saved_variable.cpp
 
 
saved_variable.h
saved_variable.h
 
 
saved_variable_hooks.h
saved_variable_hooks.h
 
 
symbolic.h
symbolic.h
 
 
variable.cpp
variable.cpp
 
 
variable.h
variable.h
 
 
variable_info.cpp
variable_info.cpp
 
 
variable_info.h
variable_info.h
 
 

README.md

Autograd

Autograd is a hotspot for PyTorch performance, so most of the heavy lifting is implemented in C++. This implies that we have to do some shuffling between Python and C++; and in general, we want data to be in a form that is convenient to manipulate from C++.

Our general model is that for any key data type that autograd manipulates, there are two implementations: a C++ type and a Python object type. For example, consider variables in autograd: we have both Variable in variable.h (the C++ type) and THPVariable in python_variable.h (the Python type.) (By the way, THP stands for TorcH Python, not to be confused with THPP, TorcH C++). Variable contains the payload of a variable, while THPVariable just contains a shared_ptr reference to Variable, as well as references to other Python objects which the Python runtime needs to know about. A lot of data accessor implementations in python_variable.cpp simply reach through to the underlying Variable and return the appropriate value.

The most complicated application of this principle is Function, which also supports users implementing custom behavior in Python. We have the following classes:

  • Node in function.h, the C++ type.
  • THPFunction in python_function.h, the Python object type. In python_function.cpp, you can see the boilerplate that tells the Python interpreter about this object.
  • PyNode in python_function.h, a subclass of Node which forwards apply to a Python THPFunction. (NOT a Python object, despite its name!)

Outside of PyNode, the C++ objects largely avoid referencing Python objects (there are a few exceptions, like pyobj in Variable, and PyNode, whose whole point is to let C++ call into Python). And pyobj in Node to ensure uniqueness of the associated python wrapper (if it exists).