Welcome to HyperPose’s Documentation!

Installation

C++ Prediction Library Installation

Docker Environment Installation

To ease the installation, you can use HyperPose library in our docker image where the environment is pre-installed.

Official Docker Image

NVidia docker support is required to execute our docker image.

The official image is on DockerHub.

# Pull the latest image.
docker pull tensorlayer/hyperpose

# Dive into the image. (Connect local camera and imshow window)
xhost +; docker run --gpus all -it -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --device=/dev/video0:/dev/video0 --entrypoint /bin/bash tensorlayer/hyperpose
# For users that cannot access a camera or X11 server. You may also use:
# docker run --rm --gpus all -it --entrypoint /bin/bash tensorlayer/hyperpose

Note that the entry point is the hyperpose-cli binary in the build directory.

Build docker from source
# Enter the repository folder.
USER_DEF_NAME=my_hyperpose
docker build -t $(USER_DEF_NAME) .
docker run --gpus all $(USER_DEF_NAME)

Build From Source

Prerequisites

  • C++ 17 Compiler. (g++7, clang++4.0, MSVC19.0 or newer)

  • CMake 3.5+

  • Third-Party

Other versions of the packages may also work but not tested.

About TensorRT installation

  • For Linux users, you highly recommended to install it in a system-wide setting. You can install TensorRT7 via the debian distributions or NVIDIA network repo(CUDA and CuDNN dependency will be automatically installed).

  • Different TensorRT version requires specific CUDA and CuDNN version. For specific CUDA and CuDNN requirements of TensorRT7, please refer to this.

  • Also, for Ubuntu 18.04 users, this 3rd party blog may help you.

Build On Ubuntu 18.04
# >>> Install OpenCV3+ and other dependencies. 
sudo apt -y install cmake libopencv-dev libgflags-dev
# !Note that the APT version OpenCV3.2 on Ubuntu18.04 has some trouble on Cameras Newer version is suggested.
# You are highly recommended to install OpenCV 4+ from scratch also for better performance.

# >>> Install dependencies to run the scripts in `${REPO}/scripts`
sudo apt install python3-dev python3-pip 

# >>> Install CUDA/CuDNN/TensorRT: https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-debian

# >>> Build HyperPose
git clone https://github.com/tensorlayer/hyperpose.git
cd hyperpose
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release && cmake --build .

Build with User Codes

You can directly write codes and execute it under the hyperpose repository.

  • Step 1: Write your own codes in hyperpose/examples/user_codes with suffix .cpp.

  • Step 2:

mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_USER_CODES=ON   # BUILD_USER_CODES is by default on
cmake --build .

  • Step 3: Execute your codes!

Just go to Quick Start to test your installation.

Python Training Library Installation

Prerequisites

Configure environment

Hyperpose training library can be directly used by putting Hyperpose in the directory and import. But it has to install the prerequist environment to make it available.

The following instructions have been tested on the environments below:

  • Ubuntu 18.04, Tesla V100-DGXStation, Nvidia Driver Version 440.33.01, CUDA Verison=10.2

  • Ubuntu 18.04, Tesla V100-DGXStation, Nvidia Driver Version 410.79, CUDA Verison=10.0

  • Ubuntu 18.04, TITAN RTX, Nvidia Driver Version 430.64, CUDA Version=10.1

  • Ubuntu 18.04, TITAN Xp, Nvidia Driver Version 430.26, CUDA Version=10.2

# >>> create virtual environment (choose yes)
conda create -n hyperpose python=3.7
# >>> activate the virtual environment, start installation
conda activate hyperpose
# >>> install cuda and cudnn using conda
conda install cudatoolkit=10.0.130
conda install cudnn=7.6.0
# >>> install tensorflow of version 2.0.0
pip install tensorflow-gpu==2.0.0
# >>> install the newest version tensorlayer from github
pip install tensorlayer==2.2.3
# >>> install other requirements (numpy<=17.0.0 because it has conflicts with pycocotools)
pip install opencv-python
pip install numpy==1.16.4
pip install pycocotools
pip install matplotlib
# >>> now the configuration is done, check whether the GPU is avaliable.
python
>>> import tensorflow as tf
>>> import tensorlayer as tl
>>> tf.test.is_gpu_available()
# >>> if the output is true, congratulation! you can import and run hyperpose now
>>> from hyperpose import Config,Model,Dataset

Extra configuration for exporting model

For training, the above configuration is enough, but to export model into onnx format for inference,one should install the following two extra library:

  • tf2onnx (necessary ,used to convert .pb format model into .onnx format model) reference

pip install -U tf2onnx

  • graph_transforms (unnecesary,used to check the input and output node of the .pb file if one doesn’t know) build graph_transforms according to reference

Get Started

Quick Start of Prediction Library

Prerequisites

  • Make sure you have HyperPose installed. (if not, you can refer to here).

  • Make sure you have svn(subversion) and python3-pip installed. (will be used in command line scripts)

For Linux users, you may:

sudo apt -y install subversion python3 python3-pip

Install Test Data

# cd to the git repo.

sh scripts/download-test-data.sh

You download them manually to ${HyperPose}/data/media/ via LINK if the network is not working.

Install Test Models

# cd to the git repo. And download pre-trained models you want. 

sh scripts/download-openpose-thin-model.sh      # ~20  MB
sh scripts/download-tinyvgg-model.sh            # ~30  MB
sh scripts/download-openpose-res50-model.sh     # ~45  MB
sh scripts/download-openpose-coco-model.sh      # ~200 MB
sh scripts/download-ppn-res50-model.sh          # ~50  MB (PoseProposal Algorithm)

You can download them manually to ${HyperPose}/data/models/ via LINK if the network is not working.

Predict a sequence of images

Using a fast model
# cd to your build directory.

# Predict all images in `../data/media`
./hyperpose-cli --source ../data/media --model ../data/models/lopps-resnet50-V2-HW=368x432.onnx --w 368 --h 432
# The source flag can be ignored as the default value is `../data/media`.

The output images will be in the build folder.

If you met logging message like ERROR: Tensor image cannot be both input and output please just ignore it.

Table of flags for hyperpose-cli

Note that the entry point of our official docker image is also hyperpose-cli in the /hyperpose/build folder.

Also see: ./hyperpose-cli --help

Flag Meaning Default
model Path to your model. ../data/models/TinyVGG-V1-HW=256x384.uff
source Path to your source.
The source can be a folder path, a video path, an image path or the key word camera to open your camera.		 ../data/media/video.avi
post Post-processing methods. This key can be paf or ppn. paf
keep_ratio The DNN takes a fixed input size, where the images must resize to fit that input resolution. However, not hurt the original human scale, we may want to resize by padding. And this is flag enable you to do inference without break original human ratio. (Good for accuracy) true
w The input width of your model. Currently, the trained models we provide all have specific requirements for input resolution. 384(for your the tiny-vgg model)
h The input height of your model. 256(for your the tiny-vgg model)
max_batch_size Maximum batch size for inference engine to execute. 8
runtime Which runtime type to use. This can be operator or stream. If you want to open your camera or producing imshow window, please use operator. For better processing throughput on videos, please use stream. operator
imshow true Whether to open an imshow window.
saving_prefix The output media resource will be named after '$(saving_prefix)_$(ID).$(format) "output"
alpha The weight of key point visualization. (from 0 to 1) 0.5
logging Print the internal logging information or not. false
Using a precise model
./hyperpose-cli --model ../data/models/openpose-thin-V2-HW=368x432.onnx --w 432 --h 368 

./hyperpose-cli --model ../data/models/openpose-coco-V2-HW=368x656.onnx --w 656 --h 368
Use PoseProposal model
./hyperpose-cli --model ../data/models/ppn-resnet50-V2-HW=384x384.onnx --w 384 --h 384 --post=ppn
Convert models into TensorRT Engine Protobuf format

You may find that it takes one or two minutes before the real prediction starts. This is because TensorRT will try to profile the model to get a optimized runtime model.

To save the model conversion time, you can convert it in advance.

./example.gen_serialized_engine --model_file ../data/models/openpose-coco-V2-HW=368x656.onnx --input_width 656 --input_height 368 --max_batch_size 20
# You'll get ../data/models/openpose-coco-V2-HW=368x656.onnx.trt
# If you only want to do inference on single images(batch size = 1), please use `--max_batch_size 1` and this will improve the engine's performance.

# Use the converted model to do prediction
./hyperpose-cli --model ../data/models/openpose-coco-V2-HW=368x656.onnx.trt --w 656 --h 368

Currently, we run the models in TensorRT float32 mode. Other data type is not supported(welcome to contribute!).

Predict a video using Operator API

./hyperpose-cli --runtime=operator --source=../data/media/video.avi

The output video will be in the building folder.

Predict a video using Stream API(faster)

./hyperpose-cli --runtime=stream --source=../data/media/video.avi
# In stream API, the imshow functionality will be closed.

Play with camera

./hyperpose-cli --source=camera
# Note that camera mode is not compatible with Stream API. If you want to do inference on your camera in real time, the Operator API is designed for it.

Quick Start of Training Library

Prerequisites

  • Make sure you have configured ‘hyperpose’ virtual environment following the training installation guide,(if not, you can refer to training installation).

  • Make sure your GPU is available now(using tf.test.is_gpu_available() and it should return True)

  • Make sure the Hyperpose training Library is under the root directory of the project(where you write train.py and eval.py)

Train a model

The training procedure of Hyperpose is to set the model architecture, model backbone and dataset. User specify these configuration using the seting functions of Config module with predefined enum value. The code for training as simple as following would work.

# >>> import modules of hyperpose
from hyperpose import Config,Model,Dataset
# >>> set model name is necessary to distinguish models (neccesarry)
Config.set_model_name(args.model_name)
# >>> set model architecture(and model backbone when in need)
Config.set_model_type(Config.MODEL.LightweightOpenpose)
Config.set_model_backbone(Config.BACKBONE.Vggtiny)
# >>> set dataset to use
Config.set_dataset_type(Config.DATA.COCO)
# >>> set training type 
Config.set_train_type(Config.TRAIN.Single_train)
# >>> configuration is done, get config object to assemble the system
config=Config.get_config()
model=Model.get_model(config)
dataset=Dataset.get_dataset(config)
train=Model.get_train(config)
# >>> train!
train(model,dataset)

Then the integrated training pipeline will start. for each model, Hyperpose will save all the related files in the direatory: ./save_dir/model_name, where model_name is the name user set by using Config.set_model_name the directory and its contents are below:

  • directory to save model ./save_dir/model_name/model_dir

  • directory to save train result ./save_dir/model_name/train_vis_dir

  • directory to save evaluate result ./save_dir/model_name/eval_vis_dir

  • directory to save dataset visualize result ./save_dir/model_name/data_vis_dir

  • file path to save train log ./save_dir/model_name/log.txt

The above code section show the simplest way to use Hyperpose training library, to make full use of Hyperpose training library, you can refer to training tutorial

Eval a model

The evaluate procedure using Hyperpose is almost the same to the training procedure: the model will be loaded from the ./save_dir/model_name/model_dir/newest_model.npz

# >>> import modules of hyperpose
from hyperpose import Config,Model,Dataset
# >>> set model name to be eval
Config.set_model_name(args.model_name)
# >>> the model architecture and backbone setting should be the same with the training configuration of the model to be evaluated.
Config.set_model_type(Config.MODEL.LightweightOpenpose)
Config.set_model_backbone(Config.BACKBONE.Vggtiny)
# >>> set dataset to use
Config.set_dataset_type(Config.DATA.COCO)
# >>> configuration is done, get config object to assemble the system
config=Config.get_config()
model=Model.get_model(config)
dataset=Dataset.get_dataset(config)
eval=Model.get_eval(config)
# >>> eval
eval(model,dataset)

Then the integrated evaluation pipeline will start, the final evaluate metrics will be output at last. It should be noted that: 1.the model architecture, model backbone, dataset type should be the same with the configuration under which model was trained. 2.the evaluation metrics will follow the official evaluation metrics of dataset

The above code section show the simplest way to use Hyperpose training library to evaluate a model trained by Hyperpose, to make full use of Hyperpose training library, you can refer to training tutorial

Export a model

To export a model trained by Hyperpose, one should follow two step:

  • (1)convert the trained .npz model into .pb format this can be done either call the export_pb.py from Hyperpose repo

    python export_pb.py --model_type=your_model_type --model_name=your_model_name

    then the converted model will be put in the ./save_dir/model_name/forzen_model_name.pb one can also export himself by loading model and using get_concrete_function by himself, please refer the tutorial for details

  • (2)convert the frozen .pb format model by tensorflow-onnx Make sure you have installed the extra requirements for exporting models from training installation
    if you don’t know the input and output name of the pb model,you should use the function summarize_graph function of graph_transforms from tensorflow

    bazel-bin/tensorflow/tools/graph_transforms/summarize_graph --in_graph=your_frozen_model.pb

    then, after knowing the input and output nodes of your .pb model,use tf2onnx

    python -m tf2onnx.convert --graphdef your_frozen_model.pb --output output_model.onnx --inputs input0:0,input1:0... --outputs output0:0,output1:0,output2:0...

    args follow inputs and outputs are the names of input and output nodes in .pb graph repectly, for example, if the input node name is x and output node name is y1,y2, then the convert bash should be:

    python -m tf2onnx.convert --graphdef your_frozen_model.pb --output output_model.onnx --inputs x:0 --outputs y1:0,y2:0

congratulation! now you are able to use the onnx model for Hyperpose prediction library.

Tutorials

Tutorial for Prediction Library

The prediction library of hyperpose provides 2 APIs styles:

  • Operator API: Imperative style. (more user manipulation space)

  • Stream API: Declarative style. (faster and simpler)

This tutorial will show you how to use them in C++ step by step. For more detailed instructions, please refer to our C++ API documents.

End-2-end Prediction Using Stream API

In this section, we’ll try to process a video via Stream API.

Before all, please make sure you have the library successfully built(See installation). And we encourage you to build the tutorial examples under the folder hyperpose/examples/user_codes.

cd examples/user_codes
touch main.cpp

# Open your editor and do coding.

cd ../..
mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=RELEASE -DBUILD_USER_CODES=ON # BUILD_USER_CODES is by default on
cmake --build .

# Execute your codes.
Include the header
#include <hyperpose/hyperpose.hpp>

The file arrangement of include files:

  • hyperpose

    • hyperpose.hpp (include all necessary headers)

    • operator (headers about Operator API)

    • stream (headers about Stream API)

    • utility

Usually, you only need to include hyperpose/hyperpose.hpp.

Prepare Your Model

We support 3 types of model file:

  • Uff: Users need to specify the input/output nodes of the network compute graph.

  • ONNX: No input/output nodes information is required.

  • Cuda Engine Protobuf: When importing Uff / ONNX models, TensorRT will do profiling to build the “best” runtime engine. To save the building time, we can export the model to Cuda Engine Protobuf format and reload it in next execution.

using namespace hyperpose;

// To use a Uff model, users needs to specify the input/output nodes.
// Here, `image` is the input node name, and `outputs/conf` and `outputs/paf` are the output feature maps. (related to the PAF algorithm)
const dnn::uff uff_model{ "../data/models/hao28-600000-256x384.uff", "image", {"outputs/conf", "outputs/paf"} };
Create Input / Output Stream

We support std::vector<cv::Mat>, cv::Mat, cv::VideoCapture as the inputs of input stream.

We also support cv::VideoWriter, NameGenerator(a callable object which generate next name for the output image) as output streams.

// For best performance, HyperPose only allows models who have fixed input network resolution.
// What is "network resolution"? Say that the input of networks are NCHW format, the "HW" is the network resolution.
const cv::Size network_resolution{384, 256};

// * Input video.
auto capture = cv::VideoCapture("../data/media/video.avi");

// * Output video.
auto writer = cv::VideoWriter(
    "output.avi", 
    capture.get(cv::CAP_PROP_FOURCC), capture.get(cv::CAP_PROP_FPS), 
    network_resolution); // Here we use the network resolution as the output video resolution.
Create DNN Engine and Post-processing Parser
// * Create TensorRT engine.
dnn::tensorrt engine(uff_model, network_resolution);

// * post-processing: Using paf.
parser::paf parser{};
Create Stream Scheduler
// * Create stream
auto stream = make_stream(engine, parser);
Connect the Stream
// * Connect input stream.
stream.async() << capture;

// * Connect ouput stream and wait.
stream.sync() >> writer;

  • We provides 2 ways of stream connection.

    • .async()

      • Input Stream: The stream scheduler will push images asynchronously. (not blocked)

      • Output Stream: The stream scheduler will generate results asynchronously. (not blocked)

    • .sync()

      • Input Stream: Blocked. This may cause deadlock if you trying to push a big number of images in a synchronous way(the buffer queue is of fixed size).

      • Output Stream: Blocked until all outputs are generated.

We recommend you to set inputs via async() and generate results via sync().

Full example

Full examples are available here.

Prediction Using Operator API

Preparation
#include <hyperpose/hyperpose.hpp>

int main() {
    using namespace hyperpose;

    const cv::Size network_resolution{384, 256};
    const dnn::uff uff_model{ "../data/models/TinyVGG-V1-HW=256x384.uff", "image", {"outputs/conf", "outputs/paf"} };

    // * Input video.
    auto capture = cv::VideoCapture("../data/media/video.avi");

    // * Output video.
    auto writer = cv::VideoWriter(
        "output.avi", capture.get(cv::CAP_PROP_FOURCC), capture.get(cv::CAP_PROP_FPS), network_resolution);

    // * Create TensorRT engine.
    dnn::tensorrt engine(uff_model, network_resolution);

    // * post-processing: Using paf.
    parser::paf parser{};
    
    while (capture.isOpened()) {
        // ..... Applying pose estimation in one batch. 
    }
}
Apply One Batch of Frames
Accumulate One Batch
std::vector<cv::Mat> batch;

// The .max_batch_size() of dnn::tensorrt is set in the initializer. (by default -> 8)
// initializer(model_config, input_size, max_batch_size = 8, dtype = float, factor = 1./255, flip_rgb = true)
for (int i = 0; i < engine.max_batch_size(); ++i) {
    cv::Mat mat;
    capture >> mat;
    if (mat.empty()) // If the video ends, break.
        break;
    batch.push_back(mat);
}

// Now we got a batch of images. -> batch.
Get Feature/Activation Maps
// * TensorRT Inference.
std::vector<internal_t> feature_map_packets = engine.inference(batch);
// using internal_t = std::vector<feature_map_t>
// Here, feature_map_packets = batch_size * feature_map_count(conf and paf) * feature_map.
Get Human Topology

One image may contain many humans. So the return type is std::vector<human_t>.

// * Paf.
std::vector<std::vector<human_t>> pose_vectors; // image_count * humans
for (auto& packet : feature_map_packets)
    pose_vectors.push_back(parser.process(packet[0]/* conf */, packet[1] /* paf */));
Visualization
// * Visualization
for (size_t i = 0; i < batch.size(); ++i) {
    cv::resize(batch[i], batch[i], network_resolution);
    for (auto&& pose : pose_vectors[i])
        draw_human(batch[i], pose); // Visualization.
    writer << batch[i];
}
Full example

Full examples are available here.

Tutorial for Training Library

Up to now, Hyperpose provides:

  • 4 types of preset model architectures:

    • Openpose

    • LightweightOpenpose

    • Poseproposal

    • MobilenetThinOpenpose

  • 7 types of common model backbone for backbone replacement:

    • MobilenetV1, MobilenetV2

    • Vggtiny, Vgg16, Vgg19

    • Resnet18, Resnet50

  • 2 types of popular dataset

    • COCO

    • MPII

Integrated pipeline

Hyperpose extract similiar models into a model class. For now, there are two classes: Openpose classes and Poseproposal classes. all model architecture can be devided into one of them.For each model class, Hyperpose privide a integrated pipeline.

Integrated train pipeline

The usage of integrated training procedure of Hyperpose can be devided into two parts:setting configuration using APIs of Config module, and getting the configured system from the Model and dataset module.

  • setting parts mainly concern: model_name, model_type, model_backbone, dataset_type and train_type

    • set_model_name will determine what the path the model related file will be put to

    • set_model_type will adopt the chosen preset model architecture(use enum value of enum class Config.MODEL)

    • set_model_backbone will replace the backbone of chosen preset model architeture(use enum value of enum class Config.BACKBONE)

    • set_dataset_type will change the dataset in the training pipeline(use enum value of enum class Config.DATA)

    • set_train_type is to choose whether use single GPU for single training or multiple GPUs for parallel training(use enum value of enum class Config.TRAIN)
      the conbination of different model architectures and model backbones will lead to huge difference of countructed model’ computation complexity (for example,Openpose architecture with default Vgg19 backbone is 200MB, while MobilenetThinOpenpose with mobilenet-variant backbone is only 18MB), thus it should be carefully considered.for more detailed information, please refer the API documents.

The basic training pipeline configuration is below:

# >>> import modules of hyperpose
from hyperpose import Config,Model,Dataset
# >>> set model name is necessary to distinguish models (neccesarry)
Config.set_model_name(args.model_name)
# >>> set model architecture using Config.MODEL enum class (neccesarry)
Config.set_model_type(Config.MODEL.LightweightOpenpose)
# >>> set model backbone using Config.BACKBONE enum class (not neccessary, each model has its default backbone)
Config.set_model_backbone(Config.BACKBONE.Vggtiny)
# >>> set dataset to use (neccesarry)
Config.set_dataset_type(Config.DATA.COCO)
# >>> set training type (not neccesary, default is single training, can use parallel training)
Config.set_train_type(Config.TRAIN.Single_train)
# >>> congratulations!, the simplest configuration is done, it's time to assemble the model and training pipeline

to use parallel training, one should set train type at first, and then choose kungfu optimizor wrap function, replace the set_train_type function as below, Kungfu also have three option: Sync_sgd,Sync_avg,Pair_avg

Config.set_train_type(Config.TRAIN.Parallel_train)
Config.set_kungfu_option(Config.KUNGFU.Sync_sgd)

And when run your program, using the following command(assuming we have 4 GPUs)

CUDA_VISIBLE_DEVICES=0,1,2,3 kungfu-run -np 4 python train.py

  • getting parts mainly concern: pass the configuration to Model module and Dataset module to assemble the system

    • Config.get_config will return a config object which contains all the configuration and is the core of the getting functions

    • Model.get_model will return a configrued model object which can forward and calcaulate loss

    • Datset.get_dataset will return a configured dataset object which can generate tensorflow dataset object used for train and evaluate, it can also visualize the dataset annotation.

    • Model.get_train will return a training pipeline, which could start running as long as receive the model object and dataset object

The basic training pipeline assembling is below:

# >>> get config object at first (neccesarry)
config=Config.get_config()
# >>> get model object (neccesarry)
model=Model.get_model(config)
# >>> get dataset object (neccesarry)
dataset=Dataset.get_dataset(config)
# >>> get train pipeline (neccesarry)
train=Model.get_train(config)
# >>> train!
train(model,dataset)
Integrated evaluate pipeline

The configuration part and assembling part of evaluate pipeline is very similair to the train pipeline’s.The only difference is that:

  • In setting parts, one doesn’t need to set train type(and thus kungfu option)

  • In getting parts, one should use Model.get_eval to get evaluate pipeline(rather than train pipeline) Thus the evaluate code should be as follows:

# >>> import modules of hyperpose
from hyperpose import Config,Model,Dataset
# >>> set model name to be eval
Config.set_model_name(args.model_name)
# >>> the model architecture and backbone setting should be the same with the training configuration of the model to be evaluated.
Config.set_model_type(Config.MODEL.LightweightOpenpose)
Config.set_model_backbone(Config.BACKBONE.Vggtiny)
# >>> set dataset to use
Config.set_dataset_type(Config.DATA.COCO)
# >>> configuration is done, get config object to assemble the system
config=Config.get_config()
model=Model.get_model(config)
dataset=Dataset.get_dataset(config)
eval=Model.get_eval(config)
# >>> eval
eval(model,dataset)

the model will be loaded from the ./save_dir/model_name/model_dir/newest_model.npz and evaluated. It should be noted that:

  • 1.the model architecture, model backbone, dataset type should be the same with the configuration under which model was trained.

  • 2.the evaluation metrics will follow the official evaluation metrics of dataset

User-defined model architecture

Hyperpose leaves freedom for user to define thier own model architecture but use the provided integrated model pipeline at the same time, the following points should be considered:

  • 1.the model should be an object of a tensorlayer.models.Model class (or inherite from this class)

  • 2.the model should have foward and cal_loss functions that has exactly the same input and output format with the preset model architectures. one can refer Model.LightweightOpenpose class for reference.
    to do this, user still need to set model_type to determine the training pipeline, here the model_type should be the model that has the similair data processing pipeline with the user’s own model. Then he can use the set_model_arch function to pass his own model object

    Config.set_model_name(your_model_name)
    Config.set_model_type(similiar_ model_type)
    Config.set_model_arch(your_model_arch)

the other configuration procedures are the same with the integrated training pipeline.

User-defined dataset

Hyperpose allows user to use their own dataset to be integrated with the training and evaluating pipeline, as long as it has the following attribute and functions:

  • 1.get_train_dataset:
    return a tensorflow dataset object where each element should be a image path and a serialized dict(using _pickle library to serialize) which at least have the three key-value pairs:
    1.1 “kpt”-a list contains keyspoints for each labeled human, for example:[[kpt1,kpt2,…,kptn],[kpt1,kpt2,…,kptn]] is a list with two labeld humans, where each kpt is a [x,y] coordinate such as [234,526],etc
    1.2 “bbx”-a list contains boundingbox for each labeled human, for example:[bbx1,bbx2] is a list with two labeled humans, where each bbx is a [x,y,w,h] array such as [234,526,60,80], necessary for pose proposal network, could be set to None for others
    1.3 “mask”-a mask (in mscoco polynomial format) used to cover unlabeled people, could be set to None

  • 2.get_eval_dataset:
    return a tensorflow dataset object where each element should be a image path and its image id.

  • 3.get_input_kpt_cvter(optional):
    return a function which changes the kpt value in your dataset dict element,used to enable your dataset keypoint annotation being in line with your model keypoint setting, or combined with other dataset with different keypoint annotation.

  • 4.get_output_kpt_cvter(optional):
    return a function which changes the model predict result to a format that easy to evaluate, used to enable your datatset to be evaluate at a formal COCO standard (using MAP) or MPII standard (using MPCH).

User-defined dataset filter

Hyperpose also leave freedom for user to define thier own dataset filter to filter the dataset as they want using set_dataset_filter function. to use this, a user should know the follow information:

  • 1.Hyperpose organize the annotations of one image in one dataset in the similiar meta classes. for COCO dataset, it is COCOMeta; for MPII dataset, it is MPIIMeta. Meta classes will have some common information such as image_id, joint_list etc, they also have some dataset-specific imformation, such as mask, is_crowd, headbbx_list etc.

  • 2.the dataset_fiter will perform on the Meta objects of the corresponding dataset, if it returns True, the image and annotaions the Meta object related will be kept, otherwise it will be filtered out. Please refer the Dataset.xxxMeta classes for better use. please refer Dataset.COCOMeta,Dataset.MPIIMeta classes for better use.

    def my_dataset_filter(coco_meta):
        if(len(coco_meta.joint_list)<5 and (not coco_meta.is_crowd)):
            return True
        else:
            return False
    Config.set_dataset_filter(my_dataset_filter)

User-defined train pipeline

Hyperpose also provides three low level functions to help user consturct thier own pipeline. For each class of model, functions of preprocess, postprocess and visualize are provided.

  • get_preprocess receives model_type from Config.MODEL and return a preprocess function The preprocess function is able to convert the annotaion into targets used for training the model.

preprocess=Model.get_preprocess(Config.MODEL.LightweightOPenpose)
conf_map,paf_map=preprocess(annos,img_height,img_width,model_hout,model_wout,Config.DATA.COCO,data_format="channels_first")
pd_conf_map,pd_paf_map=my_model.forward(input_image[np.newaxis,...])
my_model.cal_loss(conf_map,pd_conf_map,paf_map,pd_paf_map)

  • get_postprocess receives model_type from Config.MODEL and return a postprocess function The postprocess function is able to convert the model output into parsed human objects for evaluating and visualizing

pd_conf_map,pd_paf_map=my_model.forward(input_image[np.newaxis,...])
postprocess=Model.get_postprocess(Config.MODEL.LightweightOpenpose)
pd_humans=postprocess(pd_conf_map,pd_paf_map,dataset_type,data_format="channels_first")
for pd_human in pd_humans:
    pd_human.draw(input_image)

  • get_visualize receives model_type from Config.MODEL and return a visualize function The visualize function is able visualize model’s ouput feature map

pd_conf_map,pd_paf_map=my_model.forward(input_image[np.newaxis,...])
visualize=Model.get_visualize(Config.MODEL.LightweightOpenpose)
visualize(input_image,pd_conf_map,pd_paf_map,save_name="my_visual",save_dir="./vis_dir")

API Reference

C++ Prediction API

Python Training API

hyperpose package

Subpackages
hyperpose.Config package
Submodules
hyperpose.Config.config_lopps module
hyperpose.Config.config_mbtopps module
hyperpose.Config.config_opps module
hyperpose.Config.config_ppn module
hyperpose.Config.define module
class hyperpose.Config.define.BACKBONE

Bases: enum.Enum

An enumeration.

Default = 0
Mobilenetv1 = 1
Mobilenetv2 = 6
Resnet18 = 3
Resnet50 = 4
Vgg16 = 7
Vgg19 = 2
Vggtiny = 5
class hyperpose.Config.define.DATA

Bases: enum.Enum

An enumeration.

MPII = 1
MSCOCO = 0
MULTIPLE = 3
USERDEF = 2
class hyperpose.Config.define.KUNGFU

Bases: enum.Enum

An enumeration.

Pair_avg = 2
Sync_avg = 1
Sync_sgd = 0
class hyperpose.Config.define.MODEL

Bases: enum.Enum

An enumeration.

LightweightOpenpose = 1
MobilenetThinOpenpose = 3
Openpose = 0
Pifpaf = 4
PoseProposal = 2
class hyperpose.Config.define.OPTIM

Bases: enum.Enum

An enumeration.

Adam = 0
RMSprop = 1
SGD = 2
class hyperpose.Config.define.TRAIN

Bases: enum.Enum

An enumeration.

Parallel_train = 1
Single_train = 0
Module contents
hyperpose.Config.get_config()

get the config object with all the configuration information

get the config object based on the previous setting functions, the config object will be passed to the functions of Model and Dataset module to construct the system.

only the setting functions called before this get_config function is valid, thus use this function after all configuration done.

Parameters
None
Returns
config object

an edict object contains all the configuration information.

hyperpose.Config.set_batch_size(batch_size)

set the batch size in training

Parameters
arg1int

batch_size

Returns
None
hyperpose.Config.set_data_format(data_format)

set model dataformat

set the channel order of current model:

“channels_first” dataformat is faster in deployment
“channels_last” dataformat is more common the integrated pipeline will automaticly adapt to the chosen data format
Parameters
arg1string

available input:

| ‘channels_first’: data_shape N*C*H*W
| ‘channels_last’: data_shape N*H*W*C
Returns
None
hyperpose.Config.set_dataset_filter(dataset_filter)

set the user defined dataset filter

set the dataset filter as the input function. to uniformly format different dataset, Hyperpose organize the annotations of one image in one dataset in the similiar meta classes. for COCO dataset, it is COCOMeta; for MPII dataset, it is MPIIMeta. Meta classes will have some common information such as image_id, joint_list etc, they also have some dataset-specific imformation, such as mask, is_crowd, headbbx_list etc.

the dataset_fiter will perform on the Meta objects of the corresponding dataset, if it returns True, the image and annotaions the Meta object related will be kept, otherwise it will be filtered out. Please refer the Dataset.xxxMeta classes for better use.

Parameters
arg1function

a function receive a meta object as input, return a bool value indicates whether the meta should be kept or filtered out. return Ture for keeping and False for depricating the object. default: None

Returns
None
hyperpose.Config.set_dataset_path(dataset_path)

set the path of the dataset

set the path of the directory where dataset is,if the dataset doesn’t exist in this directory, then it will be automaticly download in this directory and decoded.

Parameters
arg1String

a string indicates the path of the dataset, default: ./data

Returns
None
hyperpose.Config.set_dataset_type(dataset_type)

set the dataset for train and evaluate

set which dataset to use, the process of downlaoding, decoding, reformatting of different type of dataset is automatic. the evaluation metric of different dataset follows their official metric, for COCO is MAP, for MPII is MPCH.

This API also receive user-defined dataset class, which should implement the following functions

__init__: take the config object with all configuration to init the dataset
get_parts: return a enum class which defines the key point definition of the dataset
get_limbs: return a [2*num_limbs] array which defines the limb definition of the dataset
get_colors: return a list which defines the visualization color of the limbs
get_train_dataset: return a tensorflow dataset which contains elements for training. each element should contains an image path and a target dict decoded in bytes by _pickle
get_eval_dataset: return a tensorflow dataset which contains elements for evaluating. each element should contains an image path and an image id
official_eval: if want to evaluate on this user-defined dataset, evalutation function should be implemented. one can refer the Dataset.mpii_dataset and Dataset.mscoco_dataset for detailed information.
Parameters
arg1Config.DATA

a enum value of enum class Config.DATA or user-defined dataset available options:

| Config.DATA.MSCOCO
| Config.DATA.MPII
| user-defined dataset
Returns
None
hyperpose.Config.set_dataset_version(dataset_version)
hyperpose.Config.set_domainadapt_dataset(domainadapt_train_img_paths, domainadapt_scale_rate=1)
hyperpose.Config.set_kungfu_option(kungfu_option)

set the optimizor of parallel training

kungfu distribute training library needs to wrap tensorflow optimizor in kungfu optimizor, this function is to choose kungfu optimizor wrap type

Parameters
arg1Config.KUNGFU

a enum value of enum class Config.KUNGFU available options:

| Config.KUNGFU.Sync_sgd (SynchronousSGDOptimizer, hyper-parameter-robus)
| Config.KUNGFU.Sync_avg (SynchronousAveragingOptimizer)
| Config.KUNGFU.Pair_avg (PairAveragingOptimizer, communication-efficient)
Returns
None
hyperpose.Config.set_learning_rate(learning_rate)

set the learning rate in training

Parameters
arg1float

learning rate

Returns
None
hyperpose.Config.set_log_interval(log_interval)

set the frequency of logging

set the how many iteration intervals between two log information

Parameters
arg1Int

a int value indicates the iteration number bwteen two logs default: 1

Returns
None
hyperpose.Config.set_model_arch(model_arch)

set user defined model architecture

replace default model architecture with user-defined model architecture, use it in the following training and evaluation

Parameters
arg1tensorlayer.models.MODEL

An object of a model class inherit from tensorlayer.models.MODEL class, should implement forward function and cal_loss function to make it compatible with the existing pipeline

The forward funtion should follow the signature below:

| openpose models: def forward(self,x,is_train=False) ,return conf_map,paf_map,stage_confs,stage_pafs
| poseproposal models: def forward(self,x,is_train=False), return pc,pi,px,py,pw,ph,pe

The cal_loss function should follow the signature below:

| openpose models: def cal_loss(self,stage_confs,stage_pafs,gt_conf,gt_paf,mask), return loss,loss_confs,loss_pafs
| poseproposal models: def cal_loss(self,tc,tx,ty,tw,th,te,te_mask,pc,pi,px,py,pw,ph,pe):

return loss_rsp,loss_iou,loss_coor,loss_size,loss_limb

Returns
None
hyperpose.Config.set_model_backbone(model_backbone)

set preset model backbones

set current model backbone to other common backbones different backbones have different computation complexity this enable dynamicly adapt the model architecture to approriate size.

Parameters
arg1Config.BACKBONE

a enum value of enum class Config.BACKBONE available options:

| Config.BACKBONE.DEFUALT (default backbone of the architecture)
| Config.BACKBONE.MobilenetV1
| Config.BACKBONE.MobilenetV2
| Config.BACKBONE.Vggtiny
| Config.BACKBONE.Vgg16
| Config.BACKBONE.Vgg19
| Config.BACKBONE.Resnet18
| Config.BACKBONE.Resnet50
Returns
None
hyperpose.Config.set_model_limbs(userdef_limbs)
hyperpose.Config.set_model_name(model_name)

set the name of model

the models are distinguished by their names,so it is necessary to set model’s name when train multiple models at the same time. each model’s ckpt data and log are saved on the ‘save_dir/model_name’ directory, the following directory are determined:

directory to save model ./save_dir/model_name/model_dir
directory to save train result ./save_dir/model_name/train_vis_dir
directory to save evaluate result ./save_dir/model_name/eval_vis_dir
directory to save dataset visualize result ./save_dir/model_name/data_vis_dir
file path to save train log ./save_dir/model_name/log.txt
Parameters
arg1string

name of the model

Returns
None
hyperpose.Config.set_model_parts(userdef_parts)
hyperpose.Config.set_model_type(model_type)

set preset model architecture

configure the model architecture as one of the desired preset model architectures

Parameters
arg1Config.MODEL

a enum value of enum class Config.MODEL, available options:

| Config.MODEL.Openpose (original Openpose)
| Config.MODEL.LightweightOpenpose (lightweight variant version of Openpose,real-time on cpu)
| Config.MODEL.PoseProposal (pose proposal network)
| Config.MODEL.MobilenetThinOpenpose (lightweight variant version of openpose)
Returns
None
hyperpose.Config.set_multiple_dataset(multiple_dataset_configs)
hyperpose.Config.set_official_dataset(official_flag)
hyperpose.Config.set_optim_type(optim_type)
hyperpose.Config.set_pretrain(enable)
hyperpose.Config.set_pretrain_dataset_path(pretrain_dataset_path)
hyperpose.Config.set_save_interval(save_interval)
hyperpose.Config.set_train_type(train_type)

set single_train or parallel train

default using single train, which train the model on one GPU. set parallel train will use Kungfu library to accelerate training on multiple GPU.

to use parallel train better, it is also allow to set parallel training optimizor by set_kungfu_option.

Parameters
arg1Config.TRAIN

a enum value of enum class Config.TRAIN,available options:

| Config.TRAIN.Single_train
| Config.TRAIN.Parallel_train
Returns
None
hyperpose.Config.set_useradd_data(useradd_train_img_paths, useradd_train_targets, useradd_scale_rate=1)
hyperpose.Config.set_userdef_dataset(userdef_dataset)
hyperpose.Dataset package
Subpackages
hyperpose.Dataset.mpii_dataset package
Submodules
hyperpose.Dataset.mpii_dataset.dataset module
class hyperpose.Dataset.mpii_dataset.dataset.MPII_dataset(config, input_kpt_cvter=None, output_kpt_cvter=None, dataset_filter=None)

Bases: hyperpose.Dataset.base_dataset.Base_dataset

a dataset class specified for mpii dataset, provides uniform APIs

Methods

get_eval_dataset(self[, in_list])

provide uniform tensorflow dataset for evaluating

get_train_dataset(self[, in_list])

provide uniform tensorflow dataset for training

official_eval(self, pd_json[, eval_dir])

providing official evaluation of MPII dataset

prepare_dataset(self)

download,extract, and reformat the dataset the official dataset is in .mat format, format it into json format automaticly.

visualize(self[, vis_num])

visualize annotations of the train dataset

generate_eval_data

generate_train_data

get_colors

get_dataset_type

get_input_kpt_cvter

get_output_kpt_cvter

get_parts

set_dataset_version

set_input_kpt_cvter

set_output_kpt_cvter
generate_eval_data(self)
generate_train_data(self)
get_colors(self)
get_dataset_type(self)
get_input_kpt_cvter(self)
get_output_kpt_cvter(self)
get_parts(self)
official_eval(self, pd_json, eval_dir='./eval_dir')

providing official evaluation of MPII dataset

output model metrics of PCHs on mpii evaluation dataset(split automaticly)

Parameters
arg1String

A string path of the json file in the same format of cocoeval annotation file(person_keypoints_val2017.json) which contains predicted results. one can refer the evaluation pipeline of models for generation procedure of this json file.

arg2String

A string path indicates where the result json file which contains MPII PCH metrics of various keypoint saves.

Returns
None
prepare_dataset(self)

download,extract, and reformat the dataset the official dataset is in .mat format, format it into json format automaticly.

Parameters
None
Returns
None
set_input_kpt_cvter(self, input_kpt_cvter)
set_output_kpt_cvter(self, output_kpt_cvter)
visualize(self, vis_num=10)

visualize annotations of the train dataset

visualize the annotation points in the image to help understand and check annotation the visualized image will be saved in the “data_vis_dir” of the corresponding model directory(specified by model name). the visualized annotations are from the train dataset.

Parameters
arg1Int

An integer indicates how many images with their annotations are going to be visualized.

Returns
None
hyperpose.Dataset.mpii_dataset.dataset.init_dataset(config)
hyperpose.Dataset.mpii_dataset.define module
class hyperpose.Dataset.mpii_dataset.define.MpiiPart

Bases: enum.Enum

An enumeration.

Headtop = 9
LAnkle = 5
LElbow = 14
LHip = 3
LKnee = 4
LShoulder = 13
LWrist = 15
Pelvis = 6
RAnkle = 0
RElbow = 11
RHip = 2
RKnee = 1
RShoulder = 12
RWrist = 10
Thorax = 7
UpperNeck = 8
static from_coco(human)
hyperpose.Dataset.mpii_dataset.define.opps_input_converter(mpii_kpts)
hyperpose.Dataset.mpii_dataset.define.opps_output_converter(kpt_list)
hyperpose.Dataset.mpii_dataset.define.ppn_input_converter(coco_kpts)
hyperpose.Dataset.mpii_dataset.define.ppn_output_converter(kpt_list)
hyperpose.Dataset.mpii_dataset.format module
class hyperpose.Dataset.mpii_dataset.format.MPIIMeta(image_path, annos_list)

Bases: object

Methods

to_anns_list
to_anns_list(self)
class hyperpose.Dataset.mpii_dataset.format.PoseInfo(image_dir, annos_path, dataset_filter=None)

Bases: object

Methods

get_center_list

get_headbbx_list

get_image_annos

get_image_id_list

get_image_list

get_kpt_list

get_scale_list
get_center_list(self)
get_headbbx_list(self)
get_image_annos(self)
get_image_id_list(self)
get_image_list(self)
get_kpt_list(self)
get_scale_list(self)
hyperpose.Dataset.mpii_dataset.format.generate_json(mat_path, is_test=False)
hyperpose.Dataset.mpii_dataset.generate module
hyperpose.Dataset.mpii_dataset.generate.generate_eval_data(eval_images_path, eval_annos_path, dataset_filter=None)
hyperpose.Dataset.mpii_dataset.generate.generate_train_data(train_images_path, train_annos_path, dataset_filter=None, input_kpt_cvter=<function <lambda> at 0x7f0141475158>)
hyperpose.Dataset.mpii_dataset.prepare module
hyperpose.Dataset.mpii_dataset.prepare.prepare_dataset(dataset_path)
hyperpose.Dataset.mpii_dataset.utils module
hyperpose.Dataset.mpii_dataset.utils.affine_transform(pt, t)
hyperpose.Dataset.mpii_dataset.utils.get_3rd_point(a, b)
hyperpose.Dataset.mpii_dataset.utils.get_affine_transform(center, scale, rot, output_size, shift=array([0.0, 0.0], dtype=float32), inv=0)
hyperpose.Dataset.mpii_dataset.utils.get_dir(src_point, rot_rad)
hyperpose.Dataset.mpii_dataset.visualize module
Module contents
hyperpose.Dataset.mscoco_dataset package
Submodules
hyperpose.Dataset.mscoco_dataset.dataset module
class hyperpose.Dataset.mscoco_dataset.dataset.MSCOCO_dataset(config, input_kpt_cvter=None, output_kpt_cvter=None, dataset_filter=None)

Bases: hyperpose.Dataset.base_dataset.Base_dataset

a dataset class specified for coco dataset, provides uniform APIs

Methods

get_eval_dataset(self[, in_list])

provide uniform tensorflow dataset for evaluating

get_train_dataset(self[, in_list])

provide uniform tensorflow dataset for training

official_eval(self, pd_json[, eval_dir])

providing official evaluation of COCO dataset

prepare_dataset(self)

download,extract, and reformat the dataset the official format is in zip format, extract it into json files and image files.

visualize(self, vis_num)

visualize annotations of the train dataset

generate_eval_data

generate_train_data

get_colors

get_dataset_type

get_input_kpt_cvter

get_output_kpt_cvter

get_parts

set_dataset_version

set_input_kpt_cvter

set_output_kpt_cvter
generate_eval_data(self)
generate_train_data(self)
get_colors(self)
get_dataset_type(self)
get_input_kpt_cvter(self)
get_output_kpt_cvter(self)
get_parts(self)
official_eval(self, pd_json, eval_dir='./eval_dir')

providing official evaluation of COCO dataset

using pycocotool.cocoeval class to perform official evaluation. output model metrics of MAPs on coco evaluation dataset

Parameters
arg1String

A string path of the json file in the same format of cocoeval annotation file(person_keypoints_val2017.json) which contains predicted results. one can refer the evaluation pipeline of models for generation procedure of this json file.

arg2String

A string path indicates where the json files of filtered intersection part of predict results and ground truth the filtered prediction file is stored in eval_dir/pd_ann.json the filtered ground truth file is stored in eval_dir/gt_ann.json

Returns
None
prepare_dataset(self)

download,extract, and reformat the dataset the official format is in zip format, extract it into json files and image files.

Parameters
None
Returns
None
set_input_kpt_cvter(self, input_kpt_cvter)
set_output_kpt_cvter(self, output_kpt_cvter)
visualize(self, vis_num)

visualize annotations of the train dataset

visualize the annotation points in the image to help understand and check annotation the visualized image will be saved in the “data_vis_dir” of the corresponding model directory(specified by model name). the visualized annotations are from the train dataset.

Parameters
arg1Int

An integer indicates how many images with their annotations are going to be visualized.

Returns
None
hyperpose.Dataset.mscoco_dataset.dataset.init_dataset(config)
hyperpose.Dataset.mscoco_dataset.define module
class hyperpose.Dataset.mscoco_dataset.define.CocoPart

Bases: enum.Enum

An enumeration.

LAnkle = 15
LEar = 3
LElbow = 7
LHip = 11
LKnee = 13
LShoulder = 5
LWrist = 9
Leye = 1
Nose = 0
RAnkle = 16
REar = 4
RElbow = 8
RHip = 12
RKnee = 14
RShoulder = 6
RWrist = 10
Reye = 2
hyperpose.Dataset.mscoco_dataset.define.opps_input_converter(coco_kpts)
hyperpose.Dataset.mscoco_dataset.define.opps_output_converter(kpt_list)
hyperpose.Dataset.mscoco_dataset.define.pifpaf_input_converter(coco_kpts)
hyperpose.Dataset.mscoco_dataset.define.pifpaf_output_converter(kpt_list)
hyperpose.Dataset.mscoco_dataset.define.ppn_input_converter(coco_kpts)
hyperpose.Dataset.mscoco_dataset.define.ppn_output_converter(kpt_list)
hyperpose.Dataset.mscoco_dataset.format module
class hyperpose.Dataset.mscoco_dataset.format.CocoMeta(image_id, img_url, img_meta, kpts_infos, masks, bbxs, is_crowd)

Bases: object

Be used in PoseInfo.

class hyperpose.Dataset.mscoco_dataset.format.PoseInfo(image_base_dir, anno_path, with_mask=True, dataset_filter=None, eval=False)

Bases: object

Use COCO for pose estimation, returns images with people only.

Methods

get_image_annos(self)

Read JSON file, and get and check the image list.

get_bbx_list

get_bbxs

get_image_id_list

get_image_list

get_keypoints

get_kpt_list

get_mask_list

load_images
get_bbx_list(self)
static get_bbxs(annos_info)
get_image_annos(self)

Read JSON file, and get and check the image list. Skip missing images.

get_image_id_list(self)
get_image_list(self)
static get_keypoints(annos_info)
get_kpt_list(self)
get_mask_list(self)
load_images(self)
hyperpose.Dataset.mscoco_dataset.generate module
hyperpose.Dataset.mscoco_dataset.generate.generate_eval_data(val_imgs_path, val_anns_path, dataset_filter=None)
hyperpose.Dataset.mscoco_dataset.generate.generate_train_data(train_imgs_path, train_anns_path, dataset_filter=None, input_kpt_cvter=<function <lambda> at 0x7f014149cb70>)
hyperpose.Dataset.mscoco_dataset.prepare module
hyperpose.Dataset.mscoco_dataset.prepare.prepare_dataset(data_path='./data', version='2017', task='person')

Download MSCOCO Dataset. Both 2014 and 2017 dataset have train, validate and test sets, but 2017 version put less data into the validation set (115k train, 5k validate) i.e. has more training data.

Parameters
pathstr

The path that the data is downloaded to, defaults is data/mscoco....

datasetstr

The MSCOCO dataset version, 2014 or 2017.

taskstr

person for pose estimation, caption for image captioning, instance for segmentation.

Returns
train_im_pathstr

Folder path of all training images.

train_ann_pathstr

File path of training annotations.

val_im_pathstr

Folder path of all validating images.

val_ann_pathstr

File path of validating annotations.

test_im_pathstr

Folder path of all testing images.

test_ann_pathNone

File path of testing annotations, but as the test sets of MSCOCO 2014 and 2017 do not have annotation, returns None.

References

Examples

>>> train_im_path, train_ann_path, val_im_path, val_ann_path, _, _ =         ...    tl.files.load_mscoco_dataset('data', '2017')
hyperpose.Dataset.mscoco_dataset.visualize module
Module contents
Submodules
hyperpose.Dataset.common module
hyperpose.Dataset.common.file_log(log_file, msg)
hyperpose.Dataset.common.get_domainadapt_targets(domainadapt_img_paths)
hyperpose.Dataset.common.imread_rgb_float(image_path, data_format='channels_first')
hyperpose.Dataset.common.imwrite_rgb_float(image, image_path, data_format='channels_first')
hyperpose.Dataset.common.unzip(path_to_zip_file, directory_to_extract_to)
hyperpose.Dataset.common.visualize(vis_dir, vis_num, dataset, parts, colors, dataset_name='default')
Module contents
hyperpose.Dataset.enum2dataset(dataset_type)
hyperpose.Dataset.get_dataset(config)

get dataset object based on the config object

consturct and return a dataset object based on the config. No matter what the bottom dataset type is, the APIs of the returned dataset object are uniform, they are the following APIs:

visualize: visualize annotations of the train dataset and save it in “data_vir_dir” get_dataset_type: return the type of the bottom dataset. get_train_dataset: return a uniform tensorflow dataset object for training. get_val_dataset: return a uniform tensorflow dataset object for evaluating. official_eval: perform official evaluation on this dataset.

The construction pipeline of this dataset object is below:

1.check whether the dataset file(official zip or mat) is under data_path, if it isn’t, download it from official website automaticly

2.decode the official dataset file, organize the annotations in corresponding Meta classes, conveniet for processing.

3.based on annotation, split train and evaluat part for furthur use.

if user defined thier own dataset_filter, it will be executed in the train dataset or evaluate dataset generating procedure.

use the APIs of this returned dataset object, the difference of different dataset is minimized.

Parameters
arg1config object

the config object return by Config.get_config() function, which includes all the configuration information.

Returns
dataset

a dataset object with unifrom APIs: visualize, get_dataset_type, get_train_dataset, get_val_dataset,official_eval

hyperpose.Dataset.get_pretrain_dataset(config)
hyperpose.Model package
Subpackages
hyperpose.Model.openpose package
Subpackages
hyperpose.Model.openpose.model package
Submodules
hyperpose.Model.openpose.model.lw_openpose module
class hyperpose.Model.openpose.model.lw_openpose.LightWeightOpenPose(parts=<enum 'CocoPart'>, limbs=[(1, 8), (8, 9), (9, 10), (1, 11), (11, 12), (12, 13), (1, 2), (2, 3), (3, 4), (2, 16), (1, 5), (5, 6), (6, 7), (5, 17), (1, 0), (0, 14), (0, 15), (14, 16), (15, 17)], colors=None, n_pos=19, n_limbs=19, num_channels=128, hin=368, win=368, hout=46, wout=46, backbone=None, pretraining=False, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

Cpm_stage([n_filter, in_channels, data_format])

Attributes

Dilated_mobilenet([data_format])

Attributes

Init_stage([n_filter, n_confmaps, …])

Attributes

Refinement_stage([n_filter, in_channels, …])

Attributes

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x[, is_train, stage_num, …])

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self, x)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_loss

count_params

print_params
class Cpm_stage(n_filter=128, in_channels=512, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class Dilated_mobilenet(data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class Init_stage(n_filter=128, n_confmaps=19, n_pafmaps=38, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class Refinement_stage(n_filter=128, in_channels=185, n_confmaps=19, n_pafmaps=38, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

Refinement_block(n_filter, in_channels[, …])

Attributes

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
class Refinement_block(n_filter, in_channels, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
cal_loss(self, gt_conf, gt_paf, mask, stage_confs, stage_pafs)
forward(self, x, is_train=False, stage_num=1, domainadapt=False)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
infer(self, x)

Set this network in evaluation mode.

hyperpose.Model.openpose.model.lw_openpose.conv_block(n_filter, in_channels, filter_size=(3, 3), strides=(1, 1), dilation_rate=(1, 1), W_init=<tensorlayer.initializers.TruncatedNormal object at 0x7f0140f9ad30>, b_init=<tensorlayer.initializers.TruncatedNormal object at 0x7f0140f9ad30>, padding='SAME', data_format='channels_first')
hyperpose.Model.openpose.model.lw_openpose.dw_conv_block(n_filter, in_channels, filter_size=(3, 3), strides=(1, 1), dilation_rate=(1, 1), W_init=<tensorlayer.initializers.TruncatedNormal object at 0x7f0140f9ad30>, b_init=<tensorlayer.initializers.TruncatedNormal object at 0x7f0140f9ad30>, data_format='channels_first')
hyperpose.Model.openpose.model.lw_openpose.nobn_dw_conv_block(n_filter, in_channels, filter_size=(3, 3), strides=(1, 1), W_init=<tensorlayer.initializers.TruncatedNormal object at 0x7f0140f9ad30>, b_init=<tensorlayer.initializers.TruncatedNormal object at 0x7f0140f9ad30>, data_format='channels_first')
hyperpose.Model.openpose.model.mbv2_sm_openpose module
class hyperpose.Model.openpose.model.mbv2_sm_openpose.Mobilenetv2_small_Openpose(parts=<enum 'CocoPart'>, limbs=[(1, 8), (8, 9), (9, 10), (1, 11), (11, 12), (12, 13), (1, 2), (2, 3), (3, 4), (2, 16), (1, 5), (5, 6), (6, 7), (5, 17), (1, 0), (0, 14), (0, 15), (14, 16), (15, 17)], colors=None, n_pos=19, n_limbs=19, num_channels=128, hin=368, win=368, hout=46, wout=46, backbone=None, pretraining=False, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

Init_stage([n_confmaps, n_pafmaps, …])

Attributes

Mobilenetv2_variant([data_format])

Attributes

Refinement_stage([n_confmaps, n_pafmaps, …])

Attributes

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x[, mask_conf, mask_paf, …])

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self, x)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_loss

count_params

print_params
class Init_stage(n_confmaps=19, n_pafmaps=38, in_channels=704, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class Mobilenetv2_variant(data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class Refinement_stage(n_confmaps=19, n_pafmaps=38, in_channels=761, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
cal_loss(self, gt_conf, gt_paf, mask, stage_confs, stage_pafs)
forward(self, x, mask_conf=None, mask_paf=None, is_train=False, stage_num=4, domainadapt=False)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
infer(self, x)

Set this network in evaluation mode.

hyperpose.Model.openpose.model.mbv2_sm_openpose.conv_block(n_filter=32, in_channels=3, filter_size=3, 3, strides=1, 1, act=tensorflow.nn.relu, padding='SAME', data_format='channels_first')
hyperpose.Model.openpose.model.mbv2_sm_openpose.separable_block(n_filter=32, in_channels=3, filter_size=3, 3, strides=1, 1, act=tensorflow.nn.relu, padding='SAME', data_format='channels_first')
hyperpose.Model.openpose.model.mbv2_th_openpose module
class hyperpose.Model.openpose.model.mbv2_th_openpose.MobilenetThinOpenpose(parts=<enum 'CocoPart'>, limbs=[(1, 8), (8, 9), (9, 10), (1, 11), (11, 12), (12, 13), (1, 2), (2, 3), (3, 4), (2, 16), (1, 5), (5, 6), (6, 7), (5, 17), (1, 0), (0, 14), (0, 15), (14, 16), (15, 17)], colors=None, n_pos=19, n_limbs=19, num_channels=128, hin=368, win=368, hout=46, wout=46, backbone=None, pretraining=False, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

Init_stage([n_confmaps, n_pafmaps, …])

Attributes

Mobilenetv2_variant([data_format])

Attributes

Refinement_stage([n_confmaps, n_pafmaps, …])

Attributes

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x[, is_train, stage_num, …])

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self, x)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_loss

count_params

print_params
class Init_stage(n_confmaps=19, n_pafmaps=38, in_channels=1152, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class Mobilenetv2_variant(data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class Refinement_stage(n_confmaps=19, n_pafmaps=38, in_channels=1209, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
cal_loss(self, gt_conf, gt_paf, mask, stage_confs, stage_pafs)
forward(self, x, is_train=False, stage_num=5, domainadapt=False)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
infer(self, x)

Set this network in evaluation mode.

hyperpose.Model.openpose.model.mbv2_th_openpose.conv_block(n_filter=32, in_channels=3, filter_size=3, 3, strides=1, 1, act=tensorflow.nn.relu, padding='SAME', data_format='channels_first')
hyperpose.Model.openpose.model.mbv2_th_openpose.separable_block(n_filter=32, in_channels=3, filter_size=3, 3, strides=1, 1, dilation_rate=1, 1, act=tensorflow.nn.relu, data_format='channels_first')
hyperpose.Model.openpose.model.openpose module
class hyperpose.Model.openpose.model.openpose.OpenPose(parts=<enum 'CocoPart'>, limbs=[(1, 8), (8, 9), (9, 10), (1, 11), (11, 12), (12, 13), (1, 2), (2, 3), (3, 4), (2, 16), (1, 5), (5, 6), (6, 7), (5, 17), (1, 0), (0, 14), (0, 15), (14, 16), (15, 17)], colors=None, n_pos=19, n_limbs=19, num_channels=128, hin=368, win=368, hout=46, wout=46, backbone=None, pretraining=False, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

Init_stage([n_confmaps, n_pafmaps, …])

Attributes

Refinement_stage([n_confmaps, n_pafmaps, …])

Attributes

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x[, is_train, stage_num, …])

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self, x[, stage_num])

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_loss

count_params

print_params
class Init_stage(n_confmaps=19, n_pafmaps=38, in_channels=128, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class Refinement_stage(n_confmaps=19, n_pafmaps=38, in_channels=185, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
cal_loss(self, gt_conf, gt_paf, mask, stage_confs, stage_pafs)
forward(self, x, is_train=False, stage_num=5, domainadapt=False)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
infer(self, x, stage_num=5)

Set this network in evaluation mode.

Module contents
Submodules
hyperpose.Model.openpose.define module
class hyperpose.Model.openpose.define.CocoPart

Bases: enum.Enum

An enumeration.

Background = 18
LAnkle = 13
LEar = 17
LElbow = 6
LEye = 15
LHip = 11
LKnee = 12
LShoulder = 5
LWrist = 7
Neck = 1
Nose = 0
RAnkle = 10
REar = 16
RElbow = 3
REye = 14
RHip = 8
RKnee = 9
RShoulder = 2
RWrist = 4
class hyperpose.Model.openpose.define.MpiiPart

Bases: enum.Enum

An enumeration.

Background = 15
Center = 14
Headtop = 0
LAnkle = 13
LElbow = 6
LHip = 11
LKnee = 12
LShoulder = 5
LWrist = 7
Neck = 1
RAnkle = 10
RElbow = 3
RHip = 8
RKnee = 9
RShoulder = 2
RWrist = 4
hyperpose.Model.openpose.define.get_coco_flip_list()
hyperpose.Model.openpose.define.get_mpii_flip_list()
hyperpose.Model.openpose.eval module
hyperpose.Model.openpose.eval.evaluate(model, dataset, config, vis_num=30, total_eval_num=30, enable_multiscale_search=True)

evaluate pipeline of Openpose class models

input model and dataset, the evaluate pipeline will start automaticly the evaluate pipeline will: 1.loading newest model at path ./save_dir/model_name/model_dir/newest_model.npz 2.perform inference and parsing over the chosen evaluate dataset 3.visualize model output in evaluation in directory ./save_dir/model_name/eval_vis_dir 4.output model metrics by calling dataset.official_eval()

Parameters
arg1tensorlayer.models.MODEL

a preset or user defined model object, obtained by Model.get_model() function

arg2dataset

a constructed dataset object, obtained by Dataset.get_dataset() function

arg3Int

an Integer indicates how many model output should be visualized

arg4Int

an Integer indicates how many images should be evaluated

Returns
None
hyperpose.Model.openpose.eval.infer_one_img(model, post_processor, img, img_id=- 1, enable_multiscale_search=False, is_visual=False, save_dir='./vis_dir')
hyperpose.Model.openpose.eval.visualize(img, img_id, humans, conf_map, paf_map, save_dir)
hyperpose.Model.openpose.infer module
class hyperpose.Model.openpose.infer.Connection(peak_src_id, peak_dst_id, score)

Bases: object

class hyperpose.Model.openpose.infer.Peak(peak_idx, part_idx, y, x, score)

Bases: object

class hyperpose.Model.openpose.infer.Post_Processor(parts, limbs, colors, debug=False)

Bases: object

Methods

debug_print

get_paf_vectors

get_peak_map

process

process_paf
debug_print(self, msg)
get_paf_vectors(self, limb_id, vec_src, vec_dst, paf_map)
get_peak_map(self, conf_map)
process(self, conf_map, paf_map, img_h, img_w, data_format='channels_first')
process_paf(self, peak_map, conf_map, paf_map)
hyperpose.Model.openpose.train module
hyperpose.Model.openpose.train.get_paramed_map_fn(hin, win, hout, wout, parts, limbs, flip_list=None, data_format='channels_first')
hyperpose.Model.openpose.train.parallel_train(train_model, dataset, config)

Parallel train pipeline of openpose class models

input model and dataset, the train pipeline will start automaticly the train pipeline will: 1.store and restore ckpt in directory ./save_dir/model_name/model_dir 2.log loss information in directory ./save_dir/model_name/log.txt 3.visualize model output periodly during training in directory ./save_dir/model_name/train_vis_dir the newest model is at path ./save_dir/model_name/model_dir/newest_model.npz

Parameters
arg1tensorlayer.models.MODEL

a preset or user defined model object, obtained by Model.get_model() function

arg2dataset

a constructed dataset object, obtained by Dataset.get_dataset() function

Returns
None
hyperpose.Model.openpose.train.regulize_loss(target_model, weight_decay_factor)
hyperpose.Model.openpose.train.single_train(train_model, dataset, config)

Single train pipeline of Openpose class models

input model and dataset, the train pipeline will start automaticly the train pipeline will: 1.store and restore ckpt in directory ./save_dir/model_name/model_dir 2.log loss information in directory ./save_dir/model_name/log.txt 3.visualize model output periodly during training in directory ./save_dir/model_name/train_vis_dir the newest model is at path ./save_dir/model_name/model_dir/newest_model.npz

Parameters
arg1tensorlayer.models.MODEL

a preset or user defined model object, obtained by Model.get_model() function

arg2dataset

a constructed dataset object, obtained by Dataset.get_dataset() function

Returns
None
hyperpose.Model.openpose.utils module
hyperpose.Model.openpose.utils.cal_vectormap_fast(vectormap, countmap, i, v_start, v_end)
hyperpose.Model.openpose.utils.cal_vectormap_ori(vectormap, countmap, i, v_start, v_end)
hyperpose.Model.openpose.utils.draw_results(images, heats_ground, heats_result, pafs_ground, pafs_result, masks, save_dir, name='', data_format='channels_first')

Save results for debugging.

Parameters
imagesa list of RGB images
heats_grounda list of keypoint heat maps or None
heats_resulta list of keypoint heat maps or None
pafs_grounda list of paf vector maps or None
pafs_resulta list of paf vector maps or None
masksa list of mask for people
hyperpose.Model.openpose.utils.get_colors(dataset_type)
hyperpose.Model.openpose.utils.get_flip_list(dataset_type)
hyperpose.Model.openpose.utils.get_heatmap(annos, height, width, hout, wout, parts, limbs, data_format='channels_first')
hyperpose.Model.openpose.utils.get_limbs(dataset_type)
hyperpose.Model.openpose.utils.get_parts(dataset_type)
hyperpose.Model.openpose.utils.get_vectormap(annos, height, width, hout, wout, parts, limbs, data_format='channels_first')
hyperpose.Model.openpose.utils.postprocess(conf_map, paf_map, img_h, img_w, parts, limbs, data_format='channels_first', colors=None)

postprocess function of openpose class models

take model predicted feature maps, output parsed human objects, each one contains all detected keypoints of the person

Parameters
arg1numpy array

model predicted conf_map, heatmaps of keypoints, shape C*H*W(channels_first) or H*W*C(channels_last)

arg2numpy array

model predicted paf_map, heatmaps of limbs, shape C*H*W(channels_first) or H*W*C(channels_last)

arg3Config.DATA

an enum value of enum class Config.DATA width of the model output, will be the width of the generated maps

arg4string

data format speficied for channel order available input: ‘channels_first’: data_shape C*H*W ‘channels_last’: data_shape H*W*C

Returns
list

contain object of humans,see Model.Human for detail information of Human object

hyperpose.Model.openpose.utils.preprocess(annos, img_height, img_width, model_hout, model_wout, parts, limbs, data_format='channels_first')

preprocess function of openpose class models

take keypoints annotations, image height and width, model input height and width, and dataset type, return the constructed conf_map and paf_map

Parameters
arg1list

a list of annotation, each annotation is a list of keypoints that belongs to a person, each keypoint follows the format (x,y), and x<0 or y<0 if the keypoint is not visible or not annotated. the annotations must from a known dataset_type, other wise the keypoint and limbs order will not be correct.

arg2Int

height of the input image, need this to make use of keypoint annotation

arg3Int

width of the input image, need this to make use of keypoint annotation

arg4Int

height of the model output, will be the height of the generated maps

arg5Int

width of the model output, will be the width of the generated maps

arg6Config.DATA

a enum value of enum class Config.DATA dataset_type where the input annotation list from, because to generate correct conf_map and paf_map, the order of keypoints and limbs should be awared.

arg7string

data format speficied for channel order available input: ‘channels_first’: data_shape C*H*W ‘channels_last’: data_shape H*W*C

Returns
list

including two element conf_map: heatmaps of keypoints, shape C*H*W(channels_first) or H*W*C(channels_last) paf_map: heatmaps of limbs, shape C*H*W(channels_first) or H*W*C(channels_last)

hyperpose.Model.openpose.utils.put_heatmap(heatmap, plane_idx, center, stride, sigma)
hyperpose.Model.openpose.utils.vis_annos(image, annos, save_dir, name='')

Save results for debugging.

Parameters
imagessingle RGB image
annosannotation, list of lists
hyperpose.Model.openpose.utils.visualize(img, conf_map, paf_map, save_name='maps', save_dir='./save_dir/vis_dir', data_format='channels_first', save_tofile=True)

visualize function of openpose class models

take model predict feature maps, output visualized image. the image will be saved at ‘save_dir’/’save_name’_visualize.png

Parameters
arg1numpy array

image

arg2numpy array

model output conf_map, heatmaps of keypoints, shape C*H*W(channels_first) or H*W*C(channels_last)

arg3numpy array

model output paf_map, heatmaps of limbs, shape C*H*W(channels_first) or H*W*C(channels_last)

arg4String

specify output image name to distinguish.

arg5String

specify which directory to save the visualized image.

arg6string

data format speficied for channel order available input: ‘channels_first’: data_shape C*H*W ‘channels_last’: data_shape H*W*C

Returns
None
Module contents
hyperpose.Model.pose_proposal package
Submodules
hyperpose.Model.pose_proposal.define module
class hyperpose.Model.pose_proposal.define.CocoPart

Bases: enum.Enum

An enumeration.

Instance = 1
LAnkle = 13
LEar = 17
LElbow = 6
LEye = 15
LHip = 11
LKnee = 12
LShoulder = 5
LWrist = 7
Nose = 0
RAnkle = 10
REar = 16
RElbow = 3
REye = 14
RHip = 8
RKnee = 9
RShoulder = 2
RWrist = 4
class hyperpose.Model.pose_proposal.define.MpiiPart

Bases: enum.Enum

An enumeration.

Center = 14
Headtop = 0
Instance = 15
LAnkle = 13
LElbow = 6
LHip = 11
LKnee = 12
LShoulder = 5
LWrist = 7
Neck = 1
RAnkle = 10
RElbow = 3
RHip = 8
RKnee = 9
RShoulder = 2
RWrist = 4
hyperpose.Model.pose_proposal.define.get_coco_flip_list()
hyperpose.Model.pose_proposal.define.get_mpii_flip_list()
hyperpose.Model.pose_proposal.eval module
hyperpose.Model.pose_proposal.eval.evaluate(model, dataset, config, vis_num=30, total_eval_num=30, enable_multiscale_search=False)

evaluate pipeline of poseProposal class models

input model and dataset, the evaluate pipeline will start automaticly the evaluate pipeline will: 1.loading newest model at path ./save_dir/model_name/model_dir/newest_model.npz 2.perform inference and parsing over the chosen evaluate dataset 3.visualize model output in evaluation in directory ./save_dir/model_name/eval_vis_dir 4.output model metrics by calling dataset.official_eval()

Parameters
arg1tensorlayer.models.MODEL

a preset or user defined model object, obtained by Model.get_model() function

arg2dataset

a constructed dataset object, obtained by Dataset.get_dataset() function

arg3Int

an Integer indicates how many model output should be visualized

arg4Int

an Integer indicates how many images should be evaluated

Returns
None
hyperpose.Model.pose_proposal.eval.infer_one_img(model, post_processor, img, img_id=- 1, is_visual=False, save_dir='./vis_dir/pose_proposal')
hyperpose.Model.pose_proposal.eval.visualize(img, img_id, humans, predicts, hnei, wnei, hout, wout, limbs, save_dir)
hyperpose.Model.pose_proposal.infer module
class hyperpose.Model.pose_proposal.infer.Post_Processor(parts, limbs, colors, thresh_hold=0.3, eps=1e-08, debug=True)

Bases: object

Methods

process
process(self, pc, pi, px, py, pw, ph, pe, scale_w_rate=1, scale_h_rate=1)
hyperpose.Model.pose_proposal.model module
class hyperpose.Model.pose_proposal.model.PoseProposal(parts=<enum 'CocoPart'>, limbs=[(1, 8), (8, 9), (9, 10), (1, 11), (11, 12), (12, 13), (1, 2), (2, 3), (3, 4), (1, 5), (5, 6), (6, 7), (1, 0), (0, 14), (0, 15), (14, 16), (15, 17)], colors=None, K_size=18, L_size=17, win=384, hin=384, wout=12, hout=12, wnei=9, hnei=9, lmd_rsp=0.25, lmd_iou=1, lmd_coor=5, lmd_size=5, lmd_limb=0.5, backbone=None, pretraining=False, data_format='channels_first')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x[, is_train, domainadapt])

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self, x)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_iou

cal_loss

count_params

print_params

restore_coor
cal_iou(self, bbx1, bbx2)
cal_loss(self, delta, tx, ty, tw, th, te, te_mask, pc, pi, px, py, pw, ph, pe, eps=1e-06)
forward(self, x, is_train=False, domainadapt=False)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
infer(self, x)

Set this network in evaluation mode.

restore_coor(self, x, y, w, h)
hyperpose.Model.pose_proposal.train module
hyperpose.Model.pose_proposal.train.get_paramed_map_fn(hin, win, hout, wout, hnei, wnei, parts, limbs, data_format='channels_first')
hyperpose.Model.pose_proposal.train.parallel_train(train_model, dataset, config)

Parallel train pipeline of PoseProposal class models

input model and dataset, the train pipeline will start automaticly the train pipeline will: 1.store and restore ckpt in directory ./save_dir/model_name/model_dir 2.log loss information in directory ./save_dir/model_name/log.txt 3.visualize model output periodly during training in directory ./save_dir/model_name/train_vis_dir the newest model is at path ./save_dir/model_name/model_dir/newest_model.npz

Parameters
arg1tensorlayer.models.MODEL

a preset or user defined model object, obtained by Model.get_model() function

arg2dataset

a constructed dataset object, obtained by Dataset.get_dataset() function

Returns
None
hyperpose.Model.pose_proposal.train.regulize_loss(target_model, weight_decay_factor)
hyperpose.Model.pose_proposal.train.single_train(train_model, dataset, config)

Single train pipeline of PoseProposal class models

input model and dataset, the train pipeline will start automaticly the train pipeline will: 1.store and restore ckpt in directory ./save_dir/model_name/model_dir 2.log loss information in directory ./save_dir/model_name/log.txt 3.visualize model output periodly during training in directory ./save_dir/model_name/train_vis_dir the newest model is at path ./save_dir/model_name/model_dir/newest_model.npz

Parameters
arg1tensorlayer.models.MODEL

a preset or user defined model object, obtained by Model.get_model() function

arg2dataset

a constructed dataset object, obtained by Dataset.get_dataset() function

Returns
None
hyperpose.Model.pose_proposal.utils module
hyperpose.Model.pose_proposal.utils.cal_iou(bbx1, bbx2)
hyperpose.Model.pose_proposal.utils.draw_bbx(img, img_pc, rx, ry, rw, rh, threshold=0.7)
hyperpose.Model.pose_proposal.utils.draw_edge(img, img_e, rx, ry, rw, rh, hnei, wnei, hout, wout, limbs, threshold=0.7)
hyperpose.Model.pose_proposal.utils.draw_results(img, predicts, targets, parts, limbs, save_dir, threshold=0.3, name='', is_train=True, data_format='channels_first')
hyperpose.Model.pose_proposal.utils.get_colors(dataset_type)
hyperpose.Model.pose_proposal.utils.get_flip_list(dataset_type)
hyperpose.Model.pose_proposal.utils.get_limbs(dataset_type)
hyperpose.Model.pose_proposal.utils.get_parts(dataset_type)
hyperpose.Model.pose_proposal.utils.get_pose_proposals(kpts_list, bbxs, hin, win, hout, wout, hnei, wnei, parts, limbs, img_mask=None, data_format='channels_first')
hyperpose.Model.pose_proposal.utils.non_maximium_supress(bbxs, scores, thres)
hyperpose.Model.pose_proposal.utils.postprocess(predicts, parts, limbs, data_format='channels_first', colors=None)

postprocess function of poseproposal class models

take model predicted feature maps of delta,tx,ty,tw,th,te,te_mask, output parsed human objects, each one contains all detected keypoints of the person

Parameters
arg1list

a list of model output: delta,tx,ty,tw,th,te,te_mask delta: keypoint confidence feature map, shape [C,H,W](channels_first) or [H,W,C](channels_last) tx: keypoints bbx center x coordinates, divided by gridsize, shape [C,H,W](channels_first) or [H,W,C](channels_last) ty: keypoints bbx center y coordinates, divided by gridsize, shape [C,H,W](channels_first) or [H,W,C](channels_last) tw: keypoints bbxs width w, divided by image width, shape [C,H,W](channels_first) or [H,W,C](channels_last) th: keypoints bbxs height h, divided by image width, shape [C,H,W](channels_first) or [H,W,C](channels_last) te: edge confidence feature map, shape [C,H,W,Hnei,Wnei](channels_first) or [H,W,Hnei,Wnei,C](channels_last) te_mask: mask of edge confidence feature map, used for loss caculation, shape [C,H,W,Hnei,Wnei](channels_first) or [H,W,Hnei,Wnei,C](channels_last)

arg2: Config.DATA

a enum value of enum class Config.DATA dataset_type where the input annotation list from, because to generate correct conf_map and paf_map, the order of keypoints and limbs should be awared.

arg3string

data format speficied for channel order available input: ‘channels_first’: data_shape C*H*W ‘channels_last’: data_shape H*W*C

Returns
list

contain object of humans,see Model.Human for detail information of Human object

hyperpose.Model.pose_proposal.utils.preprocess(annos, bbxs, model_hin, modeL_win, model_hout, model_wout, model_hnei, model_wnei, parts, limbs, data_format='channels_first')

preprocess function of poseproposal class models

take keypoints annotations, bounding boxs annotatiosn, model input height and width, model limbs neighbor area height, model limbs neighbor area width and dataset type return the constructed targets of delta,tx,ty,tw,th,te,te_mask

Parameters
arg1list

a list of keypoint annotations, each annotation is a list of keypoints that belongs to a person, each keypoint follows the format (x,y), and x<0 or y<0 if the keypoint is not visible or not annotated. the annotations must from a known dataset_type, other wise the keypoint and limbs order will not be correct.

arg2list

a list of bounding box annotations, each bounding box is of format [x,y,w,h]

arg3Int

height of the model input

arg4Int

width of the model input

arg5Int

height of the model output

arg6Int

width of the model output

arg7Int

model limbs neighbor area height, determine the neighbor area to macth limbs, see pose propsal paper for detail information

arg8Int

model limbs neighbor area width, determine the neighbor area to macth limbs, see pose propsal paper for detail information

arg9Config.DATA

a enum value of enum class Config.DATA dataset_type where the input annotation list from, because to generate correct conf_map and paf_map, the order of keypoints and limbs should be awared.

arg10string

data format speficied for channel order available input: ‘channels_first’: data_shape C*H*W ‘channels_last’: data_shape H*W*C

Returns
list

including 7 elements delta: keypoint confidence feature map, shape [C,H,W](channels_first) or [H,W,C](channels_last) tx: keypoints bbx center x coordinates, divided by gridsize, shape [C,H,W](channels_first) or [H,W,C](channels_last) ty: keypoints bbx center y coordinates, divided by gridsize, shape [C,H,W](channels_first) or [H,W,C](channels_last) tw: keypoints bbxs width w, divided by image width, shape [C,H,W](channels_first) or [H,W,C](channels_last) th: keypoints bbxs height h, divided by image width, shape [C,H,W](channels_first) or [H,W,C](channels_last) te: edge confidence feature map, shape [C,H,W,Hnei,Wnei](channels_first) or [H,W,Hnei,Wnei,C](channels_last) te_mask: mask of edge confidence feature map, used for loss caculation, shape [C,H,W,Hnei,Wnei](channels_first) or [H,W,Hnei,Wnei,C](channels_last)

hyperpose.Model.pose_proposal.utils.restore_coor(x, y, w, h, win, hin, wout, hout, data_format='channels_first')
hyperpose.Model.pose_proposal.utils.visualize(img, predicts, parts, limbs, save_name='bbxs', save_dir='./save_dir/vis_dir', data_format='channels_first', save_tofile=True)

visualize function of poseproposal class models

take model predicted feature maps of delta,tx,ty,tw,th,te,te_mask, output visualized image. the image will be saved at ‘save_dir’/’save_name’_visualize.png

Parameters
arg1numpy array

image

arg2list

a list of model output: delta,tx,ty,tw,th,te,te_mask delta: keypoint confidence feature map, shape [C,H,W](channels_first) or [H,W,C](channels_last) tx: keypoints bbx center x coordinates, divided by gridsize, shape [C,H,W](channels_first) or [H,W,C](channels_last) ty: keypoints bbx center y coordinates, divided by gridsize, shape [C,H,W](channels_first) or [H,W,C](channels_last) tw: keypoints bbxs width w, divided by image width, shape [C,H,W](channels_first) or [H,W,C](channels_last) th: keypoints bbxs height h, divided by image width, shape [C,H,W](channels_first) or [H,W,C](channels_last) te: edge confidence feature map, shape [C,H,W,Hnei,Wnei](channels_first) or [H,W,Hnei,Wnei,C](channels_last) te_mask: mask of edge confidence feature map, used for loss caculation, shape [C,H,W,Hnei,Wnei](channels_first) or [H,W,Hnei,Wnei,C](channels_last)

arg3: Config.DATA

a enum value of enum class Config.DATA dataset_type where the input annotation list from

arg4String

specify output image name to distinguish.

arg5String

specify which directory to save the visualized image.

arg6string

data format speficied for channel order available input: ‘channels_first’: data_shape C*H*W ‘channels_last’: data_shape H*W*C

Returns
None
Module contents
Submodules
hyperpose.Model.backbones module
class hyperpose.Model.backbones.MobilenetV1_backbone(scale_size=8, data_format='channels_last', pretraining=False)

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_loss

conv_block

count_params

print_params

separable_conv_block
cal_loss(self, label, predict)
conv_block(self, n_filter=32, in_channels=3, filter_size=3, 3, strides=1, 1, padding='SAME', name='conv_block')
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
separable_conv_block(self, n_filter=32, in_channels=3, filter_size=3, 3, strides=1, 1, name='spconv_block')
class hyperpose.Model.backbones.MobilenetV2_backbone(scale_size=8, data_format='channels_last', pretraining=False)

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

InvertedResidual([n_filter, in_channels, …])

Attributes

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_loss

count_params

print_params
class InvertedResidual(n_filter=128, in_channels=128, strides=1, 1, exp_ratio=6, data_format='channels_first', name='block')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
cal_loss(self, label, predict)
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class hyperpose.Model.backbones.Resnet18_backbone(n_filter=512, in_channels=3, scale_size=8, data_format='channels_first', pretraining=False)

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

Res_block(n_filter, in_channels[, strides, …])

Attributes

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
class Res_block(n_filter, in_channels, strides=1, 1, is_down_sample=False, data_format='channels_first', name='res_block')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class hyperpose.Model.backbones.Resnet50_backbone(in_channels=3, n_filter=64, scale_size=8, data_format='channels_first', pretraining=False, use_pool=True)

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

Basic_block([in_channels, n_filter, …])

Attributes

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_loss

count_params

print_params
class Basic_block(in_channels=64, n_filter=64, strides=1, 1, data_format='channels_first', name='basic_block')

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

count_params

print_params
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
cal_loss(self, label, predict)
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class hyperpose.Model.backbones.vgg16_backbone(in_channels=3, scale_size=8, data_format='channels_first', pretraining=False)

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_loss

conv_block

count_params

print_params
cal_loss(self, label, predict)
conv_block(self, n_filter=32, in_channels=3, filter_size=3, 3, strides=1, 1, act=None, padding='SAME', name='block')
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class hyperpose.Model.backbones.vgg19_backbone(in_channels=3, scale_size=8, data_format='channels_first', pretraining=False)

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_loss

conv_block

count_params

print_params
cal_loss(self, label, predict)
conv_block(self, n_filter=32, in_channels=3, filter_size=3, 3, strides=1, 1, act=None, padding='SAME', name='conv_default')
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
class hyperpose.Model.backbones.vggtiny_backbone(in_channels=3, scale_size=8, data_format='channels_first', pretraining=False)

Bases: tensorlayer.models.core.Model

Attributes
all_drop
all_layers

Return all layers of this network in a list.

all_params
all_weights

Return all weights of this network in a list.

config
inputs
n_weights

Return the number of weights (parameters) in this network.

nontrainable_weights

Return nontrainable weights of this network in a list.

outputs
trainable_weights

Return trainable weights of this network in a list.

Methods

__call__(self, inputs[, is_train])

Forward input tensors through this network by calling.

as_layer(self)

Return this network as a ModelLayer so that it can be integrated into another Model.

eval(self)

Set this network in evaluation mode.

forward(self, x)

Network forwarding given input tensors

get_layer(self[, name, index])

Network forwarding given input tensors

infer(self)

Set this network in evaluation mode.

load(filepath[, load_weights])

Load model from a given file, which should be previously saved by Model.save().

load_weights(self, filepath[, format, …])

Load model weights from a given file, which should be previously saved by self.save_weights().

print_all_layers(self)

release_memory(self)

WARNING: This function should be called with great caution.

save(self, filepath[, save_weights, …])

Save model into a given file.

save_weights(self, filepath[, format])

Input filepath, save model weights into a file of given format.

test(self)

Set this network in evaluation mode.

train(self)

Set this network in training mode.

cal_loss

conv_block

count_params

print_params
cal_loss(self, label, predict)
conv_block(self, n_filter=32, in_channels=3, filter_size=3, 3, strides=1, 1, act=tensorflow.nn.relu, padding='SAME', name='block')
forward(self, x)

Network forwarding given input tensors

Parameters
inputsTensor or list of Tensors

input tensor(s)

kwargs :

For other keyword-only arguments.

Returns
output tensor(s)Tensor or list of Tensor(s)
hyperpose.Model.common module
class hyperpose.Model.common.MPIIPart

Bases: enum.Enum

An enumeration.

Head = 13
LAnkle = 5
LElbow = 10
LHip = 3
LKnee = 4
LShoulder = 9
LWrist = 11
Neck = 12
RAnkle = 0
RElbow = 7
RHip = 2
RKnee = 1
RShoulder = 8
RWrist = 6
static from_coco(human)
class hyperpose.Model.common.Profiler

Bases: object

Methods

__call__(self, name, duration)

Call self as a function.

report
report(self)
hyperpose.Model.common.draw_humans(npimg, humans)
hyperpose.Model.common.get_op(graph, name)
hyperpose.Model.common.get_optim(optim_type)
hyperpose.Model.common.get_sample_images(w, h)
hyperpose.Model.common.init_log(config)
hyperpose.Model.common.load_graph(model_file)

Load a freezed graph from file.

hyperpose.Model.common.log(msg)
hyperpose.Model.common.measure(f, name=None)
hyperpose.Model.common.pad_image(img, stride, pad_value=0.0)
hyperpose.Model.common.plot_humans(image, heatMat, pafMat, humans, name)
hyperpose.Model.common.read_imgfile(path, width, height, data_format='channels_last')

Read image file and resize to network input size.

hyperpose.Model.common.regulize_loss(target_model, weight_decay_factor)
hyperpose.Model.common.rename_tensor(x, name)
hyperpose.Model.common.tf_repeat(tensor, repeats)

Args:

input: A Tensor. 1-D or higher. repeats: A list. Number of repeat for each dimension, length must be the same as the number of dimensions in input

Returns:

A Tensor. Has the same type as input. Has the shape of tensor.shape * repeats

hyperpose.Model.human module
class hyperpose.Model.human.BodyPart(parts, u_idx, part_idx, x, y, score, w=- 1, h=- 1)

Bases: object

part_idx : part index(eg. 0 for nose) x, y: coordinate of body part score : confidence score

Methods

get_part_name

get_x

get_y
get_part_name(self)
get_x(self)
get_y(self)
class hyperpose.Model.human.Human(parts, limbs, colors)

Bases: object

body_parts: list of BodyPart

Methods

draw_human

get_area

get_bbx

get_global_id

get_partnum

get_score

print

scale
draw_human(self, img)
get_area(self)
get_bbx(self)
get_global_id(self)
get_partnum(self)
get_score(self)
print(self)
scale(self, scale_w, scale_h)
Module contents
hyperpose.Model.get_evaluate(config)

get evaluate pipeline based on config object

construct evaluate pipeline based on the chosen model_type and dataset_type, the evaluation metric fellows the official metrics of the chosen dataset.

the returned evaluate pipeline can be easily used by evaluate(model,dataset), where model is obtained by Model.get_model(), dataset is obtained by Dataset.get_dataset()

the evaluate pipeline will: 1.loading newest model at path ./save_dir/model_name/model_dir/newest_model.npz 2.perform inference and parsing over the chosen evaluate dataset 3.visualize model output in evaluation in directory ./save_dir/model_name/eval_vis_dir 4.output model metrics by calling dataset.official_eval()

Parameters
arg1config object

the config object return by Config.get_config() function, which includes all the configuration information.

Returns
function

a evaluate pipeline function which takes model and dataset as input, and output model metrics

hyperpose.Model.get_model(config)

get model based on config object

construct and return a model based on the configured model_type and model_backbone. each preset model architecture has a default backbone, replace it with chosen common model_backbones allow user to change model computation complexity to adapt to application scene.

Parameters
arg1config object

the config object return by Config.get_config() function, which includes all the configuration information.

Returns
tensorlayer.models.MODEL

a model object inherited from tensorlayer.models.MODEL class, has configured model architecture and chosen model backbone. can be user defined architecture by using Config.set_model_architecture() function.

hyperpose.Model.get_postprocess(model_type)

get postprocess function based model_type

get the postprocess function of the specified kind of model to help user construct thier own evaluate pipeline rather than using the integrated train or evaluate pipeline directly when in need

the postprocess function is able to parse the model output feature map and output parsed human objects of Human class, which contains all dectected keypoints.

Parameters
arg1Config.MODEL

a enum value of enum class Config.MODEL

Returns
function

a postprocess function of the specified kind of model

hyperpose.Model.get_preprocess(model_type)

get preprocess function based model_type

get the preprocess function of the specified kind of model to help user construct thier own train and evaluate pipeline rather than using the integrated train or evaluate pipeline directly when in need.

the preprocess function is able to convert the image and annotation to the model output format for training or evaluation.

Parameters
arg1Config.MODEL

a enum value of enum class Config.MODEL

Returns
function

a preprocess function of the specified kind of model

hyperpose.Model.get_pretrain(config)
hyperpose.Model.get_train(config)

get train pipeline based on config object

construct train pipeline based on the chosen model_type and dataset_type, default is single train pipeline performed on single GPU, can be parallel train pipeline use function Config.set_train_type()

the returned train pipeline can be easily used by train(model,dataset), where model is obtained by Model.get_model(), dataset is obtained by Dataset.get_dataset()

the train pipeline will: 1.store and restore ckpt in directory ./save_dir/model_name/model_dir 2.log loss information in directory ./save_dir/model_name/log.txt 3.visualize model output periodly during training in directory ./save_dir/model_name/train_vis_dir the newest model is at path ./save_dir/model_name/model_dir/newest_model.npz

Parameters
arg1config object

the config object return by Config.get_config() function, which includes all the configuration information.

Returns
function

a train pipeline function which takes model and dataset as input, can be either single train or parallel train pipeline.

hyperpose.Model.get_visualize(model_type)

get visualize function based model_type

get the visualize function of the specified kind of model to help user construct thier own evaluate pipeline rather than using the integrated train or evaluate pipeline directly when in need

the visualize function is able to visualize model’s output feature map, which is helpful for training and evaluation analysis.

Parameters
arg1Config.MODEL

a enum value of enum class Config.MODEL

Returns
function

a visualize function of the specified kind of model

Module contents

Performance and Supports

Supports

Prediction Library

Supported Language

  • C++

Supported DNN Engine & Model Format

  • TensorRT

    • ONNX

    • Uff

    • CUDA Engine Protobuf

Supported Post-Processing Methods

  • Part Association Field(PAF)

  • Pose Proposal Networks

Released Prediction Models

We released the models on Google Drive. .onnx and .uff files are for inference.

Performance of Prediction Library

Result

Method Backbone Size Network Resolution Operator API / FPS Stream API / FPS Other Framework / FPS Batch Size
OpenPose COCO 209.3MB 656 x 368 19.78 27.32 8 (OpenPose) 8
Tiny VGG + PAF 34.7 MB 384 x 256 66.62 124.925 / 8
MobileNet + PAF 17.9 MB 432 x 368 50.89 84.32 / 8
ResNet50 + PAF 45.0 MB 432 x 368 50.89 84.32 8.5 (TF-Pose) 8
ResNet18 + Pose Proposal 50.3 MB 384 x 384 212.42 349.17 / 64

Environment: System@Ubuntu18.04, GPU@1070Ti, CPU@i7(12 logic cores).

Tested Video Source: Crazy Updown Funk(resolution@640x360, frame_count@7458, source@YouTube)

OpenPose performance is not tested with batch processing as it seems not to be implemented. (see here)

Suggestions

  • PAF post processing is slow. Batch processing will not accelerate PAF and will bring little improvement in the speed.

  • And Pose Proposal post processing is fast(over 8k FPS in single core). So any optimization(e.g. batch processing) in DNN inference will be remarkable for the throughput of the pipeline. For example, using batch size 8 we got 164 FPS, using batch size 64 we got 349 FPS, and using batch size 128 we got 383 FPS.

Insights

Overview

Why HyperPose

HyperPose provides:

  • Flexible training

    • Well abstracted APIs(Python) to help you manage the pose estimation pipeline quickly and directly

    • Dataset(COCO, MPII)

    • Pose Estimation Methods

      • Backbones: ResNet, VGG(Tiny/Normal/Thin), Pose Proposal Network.

    • Post-Processing: Part Association Field(PAF), Pose Proposal Networks.

  • Fast Prediction

    • Rich operator APIs for you to do fast DNN inference and post-processing.

    • 2 API styles:

      • Operator API(Imperative): HyperPose provides basic operators to do DNN inference and post processing.

      • Stream API(Declarative): HyperPose provides a streaming processing runtime scheduler where users only need to specify the engine, post-processing methods and input/output streams.

    • Model format supports:

      • Uff.

      • ONNX.

      • Cuda Engine Protobuf.

    • Good performance. (see here)

Training Library Design

Prediction Library Design

HyperPose Prediction Pipeline

HyperPose supports prediction pipelines described in the image blow. (Mainly for bottom-up approaches)

_images/PredArch.png

Operator API & Stream API

Operator API provides basic operators for users to manipulate the pipeline. And Stream API is based on Operator API and makes higher level scheduling on it.

Minimum Example For Operator API

To apply pose estimation to a video using Operator API:

#include <hyperpose/hyperpose.hpp>

int main() {
    using namespace hyperpose;

    const cv::Size network_resolution{384, 256};
    const dnn::uff uff_model{ "../data/models/TinyVGG-V1-HW=256x384.uff", "image", {"outputs/conf", "outputs/paf"} };

    // * Input video.
    auto capture = cv::VideoCapture("../data/media/video.avi");

    // * Output video.
    auto writer = cv::VideoWriter(
        "output.avi", capture.get(cv::CAP_PROP_FOURCC), capture.get(cv::CAP_PROP_FPS), network_resolution);

    // * Create TensorRT engine.
    dnn::tensorrt engine(uff_model, network_resolution);

    // * post-processing: Using paf.
    parser::paf parser{};

    while (capture.isOpened()) {
        std::vector<cv::Mat> batch;
        for (int i = 0; i < engine.max_batch_size(); ++i) {
            cv::Mat mat;
            capture >> mat;
            if (mat.empty())
                break;
            batch.push_back(mat);
        }

        if (batch.empty())
            break;

        // * TensorRT Inference.
        auto feature_map_packets = engine.inference(batch);

        // * Paf.
        std::vector<std::vector<human_t>> pose_vectors;
        pose_vectors.reserve(feature_map_packets.size());
        for (auto&& packet : feature_map_packets)
            pose_vectors.push_back(parser.process(packet[0], packet[1]));

        // * Visualization
        for (size_t i = 0; i < batch.size(); ++i) {
            cv::resize(batch[i], batch[i], network_resolution);
            for (auto&& pose : pose_vectors[i])
                draw_human(batch[i], pose);
            writer << batch[i];
        }
    }
}
Minimum Example For Stream API

To apply pose estimation to a video using Stream API:

#include <hyperpose/hyperpose.hpp>

int main() {
    using namespace hyperpose;

    const cv::Size network_resolution{384, 256};
    const dnn::uff uff_model{ "../data/models/TinyVGG-V1-HW=256x384.uff", "image", {"outputs/conf", "outputs/paf"} };

    // * Input video.
    auto capture = cv::VideoCapture("../data/media/video.avi");

    // * Output video.
    auto writer = cv::VideoWriter(
        "output.avi", capture.get(cv::CAP_PROP_FOURCC), capture.get(cv::CAP_PROP_FPS), network_resolution);

    // * Create TensorRT engine.
    dnn::tensorrt engine(uff_model, network_resolution);

    // * post-processing: Using paf.
    parser::paf parser{};

    // * Create stream
    auto stream = make_stream(engine, parser);

    // * Connect input stream.
    stream.async() << capture;

    // * Connect ouput stream and wait.
    stream.sync() >> writer;
}

Using the Stream API, it is much faster and with less codes!

Stream Processing in Stream API

  • Every worker is a thread.

  • Every worker communicates via a FIFO queue.

  • The DNN inference worker will do greedy batching for the inputs.

_images/StreamArch.png

Frequently Asked Questions(FAQs)

Frequently Asked Questions

Installation

No C++17 Compiler(Linux)?

  • Using apt as your package manager?

    • Install from ppa.

    • Helpful link: LINK.

  • Otherwise

    • Build a C++17 compiler from source.

Build without examples/tests?
cmake .. -DBUILD_EXAMPLES=OFF -DBUILD_TESTS=OFF
Build OpenCV from source?

Refer to here.

Network problem when installing the test models/data from the command line?

Download them manually:

Training

Prediction

TensorRT Error?

  • See the tensorrt.log. (it contains more informations about logging and is located in where you execute the binary)

  • You may meet ERROR: Tensor image cannot be both input and output when using the TinyVGG-V1-HW=256x384.uff model. And just ignore it.

Performance?

  • Usually the 1st try of execution(cold start) on small amount of data tends to be slow. You can use a longer video/more images to test the performance(or run it more than once).

  • The performance is mainly related to the followings(you can customize the followings):

    • The complexity of model(not only FLOPS but also parameter numbers): smaller is usually better.

    • The model network resolution(alse see here): smaller is better.

    • Batch size: bigger is faster(higher throughput). (For details, you can refer to Shen’s dissertation)

    • The input / output size(this mainly involves in the speed of cv::resize): smaller is better.

    • The upsampling factor of the feature map when doing post processing: smaller is better. (By default the PAF parser will upsample the feature map by 4x. We did this according to the Lightweight-OpenPose paper.)

  • Use better hardware(Good CPUs can make the post-processing faster!).

  • Use SIMD instructions of your CPU. (Compile OpenCV from source and enable the instruction sets in cmake configuration)