VideoAcceleration

From Linux Rockchip
Jump to: navigation, search

This page describes past and current findings and developments on Video Acceleration, specifically VPU support

Feel free to add any interesting informations such as links to other devices featuring Hantro Decoders and/or Encoders, links to their sources, etc

This is work in progress (take a look at new ways of research section)

Contents

Overview

Rockchip RK3X VPU (mainly RK3066 and RK3188) would be composed of:

Layers

It is interesting to take a look at graphic at http://www.freescale.com/files/training_pdf/VFTF09_AC105.pdf (page 6/7)

VPU kernel driver

None of them have been merged into upstream Linux kernel, you need a kernel from Rockchip. Usually located (when present) under:

  • arch/arm/plat-rk/vpu_service.c (when compiled it will be vpu_service.ko), at linux kernel 3.10
  • drivers/video/rockchip/vcodec/ (when compiled it will be vpu_service.ko), at linux kernel 4.4, but the vpu_service would be deprecated.
  • drivers/media/platform/rockchip-vpu/ (when compiled it will be rockchip_vpu.ko) , at linux kernel 4.4. A V4L2 version of VPU driver, but you can't use the standard steps of V4L2 m2m driver to make it work.

You have better disable the iommus phandle of vpu in RK3288 platform with kernel 4.4 from rockchip, it has been reported with some issue in it

vpu service

Output with debug options enabled (dmesg) of vpu_service:

VPU_SERVICE: vpu_service_init: baseaddr = 0x10104000 vdpu irq = 42 vepu irq = 41
VPU_SERVICE: vpu_get_clk: failed on clk_get pd_video
VPU_SERVICE: vpu_get_clk: failed on clk_get aclk_ddr_vepu
VPU_SERVICE: vpu_get_clk: failed on clk_get hclk_cpu_vcodec
vpu: power on
VPU_SERVICE: vpu_service_check_hw: checking hw id 4831
vpu: power off...done
VPU_SERVICE: vpu_service_init: init success

It creates:

  • Device node: /dev/vpu_service
  • Procfs node: /proc_vpu_service
    • output of 'cat /proc/vpu_service'

vpu v4l2

It would act likes the other Video4Linux2 device, It creates:

  • Two video nodes in /dev likes: /dev/video0, /dev/video1

Some remarks from Herman Chen (Rockchip):

/dev/vpu_mem is used for old 2.3 which do not have ion memory in kernel. On 4.0 and 4.1 we are using /dev/ion to allocate linear phisical memory used by video hardware decoder (VPU).[1]

libvpu

Can be found here

Compiles more or less easily once you realise that main() function in vpu_drv.c is unrunnable unless sockets are adapted to UNIX style (see next)

vpu_server client socket references

  • Only triggered if vpu_service kernel driver is not present
  • Might be a good idea (I have done so (tonikasch)) to #if 0 ... #endif the whole section regarding vpu_server if you are going to stick to vpu_service kernel module
  • If you are not going to stick to vpu_service kernel module, take in mind that a /dev/vpu device is needed
    • (Backporting rk29 vpu kernel driver?)
    • Changes are needed as "vpu_server" client socket needs to attach to something.
      • In Android, this seems to be /system/bin/mediaserver socket (guess made as per this Android init.rc but in linux you must find something else to attach to or even change the model.

vpu shared libraries

For the time being these are Android-only libraries so only armel

Consisting of:

  • libOMX_Core.so
  • libomxvpu_enc.so
  • libomxvpu.so
  • librk_demux.so
  • librk_on2.so

OpenMAX IL

http://omxil.sourceforge.net/


(2014-03-21) Efforts so far: Trying to register libomxvpu.so and libomxvpu_enc.so as openmax-bellagio components, with no success

  • should not work as those libraries are coded and built for Android

Userspace library and application

The rockchip has released a VA-API driver for RK3288 platform, The Gstreamer could use the VA-API to decode video.

VA-API

you could found it on Github, only the H.264 decoder is supported now.

Gstreamer

Gstreamer-VAAPI offer a way to access VPU through the VA-API backend. But you had better to use the latest Gstreamer VAAPI plugins, or the output format of Rockchip VA-API driver is NV12, which is not supported till recently.

new ways of research

kernel drivers (interesting)

We have found hx170dec and hx280enc (this one being supposedly valid for the encoder we have in rk3066 and rk3188)

Related commits:

Libraries (Android):

x170 kernel drivers

Composed of memalloc and hx170dec:

Proposed changes based on vpu_service.c:

  • hx170dec.c
#define HXDEC_LOGIC_MODULE0_BASE        0x10104200
#define VP_PB_INT_LT                    42
static const int DecHwId[] = { 0x8190, 0x8170, 0x9170, 0x9190, 0x1f52 };
  • memalloc.c
unsigned int alloc_method = MEMALLOC_MAX_OUTPUT;


x170 gstreamer plugin

Several Hantro(WebM)-based files:

  • basetype.h [*]
  • decapicommon.h
  • dwl.h
  • h264decapi.h
  • jpegdecapi.h
  • mp4decapi.h
  • mpeg2decapi.h
  • ppapi.h
  • vc1decapi.h

Seemed to be missing, but can be found at: ftp://www.at91.com/pub/demo/linux4sam_1.9/codec/ => on2-8170-libs-1.0.tar.gz

basetype.h (only) is also present in Herman Chen's Libstagefright at libstagefright/libvpu/common/

WebM Hardware (aka Hantro) [issue tracking] repositories

In reality "for issue tracking" and "only open for team members and partners"

Android Libraries

  • Libhybris: Some efforts done so far in LibHybris page
  • Ubuntu Touch: Some efforts done so far in UbuntuTouch page
    • It consists of a GNU/linux ubuntu system with a hybris-patched Android system running in a lxc container which provides the linux host of accelerated media

Documentation

(irrelevant... or not) Android documentation on Stagefright: https://source.android.com/devices/media.html

Libstagefright for Rockchip SoCs

Latest Libstagefright for Rockchip can be found here: https://github.com/HermanChen/Rockchip_4.2_release_libstagefright

As stated by Herman Chen from Rockchip Open Libvpu Project, part of it might be reused as linux library.

  • libstagefright/libvpu/common/* ==> Tweaking vpu.c and vpu_drv.c sources main() test routines can be run:
    • enabling vpu.c main() runs ok
    • enabling vpu_drv.c main() hangs, both with vpu_service.ko and without it loaded
  • libvpu/* ==> libomxvpu.so and libomxvpu_enc.so seem a component of OpenMAX IL


Other products featuring Hantro

Freescale

Freescale i.MX21

Freescale i.MX31

NuFront NS115

Header files have same structures and variables names as in Rockchip's vpu_service

Repository features lots of API header files for Hantro G1 and H1, and shared object libraries

Also OpenMax IL from Hantro

Marvell PXA

Hantro video decoder engine UIO driver from Marvell features same VPU HW decoder and encoder IDs as Rockchip's VPU (0x6731 and 0x4831)

Marvell Video Acceleration Device name is /dev/pxa-hantro

HiSilicon K3V2 (Hi3620)

STMicroelectronics SPEAr1340

Dual-core Cortex A9 embedded MPU for multimedia

Atmel AT91 Smart ARM Microcontrollers (AT91SAM)

SAM9M ARM926EJ-S Microprocessors are High-performance MPU with Hardware Video Decoder

SAM9M10

SAM9M11

SAMA5D4

Read also

Projects that are worth a look