Brief introduction

Lindsdk is a product development tool set for video encoding, decoding and transmission based on embedded platform (ARM/RISC-V Linux/RTOS) developed by Zhuhai Lindenis Technology Co., Ltd(珠海六点智能科技有限公司).

LindSDK features

Hardware platform

At present, the supported hardware platforms are as follows.

LindSDK structure

The architecture diagram of lindsdk is as follows, which is divided into OS layer, dependencies (3rd Library) layer, framework layer and app layer from bottom to top.

lindsdk_structure

Directory structure

Top Structure

framework/av

avwrapper

These are APIs for the audio/video hardware device.

Header FilesInstruction
adev_wrapper.hAPIs for Audio input/output.
disp_wrapper.hAPIs for display layer controls, used for video output.
g2d_wrapper.hAPIs for graphic 2D controls, now support YUV copy/rotate.
palloc_wrapper.hAPIs for physical memory management.
vdec_wrapper.hAPIs for hardware video decoder.

component

These are APIs for the media process.

Header FilesInstruction
lindcomponent.hThe common structure define of LindSDK.
ffcodec.hAudio encode & decode APIs based ffmpeg.
ffdemux.hMedia parser APIs based ffmpeg.
ffsink.hMedia muxer APIs based ffmpeg.
livesink.hRTSP server APIs based Live555.
lindsink.hUnified media sink(stream/file) APIs,
It is the wrappers of ffsink & livesink & lindrtpsink.
lindsource.hUnified media source APIs, now only support lindrtpsource.
lindapm.hAPIs for Acoustic Echo Cancelling(AEC)/Noise Suppression(NS).
mididec.hAPIs for MIDI audio decode.

lindplayer

These are APIs for media playback.

Reference application in sample: lindplayer, lindesplayer, lindjpg

Header FilesInstruction
lindplayer.hMedia player APIs,
support common media file(mp4,ts...) and stream video(rtsp,hls,http...).
lindesplayer.hAPIs for H264/H265 element stream playback.
lindjpeg.hAPIs for jpg picture decode/playback.

lindnet

These are the wrappers for the most commonly used socket APIs.

Lindnet supports TCP/UDP/RDT protocols. RDT is as a basic communication protocol for 'lindtunnel'.

lindtunnel

'lindtunnel' is an easy LAN communication library for multi-devices.

It consists of three roles: the server, the devices and the clients.

It's original requirements are below:(LAN IPcamera)

lindsdk_player

Header FilesInstruction
lindtunnel.hThe main APIs of tunnel.
ldt_ioctrl_def.hA set of command define for IP cameras.
'sample_ldt_dev' & 'sample_ldt_cli' are demos of a simple IP camera.
ldt_error.hThe error code define.

mpp_source

These are the wrappers for the audio/video capturing & encoding.

Reference application in sample: lindcamera

Header FilesInstruction
mpp_camera.hAPIs for camera capture/preview.
APIs of video recorde/encode.
APIs of taking pitcute.
APIs of video OSD.
mpp_audio.hAPIs for video capture and encode.
mpp_thumb.hAPIs for video thumbnail retriever.

framework/base

These are basic APIs of lindsdk.

Header FilesInstruction
lindgpioAPIs for GPIO controls.
lindinputeventAPIs for the key event. Support key event of short press and long press.
lindirtxAPIs for NEC code encode by PWM and do infrared emission.
lindmixAPIs for audio mixer configuation.
lindmodbusAPIs for RS485-modbus.
lindmotorAPIs for the motion control of motors.
lindpcmdspAPIs for a simple audio detector and the sound feature extraction.
lindrpcAPIs for a temporary Remote Procedure Call Protocol(RPC).
lindutilsAPIs for the most frequently used functions.
lindzcfgAPIs for a basic zero-config libriary.
miscAPIs for other useful functions.

sample

sample/lindcamera

Lindcamera is a reference implementation of camera application.

functionsdescription
Acquisition & PreviewSupport the opening and closing of the camera, and settings of resolution and frame rate.
RecordSupport single file recording and continuous loop recording, and support the setting of maximum recording file length or size.
Take pictureSupport capture, capture in video, and support photo resolution setting.
ThumbnailAutomatically generate JPG thumbnails after recording.
OSDSupport text and BMP watermark.
StreamingSupport RTSP/FFmpeg server streaming.

The command line parameters are described below:

sample/lindplayer

Lindplayer is a reference implementation of media player, which supports the playback of local audio, video and streaming media.

The command line parameters are described below:

When running with the "-zcfg" parameter, you do not need to enter the URL manually. However, please run the RTSP streaming media server that supports lindzcfg first, such as 'lindcamera' or 'sample_ mppsrc_ 2_ livesink'.

sample/lindesplayer

Lindesplayer is a H264/h265 element stream player.

The command line parameters are described below:

sample/lindjpg

Lindjpg is a JPG picture browser, which supports single sheet display and slide mode display.

The command line parameters are described below:

sample_midi

This is a MIDI music player。

The command line parameters are described below:

sample_adev

This is an example of audio acquisition and loopback ouput.

Run with the default parameters, it will collect PCM data from mic and then output it from line out.

The command line parameters are described below:

sample_ffdemux

This is an example of media file parsing.

The example can analyze common media files or streaming media such as MP4, TS and MP3 to obtain the element stream of audio and video.

The example can directly play an MP3 music file through parameter configuration.

The command line parameters are described below:

sample_lindzcfg

Lindzcfg is a set of simple zero configuration communication protocol defined by our company. It supports device registration and query in the LAN and hot plug.

It will generate two applications('sample_lindzcfg_cli' and 'sample_lindzcfg_srv').

sample_lindzcfg_srv is a service provider. When starting, it will register the service in the LAN (broadcast what server it can provide). Other devices (clients) can receive a corresponding message that a new service provider is ready and can communicate with it if necessary.

sample_lindzcfg_cli is a sample client. In addition to passively receiving the service registration message, it can also actively query whether there is a service provider in the LAN.

This pair of samples demonstrates how the server provider registers the service, and the client receives and queries the service.

The command line parameters of 'sample_lindzcfg_srv' are described below:

The command line parameters of 'sample_lindzcfg_cli' are described below:

sample_ldt_cli/sample_ldt_dev

This is a set of LAN IPcamera examples based on lindtunnel. 'sample_ldt_dev' is the reference implementation of IPcamera, 'sample_ldt_cli' is the reference implementation of remote monitoring app.

The command line parameters of 'sample_ldt_dev' are described below:

The command line parameters of 'sample_ldt_cli' are described below:

To control the device by the client, you need to run the 'ldt_server'!

sample_ldt_ioctrl

It will generate two applications('sample_ldt_ioctrl_dev' and 'sample_ldt_ioctrl_cli'), which are examples of custom instructions based on lindtunnel. It can also be used for cross process communication.

sample_lindnet

This is an example based on lindnet, which realizes one-to-one message sending. The client sends messages to the server, and the server loopback the received message to the client.

The command line parameters are described below:

  1. Support UDP, TCP and RDT transmission protocols. The server and client must adopt the same protocol;
  2. If TCP or RDT is adopted, the server side needs to be started first;

sample_lindsink_rtp/sample_lindsource_rtp

These two samples implement RTP streaming and receiving examples based on lindsink and lindsource:

sample_lindsink_rtp:"Camera acquisition -> video encoding -> RTP sending"

sample_lindsource_rtp:"RTP reception -> video decoding -> display output"

The command line parameters of 'sample_lindsink_rtp' are described below:

The command line parameters of 'sample_lindsource_rtp' are described below:

If RDT is adopted, the lindsource end needs to be started first;

sample_mppsrc_2_lindsink

编译后会生成sample_mppsrc_2_ffsink和sample_mppsrc_2_livesink两个应用。从camera采集音视频编码后,前者通过ffmpeg的muxer进行流媒体推送或者保存文件;后者通过live555生成rtsp服务器。

It will generate two applications('sample_mppsrc_2_ffsink' and 'sample_mppsrc_2_livesink').

sample_mppsrc_2_ffsink

This sample collects video frame from the camera and audio PCM from audio codec, and encoding them, then push the streaming media or save the file through the Muxer of ffmpeg.

The command line parameters are described below:

RTP streaming (non rtp_mpegts) using ffsink should pay attention to the following points:

  1. The receiver(player) needs to use SDP to play;
  2. Only audio or video can be pushed, and one port cannot simultaneously payload audio and video;
  3. The port number should be set to an even number;

After this sample runs, you can see the contents of SDP in the log as follows. Save them as a file with the suffix SDP. Then it can be played by players such as VLC.

For 'rtp_mpegts' streaming, you can play it by URL like this: rtp://ip:port/name

sample_mppsrc_2_livesink

This sample show collecting audio and video for encoding, and creatint a RTSP server based live555.

The command line parameters are described as follows:

After sample_mppsrc_2_livesink starts, you can play it by URL like this: rtsp://ip:port/name

How to use?

Build LindSDK

Package firmware

The packaging script is located in 'build/configs/config_XXX.sh'.

Through make_release_rootfs command copies the binary files generated by lindsdk into rootfs of the Linux SDK, and then packs them into firmware through the packaging tool of the Linux SDK.

Configuration files

lindsdk.ini

Lindsdk adopts ini format configuration file, and the path is:

lindsdk.ini defines sections such as device, lindsdk, debug, lindmpp, etc.

sectionsdescription
deviceDefine the equipment manufacturer, product name, model, version and other information
lindsdkDefine the writable user data partition, log level, log saving path and other information
debugDefine common debugging switches
lindmppDefine the channel of MPP
app_xxxCustomer section, which is used to extend the configuration information

section:debug

For example, the debug section defines the following debug switches:

The configuration can be obtained in the source code (refer to lindcamera) in the following ways:

It depends on the header file of 'osal_prop.h' and the library 'libldutils.so'

section:lindmpp

键(key)值(value)说明
version0x01000000version
camera00: disable, 1: enableEnable mark; When there are multiple video devices, create another set of configurations of camera1, camera2, ...
camera0_dev_main0: /dev/video0, 1: /dev/video1, ...Video device node ID of the main channel
camera0_dev_sub0: /dev/video0, 1: /dev/video1, ...Video device node ID of the subchannel; Video devices using ISPs have subchannels.
camera0_dev_isp0: ISP0, 1: ISP1Specify which ISP to use. There are two ISPs in V5 and one ISP in v536/v833; Set '-1' for video device without ISP (such as HDMI in)
camera0_tunnel_recordformat: [vi_dev,vi_chn,venc_chn,ai_dev,ai_chn,aenc_chn]Video record channel definition
camera0_tunnel_pictureformat: [vi_dev,vi_chn,venc_chn]Takeing picture channel definition
camera0_tunnel_vencNformat: [vi_dev,vi_chn,venc_chn]Encoded channel definition, up to 3 channels
camera0_tunnel_yuvNformat: [vi_dev,vi_chn]YUV data channel definition, up to 2 channels
camera0_tunnel_previewformat: [vi_dev,vi_chn,vo_dev,vo_chn]Preview channel definition
camera0_tunnel_osdNformat: [mod_id,dev_id,chn_id]OSD channel definition, up to 8 channels
  1. A camera can correspond to a set of the above configurations. When opening the camera, the corresponding parameters are adopted according to the specified camera ID;
  2. For the camera using the ISP, two video nodes can be generated, corresponding to the master device (camera0_dev_main) and the sub device (camera0_dev_sub), which can be used independently;
  3. For video devices that do not use ISP (for example, HDMI in), there will only be a main video device;
  4. Unnecessary channle can be set '-1';
  5. Please refer to the source code of lindcamera.

lindprop

The tools attached to lindsdk can view or modify lindsdk.ini configuration file, which can be used as follows:

lindmix.ini

This is the audio path configuration file. It can be accessed through interface(lindmix.h) to configure the audio input and output path.

Typical application scenarios

Player

lindsdk_player

Recorder

lindsdk_streamer_recorder

Streamer

IPCamera

Babymonitor