[Work in progress] Adding an option --serve tcp:localhost:port

[Darkroll76]

Hi everyone,

Following this post: #1073 and suggested by @rom1v, I started to work on an option --serve. This option would allow forwarding the video stream to any clients connected. It could be a C# Unity App, Java App or C# UWP App or whatever.

The command would be --serve tcp:localhost:1234

It could be a great solution for people like me, who wants to display the video stream inside a desktop application without redeveloping and rebuilding the scrcpy client in the desired language.

I create this post to give some updates and request some help in C development.

Following these indications :

Basically, when you receive a packet, it is decoded + recorded (if necessary):

scrcpy/app/src/stream.c

Lines 81 to 97 in 3935660

	static bool
	process_frame(struct stream *stream, AVPacket *packet) {
	if (stream->decoder && !decoder_push(stream->decoder, packet)) {
	return false;
	}
	if (stream->recorder) {
	packet->dts = packet->pts;
	if (!recorder_push(stream->recorder, packet)) {
	LOGE("Could not send packet to recorder");
	return false;
	}
	}
	return true;
	}

We can, in addition, write it to a socket.

But in fact, it could even be written earlier (without even parsing it):

scrcpy/app/src/stream.c

Line 229 in 3935660

bool ok = stream_recv_packet(stream, &packet);

Originally posted by @rom1v in #1073 (comment)

So, we create another socket (for testing, we're using localhost and port 27015)
In stream.c -> run_stream(void *data)

//open another socket on port 27015
socket_t Listensocket;
Listensocket = net_listen(IPV4_LOCALHOST, 27015, 1);
if (Listensocket == INVALID_SOCKET) {
	LOGI("Listen Error");
	net_close(Listensocket);
	return false;
}

socket_t ClientSocket;
ClientSocket = net_accept(Listensocket);
if (ClientSocket == INVALID_SOCKET) {
	LOGI("Client Error");
	net_close(Listensocket);
	return false;
}

net_close(Listensocket);

// We must only pass complete frames to av_parser_parse2()!
// It's more complicated, but this allows to reduce the latency by 1 frame!
stream->parser->flags |= PARSER_FLAG_COMPLETE_FRAMES;

int cpt = 0;

for (;;) {
	AVPacket packet;
	bool ok = stream_recv_packet(stream, &packet);

	if (!ok) {
		// end of stream
		break;
	}

	ok = stream_push_packet(stream, &packet);

	cpt++;
	net_send(ClientSocket, packet.buf, packet.size);
	printf("Send : %d", cpt);
	
        av_packet_unref(&packet);
	if (!ok) {
		// cannot process packet (error already logged)
		break;
	}
}

We 're not sure that we are sending the right thing yet but this is our first step.
Everyone who wants to help is welcomed 😃 !

Thanks,

[Darkroll76]

Is it possible to send frames already decoded by FFmpeg instead of forwarding packet? If yes, any idea in which function I should take it and write it to a socket?

[rom1v]

(sorry, I don't have much time right now to review)

Is it possible to send frames already decoded by FFmpeg instead of forwarding packet?

That's a bad idea, the decoded frames are way bigger than the encoded ones. You want to transmit the encoded stream, and decode on the client.

¹ For example, in YUV 4:2:0 at 8 bit/color, each pixel takes 16 bits. In 1920×1080, that's 4Mb/frame. At 60fps, 237Mb/s.

[Darkroll76]

(sorry, I don't have much time right now to review)

Don't be sorry! You're very helpful with everyone so thank you 😉.

That's a bad idea, the decoded frames are way bigger than the encoded ones. You want to transmit the encoded stream, and decode on the client.

Ok haha, I asked because I'm having trouble decoding H264 stream on my client because I'm using C# with Unity 3D and, by this fact, it's more complicated to use FFmpeg library to decode since it's less documented and have some compatibility issues that's requiring additional development.

[rom1v]

Note that if you just want to test your implementation of the --serve option, you can just use VLC as client:

vlc --demux h264 tcp://localhost:1234 

(the latency will be higher because VLC bufferizes, but it should work)

The behavior of the --serve option should be ~ the same as:

# listen on localhost:1234 and stream the ouput of screenrecord
adb exec-out screenrecord --output-format h264 - | nc -l -p 1234

[dedosmedia]

@Darkroll76
I hope you make progress with this important feature!
I don't have so advanced C knowledge, so I won't be able to help you. Anyway, Good Luck!

[dedosmedia]

@Darkroll76 I am also working on this feature... https://github.com/dedosmedia/scrcpy/tree/h264
This is what I have till now, it's almost the same as you.

@rom1v
When I launch scrcpy, it will wait for a connection to port 27184 (the default, but can be overwritten by --serve PORT). Once that I launch:

vlc --demux h264 tcp://localhost:27184

scrcpy continues its process and launch SDL_Window, where I can see my phone screen, however, VLC does not play anything.

When I do the same, but using Putty (raw mode, port 27184). I can see that I receive a lot of garbage, much more when I handle my screen's phone.

Maybe we are not sending the right thing. Excuse but I don't understand anything about AVFrame, AVPacket, frame, pts, header... etc
I am using the code @Darkroll76 used for doing it:

ok = stream_push_packet(stream, &packet);
net_send(ClientSocket, packet.buf, packet.size);
av_packet_unref(&packet);

Maybe we are not sending the right thing... I know you are busy with other matters, however, if you have a few free minutes, please give us a hand to get this working. Thanks in advanced

[Darkroll76]

[UPDATE]

Hi,

I almost did it. @dedosmedia I changed to packet.data instead. I'm about to create a fork today.
For now, only TCP, "localhost" and the port are taken as parameters.

I gave the possibility the display the screen (or not) independently from --serve option. So, in order to only forward the stream, we should specify scrcpy --serve tcp:localhost:1234 --no-display.

@rom1v What about allowing recording and serve at the same time?

TODO:

• Need to convert the IP address as char *IP to uint32_t (Currently just checking if it's "localhost", then I set value to IPV4_LOCALHOST)
• [update] it's a Windows issue: Handle stopping the process (like doing Ctrl+C) => I handle when the client is lost, it will stop the serve, but when I stop the process by force, MSYS is crashing. I guess it's because It's not closing properly.
• Handle more than one client (accept() is called only one time for testing)

IDEAS:

• Change --serve to --forward-to or --forward-on could be more clear about what it's doing ?

[rom1v]

I changed to packet.data instead

Good catch :)

What about allowing recording and serve at the same time?

Yes, serve will not prevent recording.

(and it should be done from another thread, not to block the stream on blocking calls).

Allow also UDP

Nope, UDP does not guarantee packet ordering and does not retransmit lost packets, so the stream will be corrupted as soon as one of these events occur.

[Darkroll76]

[UPDATE]
Here is the fork and branch: https://github.com/Darkroll76/scrcpy/tree/serve

Yes, serve will not prevent recording.

Ok, I was wondering if it should be allowed or not.

Nope, UDP does not guarantee packet ordering and does not retransmit lost packets, so the stream will be corrupted as soon as one of these events occur.

So, we can remove tcp as parameter: --serve IP:PORT

[rom1v]

So, we can remove tcp as parameter: --serve IP:PORT

For now, we can. In theory, we could also serve over other protocols (http://, rtsp://…).

[npes87184]

Very interesting idea!

I simply viewed the code, there are several coding style needs to be refined.

For example,

• We use space not tab
• The left parentheses should the same line as if, while etc.

After feature completed, I suggest to fix coding style before sending PR.

Overall, I like this idea, we can build some application after supporting this feature. And it can be a temporary workaround before libscrcpy finishing.

[rom1v]

It's a great feature for streaming to OBS without opening a scrcpy window too 😉

[Darkroll76]

@npes87184 Are you talking about the code shown in my first post? Because it was just for testing.
Thanks for this quick review 😉

[dedosmedia]

It works now!
Although sometimes after some seconds the video on VLC becomes "corrupted", anyway the video on scrcpy window is always fine. I suppose it's due to VLC because of its buffering.

You both Rock guys!

[rom1v]

It works now!

👍

It's a great feature for streaming to OBS without opening a scrcpy window too 😉

Arf, it does not work: OBS supports opening a stream from VLC, but not adding extra parameters (--demux h264). 😞

[solnyshok]

hi, thanks a lot for your work. can I ask for a feature package this (tcp) h264 stream into a v4l-like video capture device which can then be used by any application like skype, zoom, obs, etc.

[rom1v]

@solnyshok Yes, I thought there was already a feature request for that, but I didn't find it.

Could you create one please?

[Maciejszuchta]

I'm really looking forward to this feature. Can't wait to use it on my website :D

[AVTurovskiy]

Ok haha, I asked because I'm having trouble decoding H264 stream on my client because I'm using C# with Unity 3D and, by this fact, it's more complicated to use FFmpeg library to decode since it's less documented and have some compatibility issues that's requiring additional development.

Hello

I am trying to implement a scrcpy in c# to extend some functionality in a familiar environment. But, there are some problems with video decoding. Have you found a good solution to decode and show raw h264 in c#?
I've tried https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client, but it doesn't seem stable enough.
Perhaps you could share some examples, or thoughts about this?

[cuntoulishifu]

Ok haha, I asked because I'm having trouble decoding H264 stream on my client because I'm using C# with Unity 3D and, by this fact, it's more complicated to use FFmpeg library to decode since it's less documented and have some compatibility issues that's requiring additional development.

Hello

I am trying to implement a scrcpy in c# to extend some functionality in a familiar environment. But, there are some problems with video decoding. Have you found a good solution to decode and show raw h264 in c#?
I've tried https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client, but it doesn't seem stable enough.
Perhaps you could share some examples, or thoughts about this?

I have the same problem as you, did you resolve this? I mean to decode and show raw h264 in c#.

[AVTurovskiy]

Ok haha, I asked because I'm having trouble decoding H264 stream on my client because I'm using C# with Unity 3D and, by this fact, it's more complicated to use FFmpeg library to decode since it's less documented and have some compatibility issues that's requiring additional development.

Hello
I am trying to implement a scrcpy in c# to extend some functionality in a familiar environment. But, there are some problems with video decoding. Have you found a good solution to decode and show raw h264 in c#?
I've tried https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client, but it doesn't seem stable enough.
Perhaps you could share some examples, or thoughts about this?

I have the same problem as you, did you resolve this? I mean to decode and show raw h264 in c#.

No. I am still using ffmpeg. But it sometimes crashes and so I have to reset the connection. All the wrapper libraries I've tried have similar problems and none have proven to be stable enough. Reducing the frame rate slightly improves the situation.
Maybe you need to understand the structure of the incoming H264 and write your own wrapper or decoder.
If you or someone else comes up with a solution, please let me know.

[JMLX42]

Hello,

Would this work help for #1939 (comment) ?

Regards,

[nguyenviettuan96]

Hi,
Could you tell me how to build from your source on mac ?
Thanks

[qaisbayabani]

https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client is 100% stable now and also without adb server and client updated on git

[qaisbayabani]

Ok haha, I asked because I'm having trouble decoding H264 stream on my client because I'm using C# with Unity 3D and, by this fact, it's more complicated to use FFmpeg library to decode since it's less documented and have some compatibility issues that's requiring additional development.

Hello

I am trying to implement a scrcpy in c# to extend some functionality in a familiar environment. But, there are some problems with video decoding. Have you found a good solution to decode and show raw h264 in c#? I've tried https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client, but it doesn't seem stable enough. Perhaps you could share some examples, or thoughts about this?

UPDATE now its Stable

[qaisbayabani]

yup