[27.x]: Make log reading more robust to errors

[cpuguy83]

Backport of #47983
The only one commit that needed some (minor) touchup was the journald changes.

---

• fixes / addresses Error grabbing logs: invalid character '\x00' looking for beginning of value docker/for-linux#140

Fundamentally this change (beyond the minor refactor and tracing spans) is to not treat errors reading from log files as fatal to providing logs to the client.

The need for this change is because often log files may be corrupted either due to old bugs in dockerd or for other reasons.
One specific thing we have seen seems to be caused by ext4 by default is not doing full journaling. On power loss or kernel panic this can cause the log data to be corrupted.
We see this mostly on IoT devices which are often not on battery backup or frequently hard cut from power.

Instead of erroring out a stream on errors from a particular file this just skips on to the next file.

I had made a previous attempt to be a little more clever and scrub the file to try and find a place where we can start parsing again but realized this can cause serious issues and is generally error prone.

This also takes care of a long-standing todo to handle log files lazily (or at least as lazily as possible) so we only end up decompressing files just-in-time.

Unfortunately we don't have a way to return warnings to the client that we had to skip files, however with the tracing changes at least it could be seen there and in the daemon logs.

When reading logs with the `jsonfile` or `local` log drivers, any errors while trying to read or parse underlying log files will cause the rest of the file to be skipped and move to the next log file (if one exists) rather than returning an error to the client and closing the stream.
The errors are viewable in the dockerd logs and exported to traces when tracing is configured.

When reading log files, compressed log files are now only decompressed when needed rather than decompressing all files before starting the log stream.

[thaJeztah]

LGTM

but could use a second pair of eyes

[thaJeztah]

Double checked if this interface change would affect logging plugins, but those implement a separate interface;

moby/daemon/logger/plugin.go

Lines 21 to 27 in da3cc1c

	// logPlugin defines the available functions that logging plugins must implement.
	type logPlugin interface {
	StartLogging(streamPath string, info Info) (err error)
	StopLogging(streamPath string) (err error)
	Capabilities() (cap Capability, err error)
	ReadLogs(info Info, config ReadConfig) (stream io.ReadCloser, err error)
	}

And there's an adaptor for those; see further up the diff 😄