You have probably heard that starting with Chrome 80 in February, the behavior of cookies will change. This is a breaking change and effects every single web application on the internet.
Microsoft has patched their supported platforms (ASP.NET, Katana 4 and ASP.NET Core) and provides instructions how to deal with the changes until the web has stabilized again.
IdentityServer3 runs on Katana 3, which is not supported by Microsoft anymore. We announced the end of free maintenance for IdentityServer3 already end of 2017 and started offering a security maintenance program mid 2019.
We are aware that still many companies out there use IdentityServer3 – which means their applications will break in the next months because of the changes mentioned above.
There is no easy fix for this, since the underlying platform itself does not support the new cookie semantics. We took some engineering effort to update the old IdentityServer3 code-base to support the 2020 SameSite behavior, and make this available to our IdentityServer3 security maintenance customers. If you are not already in that program, please contact us immediately.
leastprivilege.com 
Does this affect IdentityServer3.AccessTokenValidation package too?
no.
Does this only affect IdentityServer3 and not IdentityServer4?
It is really more a platform issue than an IdentityServer issue. Make sure you are on the latest supported .NET Core and follow Microsoft’s instructions.
Sure, they use different platforms.
But, did one have the Samesite-Problems switching to IdentityServer4?
I can see how this would affect applications that use an iframe for token refresh.
Would any other scenarios be affected?
Any scenario where the client and the server a cross-domain and a browser is involved, will be affected.
If I enable new samesite policies from the chrome:flags and after that applications are working fine is everything ok?
https://knowyourtoolset.com/2020/01/testing-for-the-new-samesite-cookie-handling-changes/