Access tokens can come in two shapes: self-contained and reference.
Self-contained tokens are using a protected, time-limited data structure that contains metadata and claims to communicate the identity of the user or client over the wire. A popular format would be JSON Web Tokens (JWT). The recipient of a self-contained token can validate the token locally by checking the signature, expected issuer name and expected audience or scope.
Reference tokens (sometimes also called opaque tokens) on the other hand are just identifiers for a token stored on the token service. The token service stores the contents of the token in some data store, associates it with an infeasible-to-guess id and passes the id back to the client. The recipient then needs to open a back-channel to the token service, send the token to a validation endpoint, and if valid, retrieves the contents as the response.
A nice feature of reference tokens is that you have much more control over their lifetime. Where a self-contained token is hard to revoke before its expiration time, a reference token only lives as long as it exists in the STS data store. This allows for scenarios like
- revoking the token in an “emergency” case (lost phone, phishing attack etc.)
- invalidate tokens at user logout time or app uninstall
The downside of reference tokens is the needed back-channel communication from resource server to STS.
This might not be possible from a network point of view, and some people also have concerns about the extra round-trips and the load that gets put on the STS. The last two issues can be easily fixed using caching.
I presented this concept of the last years to many of my customers and the preferred architecture is becoming more and more like this:
If the token leaves the company infrastructure (e.g. to a browser or a mobile device), use reference tokens to be in complete control over lifetime. If the token is used internally only, self contained tokens are fine.
I am also mentioning (and demo-ing) reference tokens here starting minute 36.
IdentityServer3 supports the reference token concept since day one. You can set the access token type to either JWT or Reference per client, and the ITokenHandleStore interface takes care of persistence and revocation of reference tokens.
For validating reference tokens we provide a simple endpoint called the access token validation endpoint. This endpoint is e.g. used by our access token validation middleware, which is clever enough to distinguish between self-contained and reference tokens and does the validation either locally or using the endpoint. All of this is completely transparent to the API.
You simply specify the Authority (the base URL of IdentityServer) and the middleware will use that to pull the configuration (keys, issuer name etc) and construct the URL to the validation endpoint:
app.UseIdentityServerBearerTokenAuthentication(new IdentityServerBearerTokenAuthenticationOptions { Authority = "https://localhost:44333/core", RequiredScopes = new[] { "api1" } });
The middleware also supports caching and scope validation – check the docs here.
There are also multiple ways to revoke a token – e.g. through the application permission self-service page, the token revocation endpoint, by writing code against the ITokenHandle store interface (e.g. from your user service to clean up tokens during logout) or by simply deleting the token from your data store.
One thing our validation endpoint does not support is authentication – this is a non issue as long as you don’t want to use the reference token mechanism for confidentiality.
Token Introspection
Many token services have a reference token feature, and all of them, like us, invented their own proprietary validation endpoint. A couple of weeks ago RFC 7662 – “OAuth 2.0 Token Introspection”, which defines a standard protocol, has been published.
IdentityServer3 v2.2 as well as the token validation middleware starting with v2.3 have support for it.
The most important difference is that authentication is now required to access the introspection endpoint. Since this endpoint is not accessed by clients, but by resource servers, we hang the credential (aka secret) off the scope definition, e.g.:
var api1Scope = new Scope { Name = "api1", Type = ScopeType.Resource, ScopeSecrets = new List<Secret> { new Secret("secret".Sha256()) } };
For secret parsing and validation we use the same extensible mechanism that we use for client secrets. That means you can use shared secrets, client certificates or anything custom.
This also means that only scopes that are included in the access token can introspect the token. For any other scope, the token will simply look like invalid.
IdentityModel has a client library for the token introspection endpoint which pretty much self explanatory:
var client = new IntrospectionClient( "https://localhost:44333/core/connect/introspect", "api1", "secret"); var request = new IntrospectionRequest { Token = accessToken }; var result = client.SendAsync(request).Result; if (result.IsError) { Console.WriteLine(result.Error); } else { if (result.IsActive) { result.Claims.ToList().ForEach(c => Console.WriteLine("{0}: {1}", c.Item1, c.Item2)); } else { Console.WriteLine("token is not active"); } }
This client is also used in the validation middleware. Once we see the additional secret configuration, the middleware switches from the old validation endpoint to the new introspection endpoint:
app.UseIdentityServerBearerTokenAuthentication(new IdentityServerBearerTokenAuthenticationOptions { Authority = "https://localhost:44333/core", RequiredScopes = new[] { "api1" }, ClientId = "api1", ClientSecret = "secret" });
Once you switched to introspection, you can disable the old validation endpoint on the IdentityServerOptions:
var idsrvOptions = new IdentityServerOptions { Factory = factory, SigningCertificate = Cert.Load(), Endpoints = new EndpointOptions { EnableAccessTokenValidationEndpoint = false } };
Reference tokens are a real problem solver for many situations and the inclusion of the introspection spec and authentication makes this mechanism even more robust and a good basis for future features around access token lifetime management (spoiler alert).
HTH
leastprivilege.com 
Any reason the server couldn’t hold the reference Id and give the jwt out still? Wouldn’t fix the need for the back channel but could open up other possibilities.
Last time I dealt with a similar problem I looked into jwt trust trees, like SSL certs work. Didn’t seem like an amazing solution but it led me to come across Google Macaroons and hmac chaining which was very neat.
Then combine them with server side reference tokens for revocation. Although I guess you could still do that by signing the reference tokens.
Sure – you can implement it in any way it makes sense for you. I am just describing the way we implemented it.
The validation through the introspection endpoint has only one slight flaw: you only get the claims for that specific scope. So you most likely would have to define a scope which is some sort of superset over all scopes in your application, otherwise the validation middleware would only set the claims of that scope and you might end up asking yourself where your claims are because you did authenticate for multiple scopes… :)
I don’t follow – the scope claims are “scrubbed” to contain only the scope of the caller. This is per spec. All other claims are the same.
For example: securing an API with bearer token authentication while using a reference token.
If the token was requested for multiple scopes, then the authentication middleware will only get the claims for the scope which is used to access the introspection endopoint – all other claims for other scopes are not present in the current identity. This can be an issue if you need claims from multiple scopes (e.g. I partitioned some identity claims into separate scopes and some resource scopes have their own claims)
I still don’t follow. I only remove the claims of type scope that are not equal the scope name of the caller. Check the source code to verify that.
I do have claims which are specific to certain scopes. For example: automated services that authenticate with client credentials or a similar (custom flow) have claims that only make sense for these kinds of services/authentication type (e.g. additional out-of-band authentication).
For example:
Identity scopes and claims:
user context – full name, group Id to restrict access to data of their own group, “is service” etc, for both automated and regular users
automated user context – service id, type, “is service” etc.
Resource scopes:
user frontend – same claims as user context
admin frontend – only “is admin” claim (for now)
On the user frontend MVC/API I can use the user frontend scope and get all the claims.
On the backend WCF services however I can be accessed from both admin and user frontends and would need to have all claims from both user and admin frontend available there (because admin would be allowed to edit all data while user only his own or that of his group).
Would it make sense to use reference tokens to implement the api keys in an api gateway?
sure – why not.
OK cool. hi Dominick BTW :-)
So when’s your new Pluralsight course going to be read? :-)
hey ;)
Well – probably after aspnet5 is released…
OK look forward to it
Where are you getting the token from? Every time I send the contents of my authorization header, I get Active=false back from my IdentityServer. Thanks!
With contents you mean all of it? Only send the token – not the whole header.
So when I try something like this: (mainly stolen from the internet)
public override void OnActionExecuting(HttpActionContext actionContext)
{
var client = new IntrospectionClient(“https://myidp.com/identity/connect/introspect”, “myclientid”,
“mysecret”);
var request = new IntrospectionRequest
{
Token = actionContext.ControllerContext.Request.Headers.Authorization.Parameter
};
var result = client.SendAsync(request).Result;
if (result.IsError)
{
Console.WriteLine(result.Error);
}
else
{
if (result.IsActive)
{
result.Claims.ToList().ForEach(c => Console.WriteLine(“{0}: {1}”,
c.Item1, c.Item2));
}
else
{
Console.WriteLine(“token is not active”);
}
}
}
I always get inactive back. No difference when I manually trim the token to just the base64 (?) value. Any suggestions?
I don’t know. Check our sample – this is working
Just a quick note of thanks — this was an awesome write up that helped me a lot. Cheers!
In your video why do you recommend reference tokens for mobile apps and not SPAs?
Can’t remember – I recommend reference tokens whenever you need tight control over token lifetime.
the reference is can work,but the AccessTokenLifetime can not work right
i set the AccessTokenLifetime =604800,but the token will timeout in 2-4 hours.not 7 days.
i think the question is that,refrence token is a cache key in identityserver3,so when the identityserver3 server there is no customer visit for a long time,so the identityserver3 cache will Garbage collection,
so the refrence token is remove,resulting in AccessTokenLifetime is not work
Hey,
please report issues like that on the github issue tracker. thanks!
If you have a chain of microservice calls initiated from a SPA web app, is it acceptable (even a best practice?) to pass around the reference (or even a “self-contained”) token the first service receives to the other services downstream so that you persist the user’s identity (assuming downstream services need access to the user’s identity)?
Not really – this is called “poor man’s delegation” – if you need to pass around the identity of the user, why not transmit it as payload?
See http://docs.identityserver.io/en/release/topics/extension_grants.html#example-simple-delegation-using-an-extension-grant – they explain there how to not “unpack token into request payload” but rather how to delegate tokens from edge service to futher services.
We’re creating a client/resource server fronting a Java based application server which is wrappered to fit into the old ASP.NET pattern. We’re using a Java Servlet Filter using a 3rd party Java library to interwork with the Identity Server 4 as an IdP/OP/server. We’ve validated that basic authentication, refresh, etc. works for Hybrid Workflow as the Java Library and IS 4 conform to the OIDC spec.
We’re now investigating using either the Access token validation endpoint or Introspection endpoint to allow revoking access_tokens or refresh_tokens. The Java library doesn’t support reference access_tokens so we’re trying to understand our options in supporting revoking using JWT access_tokens.
One option is to use the (documented for IS3) Access token validation endpoint – but it’s not clear if this is supported for IS4 – and has the shortcoming of not validating refresh tokens. The saving grace is any attempt to use a revoked refresh token will be that the refresh token is invalid – or so we assume.
As the IS=3/4 assumes that any clients are new ASP.NET Core and Middleware, the examples of using the Introspection endpoint don’t translate to the detail of HTTP requests (or how to generate them from other than .NET).
In particular:
– it’s not clear if the generation of a scope with a secret is done on the IS4 server side or the client side. If on the server side, what .cs file would this be declared in?
var api1Scope = new Scope
{
Name = “api1”,
Type = ScopeType.Resource,
ScopeSecrets = new List
{
new Secret(“secret”.Sha256())
}
};
– it’s not clear what the http/https request looks like that can be implemented in Java that is implemented by the following .NET code to support authentication for the introspection endpoint.
app.UseIdentityServerBearerTokenAuthentication(new IdentityServerBearerTokenAuthenticationOptions
{
Authority = “https://localhost:44333/core”,
RequiredScopes = new[] { “api1” },
ClientId = “api1”,
ClientSecret = “secret”
});
Which leads to understanding the underlying raw interaction in Java (or any .NET implementation) of the following:
var client = new IntrospectionClient(
“https://localhost:44333/core/connect/introspect”,
“api1”,
“secret”);
var request = new IntrospectionRequest
{
Token = accessToken
};
var result = client.SendAsync(request).Result;
if (result.IsError)
{
Console.WriteLine(result.Error);
}
else
{
if (result.IsActive)
{
result.Claims.ToList().ForEach(c => Console.WriteLine(“{0}: {1}”,
c.Item1, c.Item2));
}
else
{
Console.WriteLine(“token is not active”);
}
}
IS4 supports introspection which is a standard protocol – check our docs
https://identityserver4.readthedocs.io/en/release/topics/reference_tokens.html
Great article! One question, since reference tokens can be revoked at any time is it ever necessary to use them with refresh tokens?
They are complementary concepts. The use of refresh tokens gives you “auto expiration” on the access token.
Hello Dominick,
Thank you for the great article and your works.
I can’t see much emphasize around the web that reference token also solves the problem to avoid including tons of claims inside the access token for complicated or multi-tenant applications. Reference token solves this problem right? And it allows runtime handling of claims whereever needed. I just wanted to make sure reference token solves this problem.
Additionally when you say caching solves the round trip problem, can we cache the retrieved claims from back-end channel, at client(web api) side? Is there any example or direction can you point for client side caching claims? Or is that; for every api access, the api communicates with Identity Server’s back-end channel, and only identity server can cache claims?
Well – the main benefit of reference tokens is immediate revocability. If you want to use it because your tokens include “tons of claims” – maybe something else is wrong.
You can cache wherever you want – typically at the “validator” and you need to decide how “fresh” your token needs to be.
Hi Dominick,
Thanks for the article!
Is it possible to recognize a reference token from a self contained token? When requesting a token there are no indicators stating the type of the token. The reference token seems to be a hash, so length could be an indicator, but it might not be impossible a self contained token matches the length as well. As the type is set on client level, is knowledge about the client the only way?
Thanks for your reply. Cheers!
Knowing that at least IdentityServer will not use dots in the reference token (as opposed to the 3 dots of a JWT) – that’s how I do it…
That makes sense… Thanks! :D