add google auth to the internal/staff only docs endpoints #3141

johnduffell · 2025-10-13T11:40:03Z

To merge into #3134 before it can go live

In the above PR #3134 I added a docs endpoint to serve up some internal information about discounts. We would like to make it less easy to access, and it would be a good idea in general if we could protect endpoints for staff only (including ones that may have side effects)

Google auth is an established system that would let us know that people are internal to the guardian, so it would be ideal to protect endpoints like the above.

This PR adds google auth to the public docs endpoint.

It does that by

integrating the docs lambda with a cognito pool authoriser, so all requests need a valid Basic authorisation header with an id_token supplied (access_token is also possible if we scope it)
adding a guardian internal google auth app in the google cloud console
putting the relevant google secrets in parameter store (manual)
adding a new staff-access handler containing
a stack containing cognito user pools and other associated resources
a lambda to be the "frontend" for the oauth flow via cognito and in turn google auth

See inline comments.

TODO before going live (also see longer term things in readme.md)

add repo config to PROD
add tests
add some runManual for local/CODE where appropriate
review for any security holes
make sure nothing can be forwarded inappropriately by the proxy, e.g. the auth cookie to an untrusted backend

# Conflicts: # cdk/lib/__snapshots__/discount-api.test.ts.snap # cdk/lib/cdk/policies.ts # cdk/lib/cdk/sr-api-lambda.ts # cdk/lib/discount-api.ts # update-stack.sh

johnduffell · 2025-10-23T13:44:27Z

cdk/lib/__snapshots__/discount-api.test.ts.snap

+                },
+                ":userpool/",
+                {
+                  "Fn::ImportValue": "UserPoolId-CODE",


the user pool id is exported from the staff-access stack - it must be deployed first for this to successfully deploy

johnduffell · 2025-10-23T13:46:24Z

cdk/lib/__snapshots__/discount-api.test.ts.snap

+        },
        "HttpMethod": "GET",
        "Integration": {
          "IntegrationHttpMethod": "POST",


consider to open and review the "Large diff" snapshot below (it's a brand new file)

johnduffell · 2025-10-23T13:47:48Z

cdk/lib/cdk/SrApiLambda.ts

+	/**
+	 * Set this to false to skip adding the main proxy endpoint
+	 */
+	proxy?: boolean;


if we add the base endpoint {proxy+} then it conflicts with the {app}/{proxy+} one I add manually later. TODO Perhaps I should just use that rather than trying to add an exception here

…ity-list-auth # Conflicts: # cdk/lib/cdk/SrApiLambda.ts # cdk/lib/cdk/sr-api-lambda.ts

johnduffell · 2025-10-23T19:25:22Z

cdk/lib/__snapshots__/discount-api.test.ts.snap

+          },
+        ],
+        "RestApiId": {
+          "Ref": "discountapilambdadiscountapiCODEE1E5C869",


the authoriser has to be in the same stack as the RestApi because it references the RestApi directly.

johnduffell · 2025-10-23T19:27:19Z

cdk/lib/discount-api.ts

+		const userPoolId = Fn.importValue(`UserPoolId-${stage}`);
+
+		const userPool = UserPool.fromUserPoolId(
+			this,
+			'ImportedUserPool',
+			userPoolId,
+		);
+
+		const cognitoAuthorizer = new CognitoUserPoolsAuthorizer(
+			this,
+			'CognitoAuthorizer',
+			{
+				cognitoUserPools: [userPool],
+			},
+		);


todo this can be hidden in the SrApiLambda

johnduffell · 2025-10-23T19:30:15Z

cdk/lib/staff-access.ts

+				// CFN doesn't support secret values https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references-ssm-secure-strings.html#template-parameters-dynamic-patterns-resources
+				// and using AwsCustomResource to read it also doesn't work as it seems to not find the built in lambda to implement it
+				clientSecretValue: SecretValue.unsafePlainText(googleOAuthClientSecret),


although this google auth is only protecting private information rather than secrets or PII, it would still be nice to protect the client secret according to prevailing security standards

johnduffell · 2025-10-23T21:32:24Z

cdk/lib/staff-access.ts

+				errorImpact:
+					'staff are getting errors when viewing docs served by our API layer',
+			},
+			isPublic: true,


the staff-access http server is public, as all auth is enforced by the downstream lambdas

johnduffell · 2025-10-23T21:34:54Z

modules/routing/src/router.ts

+		path?: z.Schema<TPath, ZodTypeDef, unknown>;
+		body?: z.Schema<TBody, ZodTypeDef, unknown>;


this allows us to use .transform on the schema, i.e. we can have a different output type to the input type.

…ity-list-auth

johnduffell · 2025-11-05T13:42:42Z

cdk/lib/__snapshots__/discount-api.test.ts.snap

+        },
+        "Type": "COGNITO_USER_POOLS",
+      },
+      "Type": "AWS::ApiGateway::Authorizer",


each secured lambda needs to have an authoriser that points to the user pool that's exported from the staff-access stack, this is so it can check bearer tokens
It has to be in this stack as it references the RestApi directly - only the User pool etc can be shared.

johnduffell · 2025-11-05T13:44:58Z

cdk/lib/__snapshots__/staff-access.test.ts.snap

+    "UserPoolId": {
+      "Export": {
+        "Name": "UserPoolId-CODE",
+      },
+      "Value": {
+        "Ref": "UserPool6BA7E5F2",
+      },


user pool is exported (separately for CODE and PROD) so api gateway can check the tokens

johnduffell · 2025-11-05T14:20:39Z

cdk/lib/cdk/SrAppConfigKey.ts

+export function buildAppConfigKey(scope: SrStack, configKey: string) {
+	return (
+		'/' + [scope.stage, scope.stack, scope.app].join('/') + '/' + configKey
+	);
+}


extract for reusability

johnduffell · 2025-11-05T14:21:20Z

cdk/lib/cdk/SrRestDomain.ts

+export function domainForStack(scope: SrStack, suffixProdDomain?: boolean) {
+	const isProd = scope.stage === 'PROD';
+	const cert = certForStack[scope.stack];
+	const domainName = `${scope.app}${isProd && !suffixProdDomain ? '' : '-' + scope.stage.toLowerCase()}.${cert.domainName}`;
+	return { cert, domainName };
+}


extract for reuse to calculate the allowed cognito redirect url
produces something like discount-api-code.support.guardianapis.com as well as the necessary cert info

johnduffell · 2025-11-05T14:23:56Z

cdk/lib/cdk/SrApiLambda.ts

 		}
 	}

-	addPublicPath(path: string) {


I replaced addPublicPath as we don't need it any more, but we can bring it back if we genuinely want some public paths on a private lambda in future.

johnduffell · 2025-11-10T14:04:08Z

handlers/staff-access/README.md

+## TODO (in later PRs)
+- consider to add a neater fastly domain https://staff-access.guardianapis.com/discount-api/docs
+- some kind of index so we can find all the endpoints
+- make the google client secret a securestring if possible (not easy as it's needed in the CFN)


I tried for a while to get it working, it seems like Aws Resource type should be able to do it, but when synthing the cdk, it produces something that needs bucket/key parameters for a lambda, which then fails to deploy as I don't know what value it needs.
I found something to do with cdk bootstrap that we should be doing, but I couldn't run it because of drift in the stack, I suspect we deleted stuff that was unused. I wasn't sure if I was too far off piste by that point - might well need Devx input to resolve properly.

johnduffell · 2025-11-10T14:10:23Z

handlers/staff-access/src/index.ts

+				targetApp: z.string().regex(/^[a-z-]{1,20}$/), // only accept reasonable values for the lambda name
+				targetPath: z.string().regex(/^[/a-zA-Z0-9]{1,20}$/), // only really boring paths allowed


I've been as strict as possible with these, as I don't want any risk of people putting in strange characters and hitting things they shouldn't. It's possible we might need to slacken them off, especially if we want url parameters to work (correctly escaped!)

…ity-list-auth # Conflicts: # modules/aws/src/appConfig.ts # modules/zuora/src/zuoraClient.ts

johnduffell · 2025-11-19T10:29:35Z

modules/routing/src/logger.ts

 		}
 		if (value instanceof Error) {
-			return value.stack ?? '';
+			return (value.stack ?? '') + '\n' + this.objectToPrettyString(value);


stack returns the message and stack trace, but any other properties are not shown.
This change will log any other properties, which is useful for HTTP errors where most information is in other properties.

johnduffell · 2025-11-19T10:32:10Z

modules/routing/src/router.ts

+	const lastRoutePart = routeParts[routeParts.length - 1]!;
+	const routeIsGreedy = lastRoutePart.endsWith('+}');
+	let adjustedEventParts: string[];
+	if (routeIsGreedy && routeParts.length < eventParts.length) {
+		const excessParts = eventParts.slice(routeParts.length - 1);
+		const joinedGreedyValue = excessParts.join('/');
+		adjustedEventParts = [
+			...eventParts.slice(0, routeParts.length - 1),
+			joinedGreedyValue,
+		];
+	} else {
+		adjustedEventParts = eventParts;
+	}


this lets us use + at the end of a pattern to make it greedy. This is the same as what API gateway does.

So if I have {app}/{proxy+} and the input is cheese/multi/part/path then it will make app=cheese and proxy=multi/part/path

…t-auth # Conflicts: # cdk/lib/discount-api.ts

…ity-list-auth

…ity-list-auth # Conflicts: # BUILDCHECK.md # buildcheck/data/build.ts

WIP add gauth to the docs endpoint

45307fb

johnduffell mentioned this pull request Oct 13, 2025

add discount list html based endpoint #3134

Draft

johnduffell added 2 commits October 22, 2025 10:57

Merge branch 'refs/heads/main' into jd/discount-eligibility-list-auth

32180ee

# Conflicts: # cdk/lib/__snapshots__/discount-api.test.ts.snap # cdk/lib/cdk/policies.ts # cdk/lib/cdk/sr-api-lambda.ts # cdk/lib/discount-api.ts # update-stack.sh

separate staff-access handler

bf19f40

johnduffell commented Oct 23, 2025

View reviewed changes

Merge branch 'jd/discount-eligibility-list' into jd/discount-eligibil…

fc5a0fe

…ity-list-auth # Conflicts: # cdk/lib/cdk/SrApiLambda.ts # cdk/lib/cdk/sr-api-lambda.ts

johnduffell force-pushed the jd/discount-eligibility-list-auth branch from 4ff0ec3 to fc5a0fe Compare October 23, 2025 13:50

delete the user pool when the stack is dropped

1650cdc

johnduffell commented Oct 23, 2025

View reviewed changes

johnduffell added 2 commits October 23, 2025 22:21

simplify routing

4552035

move authoriser to apilambda

3605d21

johnduffell commented Oct 23, 2025

View reviewed changes

johnduffell added 2 commits October 23, 2025 22:37

tidy up comments etc

9181021

Merge branch 'jd/discount-eligibility-list' into jd/discount-eligibil…

3622358

…ity-list-auth

johnduffell commented Nov 5, 2025

View reviewed changes

johnduffell added 4 commits November 5, 2025 14:36

better readme and config parameter names

2465dbc

update longer term todos

f042d00

extract and use restClient in order to get standard logging

ab3240c

add quick start to readme

780fe56

johnduffell commented Nov 10, 2025

View reviewed changes

johnduffell force-pushed the jd/discount-eligibility-list-auth branch 3 times, most recently from 81a1384 to 43dd9c2 Compare November 10, 2025 14:21

johnduffell added 2 commits November 10, 2025 14:46

lint and test fix

90279d4

tidy up oauth callback handler

863d237

johnduffell force-pushed the jd/discount-eligibility-list-auth branch from 43dd9c2 to 863d237 Compare November 10, 2025 14:46

Merge branch 'jd/discount-eligibility-list' into jd/discount-eligibil…

61e41a7

…ity-list-auth # Conflicts: # modules/aws/src/appConfig.ts # modules/zuora/src/zuoraClient.ts

johnduffell force-pushed the jd/discount-eligibility-list-auth branch from f3afc3c to 61e41a7 Compare November 19, 2025 09:51

fix product-switch test missing mock headers

fb1fb56

johnduffell commented Nov 19, 2025

View reviewed changes

johnduffell added 3 commits November 19, 2025 10:38

better naming for upstream api client types

d4fd701

revert unnecessary refactors

5dc1d1d

extract reusable rest client and allow router path parsing

7327d84

johnduffell mentioned this pull request Nov 19, 2025

extract reusable rest client and allow router path parsing #3231

Closed

johnduffell changed the base branch from jd/discount-eligibility-list to jd/extract-restclient November 19, 2025 10:58

Merge branch 'jd/extract-restclient' into jd/discount-eligibility-lis…

59abce1

…t-auth # Conflicts: # cdk/lib/discount-api.ts

johnduffell mentioned this pull request Nov 19, 2025

extract reusable rest client and allow greedy router path parsing #3232

Open

Merge branch 'jd/discount-eligibility-list' into jd/discount-eligibil…

4f6c302

…ity-list-auth

johnduffell changed the base branch from jd/extract-restclient to jd/discount-eligibility-list November 19, 2025 11:13

johnduffell added 3 commits November 19, 2025 14:35

Merge branch 'jd/discount-eligibility-list' into jd/discount-eligibil…

ff7af1a

…ity-list-auth

Merge branch 'jd/discount-eligibility-list' into jd/discount-eligibil…

4df887f

…ity-list-auth # Conflicts: # BUILDCHECK.md # buildcheck/data/build.ts

merge composition based rest client

ac3f498

johnduffell mentioned this pull request Dec 1, 2025

add router path parsing and better error logging #3261

Merged

		path?: z.Schema<TPath, ZodTypeDef, unknown>;
		body?: z.Schema<TBody, ZodTypeDef, unknown>;

		targetApp: z.string().regex(/^[a-z-]{1,20}$/), // only accept reasonable values for the lambda name
		targetPath: z.string().regex(/^[/a-zA-Z0-9]{1,20}$/), // only really boring paths allowed

add google auth to the internal/staff only docs endpoints #3141

Are you sure you want to change the base?

add google auth to the internal/staff only docs endpoints #3141

Uh oh!

Conversation

johnduffell commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnduffell Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnduffell Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnduffell Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnduffell Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

johnduffell commented Oct 13, 2025 •

edited

Loading

johnduffell Oct 23, 2025 •

edited

Loading

johnduffell Oct 23, 2025 •

edited

Loading

johnduffell Oct 23, 2025 •

edited

Loading

johnduffell Nov 5, 2025 •

edited

Loading