create a project wrapper to run e2e tests #1432
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -o errexit | ||
| set -o nounset | ||
| set -o pipefail | ||
|
|
||
| echo "> cloud-e2e tests" | ||
|
|
||
| # The actual e2e suite is a self-contained shell script. It is told which STACKIT | ||
| # project to test against via the STACKIT_SERVICE_ACCOUNT_KEY and STACKIT_PROJECT_ID | ||
| # environment variables. | ||
| e2e_suite="$(dirname "$0")/../test/e2e/run-cloud-e2e.sh" | ||
|
|
||
| # When running in CI, the project-wrapper creates a throw-away STACKIT portal project, | ||
| # runs the suite against it, and deletes the project again afterwards. Locally we run | ||
| # the suite directly against the developer's own credentials. | ||
| if [ -n "${CI:-}" ]; then | ||
| go run "$(dirname "$0")/../test/project-wrapper" "$e2e_suite" "$@" | ||
| else | ||
| "$e2e_suite" "$@" | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # placeholder for the coming e2e suite; replace with the real tests. | ||
| # When run in CI via hack/test-e2e.sh, the project-wrapper injects | ||
| # STACKIT_SERVICE_ACCOUNT_KEY and STACKIT_PROJECT_ID into the environment before | ||
| # this script runs, pointing at a freshly created, throw-away STACKIT project. | ||
|
|
||
| set -o errexit | ||
| set -o nounset | ||
| set -o pipefail | ||
|
|
||
| echo "> Running cloud-e2e suite against a STACKIT project" | ||
| echo "> STACKIT_PROJECT_ID=${STACKIT_PROJECT_ID:-<unset>}" | ||
| echo "> K8S_VERSION=${K8S_VERSION:-<unset>}" | ||
| echo "> IMAGE_VERSION=${IMAGE_VERSION:-<unset>}" | ||
| echo "> IMAGE_NAME=${IMAGE_NAME:-<unset>}" | ||
| echo "> MACHINE_TYPE=${MACHINE_TYPE:-<unset>}" | ||
| echo "TODO: implement the cloud-e2e test suite" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,289 @@ | ||||||
| package main | ||||||
|
|
||||||
| import ( | ||||||
| "context" | ||||||
| "crypto/rand" | ||||||
| "encoding/hex" | ||||||
| "encoding/json" | ||||||
| "fmt" | ||||||
| "log" | ||||||
| "os" | ||||||
| "os/exec" | ||||||
| "os/signal" | ||||||
| "syscall" | ||||||
| "time" | ||||||
|
|
||||||
| authorization "github.com/stackitcloud/stackit-sdk-go/services/authorization/v2api" | ||||||
| resourcemanager "github.com/stackitcloud/stackit-sdk-go/services/resourcemanager/v0api" | ||||||
| serviceaccount "github.com/stackitcloud/stackit-sdk-go/services/serviceaccount/v2api" | ||||||
| "k8s.io/apimachinery/pkg/util/wait" | ||||||
| "k8s.io/utils/set" | ||||||
|
|
||||||
| "github.com/stackitcloud/cloud-provider-stackit/test/project-wrapper/sdk" | ||||||
| ) | ||||||
|
|
||||||
| /* | ||||||
| # Purpose | ||||||
|
|
||||||
| This wrapper is built to run in CI to create a STACKIT portal project to run the end-to-end tests in. | ||||||
| The project will be deleted after the tests have run. | ||||||
|
|
||||||
| # Required environment variables: | ||||||
|
|
||||||
| STACKIT_REGION: the region of the STACKIT API endpoint, usually `eu01` | ||||||
| STACKIT_SERVICE_ACCOUNT_TOKEN: service account token with permissions to create projects in `PORTAL_FOLDER_ID` | ||||||
| STACKIT_SERVICE_ACCOUNT_EMAIL: the e-mail address of the service account token | ||||||
| BILLING_REFERENCE: a valid billing reference for the created portal project | ||||||
| PROJECT_OWNER: string representing how is responsible for the created account | ||||||
| PORTAL_FOLDER_ID: the folder in the portal overview in which the e2e portal project will be created | ||||||
| */ | ||||||
|
|
||||||
| const ( | ||||||
| readinessWaitSeconds = 10 | ||||||
| readinessRetries = 30 | ||||||
| ) | ||||||
|
|
||||||
| func main() { | ||||||
| if err := checkRequiredEnvironmentVariables(); err != nil { | ||||||
| log.Println(err) | ||||||
| os.Exit(1) | ||||||
| } | ||||||
|
|
||||||
| if err := run(); err != nil { | ||||||
| log.Println(err) | ||||||
| os.Exit(1) | ||||||
| } | ||||||
| } | ||||||
|
|
||||||
| func run() error { | ||||||
| ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) | ||||||
| defer cancel() | ||||||
|
|
||||||
| stackitClient, err := sdk.NewClient() | ||||||
| if err != nil { | ||||||
| return err | ||||||
| } | ||||||
|
|
||||||
| stackitProjectID, err := createPortalProject(ctx, stackitClient) | ||||||
| if err != nil { | ||||||
| return err | ||||||
| } | ||||||
| defer func() { | ||||||
| log.Printf("Deleting portal project %s.", stackitProjectID) | ||||||
| cleanupErr := deletePortalProject(context.Background(), stackitClient, stackitProjectID) | ||||||
| if cleanupErr != nil { | ||||||
| log.Printf("Deleting protal project %s failed: %v", stackitProjectID, cleanupErr) | ||||||
|
Member
There was a problem hiding this comment.
Suggested change
|
||||||
| } | ||||||
| }() | ||||||
|
|
||||||
| log.Printf("Created project %s. Waiting for it to become ACTIVE.", stackitProjectID) | ||||||
| err = waitForProjectReadiness(ctx, stackitClient, stackitProjectID) | ||||||
| if err != nil { | ||||||
| return err | ||||||
| } | ||||||
|
|
||||||
| saKeyJSON, err := createServiceAccountAndKey(ctx, stackitProjectID) | ||||||
| if err != nil { | ||||||
| return err | ||||||
| } | ||||||
|
|
||||||
| // gosec G204/G702 warns when cmd input comes from an external source like os.Args | ||||||
| // in our case this is expected behavior and the risk of malicious behavior | ||||||
| // is limited in our CI tooling. Therefore, we can ignore it. | ||||||
| cmd := exec.CommandContext(ctx, os.Args[1], os.Args[2:]...) // #nosec G204 G702 | ||||||
| cmd.Env = append(os.Environ(), | ||||||
| fmt.Sprintf("STACKIT_SERVICE_ACCOUNT_KEY=%s", saKeyJSON), | ||||||
| fmt.Sprintf("STACKIT_PROJECT_ID=%s", stackitProjectID), | ||||||
| ) | ||||||
|
|
||||||
| cmd.Stdin = os.Stdin | ||||||
| cmd.Stdout = os.Stdout | ||||||
| cmd.Stderr = os.Stderr | ||||||
|
|
||||||
| cmd.Cancel = func() error { | ||||||
| log.Println("ignoring signal as it should already have been sent to all child processes") | ||||||
| return nil | ||||||
| } | ||||||
| cmd.WaitDelay = 5 * time.Minute | ||||||
|
|
||||||
| cmderr := cmd.Run() | ||||||
|
|
||||||
| if cmderr != nil { | ||||||
| return fmt.Errorf("integration tests failed: %v", cmderr) | ||||||
|
Member
There was a problem hiding this comment.
Suggested change
|
||||||
| } | ||||||
|
|
||||||
| return nil | ||||||
| } | ||||||
|
|
||||||
| // checkRequiredEnvironmentVariables verifies if the required environment variables for the STACKIT service are set. | ||||||
| func checkRequiredEnvironmentVariables() error { | ||||||
| requiredVars := []string{ | ||||||
|
Member
There was a problem hiding this comment. Doc comment (lines 17-22) says STACKIT_SERVICE_ACCOUNT_TOKEN is required |
||||||
| "STACKIT_REGION", | ||||||
| "STACKIT_SERVICE_ACCOUNT_KEY", | ||||||
| "STACKIT_SERVICE_ACCOUNT_EMAIL", | ||||||
| "BILLING_REFERENCE", | ||||||
| "PROJECT_OWNER", | ||||||
| "PORTAL_FOLDER_ID", | ||||||
| } | ||||||
|
|
||||||
| for _, varName := range requiredVars { | ||||||
| if os.Getenv(varName) == "" { | ||||||
| return fmt.Errorf("error: environment variable '%s' is not set", varName) | ||||||
| } | ||||||
| } | ||||||
|
|
||||||
| return nil | ||||||
| } | ||||||
|
|
||||||
| // createPortalProject creates a new project in the STACKIT portal using the provided client. | ||||||
| // It generates a random suffix for the project name and uses the provided context for any necessary operations. | ||||||
| // Returns a string representing the ID of the newly created project, or an error if the project creation fails. | ||||||
| func createPortalProject(ctx context.Context, client *sdk.Client) (string, error) { | ||||||
| projectName := fmt.Sprintf("provider-stackit-integration-%s", generateRandomSuffix(10)) | ||||||
|
|
||||||
| portalProject, err := client.CreateProject( | ||||||
| ctx, | ||||||
| os.Getenv("PORTAL_FOLDER_ID"), | ||||||
| projectName, | ||||||
| map[string]string{ | ||||||
| "billingReference": os.Getenv("BILLING_REFERENCE"), | ||||||
| "scope": "PUBLIC", | ||||||
| "purpose": "provider-stackit-integration-tests", | ||||||
| "owner": os.Getenv("PROJECT_OWNER"), | ||||||
| }, | ||||||
| os.Getenv("STACKIT_SERVICE_ACCOUNT_EMAIL"), | ||||||
| ) | ||||||
|
|
||||||
| if err != nil { | ||||||
| return "", err | ||||||
| } | ||||||
| if portalProject.ProjectId == "" { | ||||||
| return "", fmt.Errorf("error: no project ID found in new portal project '%s'", projectName) | ||||||
| } | ||||||
| return portalProject.ProjectId, nil | ||||||
| } | ||||||
|
|
||||||
| func assignRoleToServiceAccount(ctx context.Context, projectID, email string, roles set.Set[string]) error { | ||||||
| client, err := authorization.NewAPIClient() | ||||||
| if err != nil { | ||||||
| return err | ||||||
| } | ||||||
|
|
||||||
| _, err = client.DefaultAPI.AddMembers(ctx, projectID).AddMembersPayload(authorization.AddMembersPayload{Members: sdk.GetMembersForRoles(email, roles), ResourceType: "project"}).Execute() | ||||||
| if err != nil { | ||||||
| return err | ||||||
| } | ||||||
| return nil | ||||||
| } | ||||||
|
|
||||||
| func RetryWithBackoff[T any](ctx context.Context, backoff wait.Backoff, fn func() (T, error)) (T, error) { | ||||||
| var result T | ||||||
| var lastErr error | ||||||
|
|
||||||
| waitErr := wait.ExponentialBackoffWithContext(ctx, backoff, func(_ context.Context) (bool, error) { | ||||||
| val, err := fn() | ||||||
| if err != nil { | ||||||
| lastErr = err | ||||||
| //nolint:nilerr // Returning nil causes a retry; returning err would stop the backoff. | ||||||
| return false, nil | ||||||
| } | ||||||
| result = val | ||||||
| return true, nil | ||||||
| }) | ||||||
|
|
||||||
| if waitErr != nil { | ||||||
| return result, fmt.Errorf("backoff failed: %w, last operational error: %v", waitErr, lastErr) | ||||||
| } | ||||||
|
|
||||||
| return result, nil | ||||||
| } | ||||||
|
|
||||||
| func createServiceAccountAndKey(ctx context.Context, projectID string) (string, error) { | ||||||
| saClient, err := serviceaccount.NewAPIClient() | ||||||
| if err != nil { | ||||||
| return "", fmt.Errorf("creating API client: %v", err) | ||||||
| } | ||||||
|
|
||||||
| createAccountPayload := serviceaccount.CreateServiceAccountPayload{ | ||||||
| Name: "ske-intgrtn-tst", | ||||||
| } | ||||||
| resp, err := saClient.DefaultAPI.CreateServiceAccount(ctx, projectID).CreateServiceAccountPayload(createAccountPayload).Execute() | ||||||
| if err != nil { | ||||||
| return "", fmt.Errorf("error when calling CreateServiceAccount: %v", err) | ||||||
| } | ||||||
| mail := resp.Email | ||||||
| validUntil := time.Now().Add(time.Hour * 3) | ||||||
|
|
||||||
| roles := set.New(sdk.ServiceAccountRoles...) | ||||||
| err = assignRoleToServiceAccount(ctx, projectID, mail, roles) | ||||||
| if err != nil { | ||||||
| return "", fmt.Errorf("error when calling AssignRoleToServiceAccount: %v", err) | ||||||
| } | ||||||
|
|
||||||
| var saKey *serviceaccount.CreateServiceAccountKeyResponse | ||||||
| saKey, err = RetryWithBackoff(ctx, wait.Backoff{ | ||||||
| Duration: 3 * time.Second, | ||||||
| Factor: 2.0, | ||||||
| Steps: 5, | ||||||
| }, func() (*serviceaccount.CreateServiceAccountKeyResponse, error) { | ||||||
| saKey, err = saClient.DefaultAPI.CreateServiceAccountKey(ctx, projectID, mail).CreateServiceAccountKeyPayload(serviceaccount.CreateServiceAccountKeyPayload{ValidUntil: &validUntil}).Execute() | ||||||
| return saKey, err | ||||||
| }) | ||||||
| if err != nil { | ||||||
| return "", fmt.Errorf("error when calling CreateServiceAccountKey: %v", err) | ||||||
| } | ||||||
|
|
||||||
| saKeyJSON, err := json.Marshal(saKey) | ||||||
| if err != nil { | ||||||
| return "", fmt.Errorf("error marshaling SA Key to JSON: %v", err) | ||||||
| } | ||||||
|
|
||||||
| return string(saKeyJSON), nil | ||||||
| } | ||||||
|
|
||||||
| // generateRandomSuffix generates and returns a random alphanumeric string of the specified length. | ||||||
| func generateRandomSuffix(length int) string { | ||||||
| bytes := make([]byte, length) | ||||||
| if _, err := rand.Read(bytes); err != nil { | ||||||
| panic(err) | ||||||
| } | ||||||
| return hex.EncodeToString(bytes)[:length] | ||||||
| } | ||||||
|
|
||||||
| // waitForProjectReadiness waits for a specified portal project to reach the ACTIVE lifecycle state. | ||||||
| // The function waits readinessWaitSeconds in between status checks. | ||||||
| // If the project becomes active within readinessRetries, the function returns nil. | ||||||
| // If the project does not become active within readinessRetries, the function returns an error indicating a timeout. | ||||||
| func waitForProjectReadiness(ctx context.Context, client *sdk.Client, stackitProjectID string) error { | ||||||
| for range readinessRetries { | ||||||
| project, err := client.GetProject(ctx, stackitProjectID) | ||||||
| if err != nil { | ||||||
| log.Printf("Error getting project: %v", err) | ||||||
| log.Printf("Retrying in %v seconds.", readinessWaitSeconds) | ||||||
|
|
||||||
| select { | ||||||
| case <-ctx.Done(): | ||||||
| return fmt.Errorf("context canceled while waiting for project '%s' to become active", stackitProjectID) | ||||||
| case <-time.After(readinessWaitSeconds * time.Second): | ||||||
| continue | ||||||
| } | ||||||
| } | ||||||
|
|
||||||
| if project.LifecycleState == resourcemanager.LIFECYCLESTATE_ACTIVE { | ||||||
| log.Printf("Project '%s' is now active.", stackitProjectID) | ||||||
| return nil | ||||||
| } | ||||||
|
|
||||||
| log.Printf("Project is not ACTIVE yet, retrying in %v seconds.", readinessWaitSeconds) | ||||||
| time.Sleep(readinessWaitSeconds * time.Second) | ||||||
| } | ||||||
| return fmt.Errorf("timeout waiting for project '%s' to become active", stackitProjectID) | ||||||
| } | ||||||
|
|
||||||
| // deletePortalProject deletes the given project from the STACKIT portal using the provided client. | ||||||
| func deletePortalProject(ctx context.Context, client *sdk.Client, portalProjectID string) error { | ||||||
| if err := client.DeleteProject(ctx, portalProjectID); err != nil { | ||||||
| return fmt.Errorf("error deleting project: %w", err) | ||||||
| } | ||||||
| return nil | ||||||
| } | ||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the delete API call hangs, the process blocks indefinitely. We should at least use a context with timeout.