sample-tracking
Admin Console Documentation
Instructions for the usage of the admin console can be found on the eReq Admin Console page.
Tech Stack
Server side ring app:
- Static asset serving
- JSON REST API with echo endpoint for prototyping
- Hot code reloading
Client side re-frame app:
- Routing using bidi and pushy
- Event interceptor that validates app DB against spec in development
- Components structured into separate namespaces each with their own db spec, event handlers, subscriptions, and views. Namespaced using the re-frame synthetic namespace pattern
- Pages separated out with potential to use parallel structure to components as their complexity grows
To demonstrate things a homepage with a sign up form that POSTs to an API which just echo's back the response is provided.
Development Mode
Start figwheel-main:
$ lein fig:build
This should start a ring server and automatically building your application. Once it's ready, it should open a new browser window with the application for you.
Node packages and interop
Setup using this guide: Figwheel-main and NPM Modules
To add new modules, add them to npm
$ npm install --save <package>
And then import the package add it to the window in src/js/index.js. Before starting the webserver, run the following commands to update the bundle of external modules:
$ npm install
$ npx webpack --mode=development
Creating and serving the database
Download Datomic Pro and run the following commands from within the unzipped folder.
Start the transactor with your license in the properties in one terminal.
./bin/transactor ../dev-transactor.properties
In s separate terminal, start the repl and delete any existing databases and recreate them. You can skip the delete-database step if this is your first time creating them.
./bin/repl
Clojure 1.10.1
user=> (require 'datomic.api)
nil
user=> (datomic.api/delete-database "datomic:dev://localhost:4334/ereq-dev")
true
user=> (datomic.api/create-database "datomic:dev://localhost:4334/ereq-dev")
true
user=> (datomic.api/delete-database "datomic:dev://localhost:4334/ereq-test")
true
user=> (datomic.api/create-database "datomic:dev://localhost:4334/ereq-test")
true
Start serving the databases in separate terminals.
./bin/run -m datomic.peer-server -h localhost -p 8998 -a myaccesskey,mysecret -d ereq-dev,datomic:dev://localhost:4334/ereq-dev
./bin/run -m datomic.peer-server -h localhost -p 9119 -a myaccesskey,mysecret -d ereq-test,datomic:dev://localhost:4334/ereq-test
Populating the database
Initializing the database
You can run the following command to transact the schema and add initial form data:
lein run test-setup
Editing and transacting the schema
This project uses Datomic as its database. Datomic configuration defaults are stored in resources/config/datomic.edn,
and can be overridden by environment variables defined in the same file. The database schema is stored
in src/clj/org/parkerici/sample_tracking/db/schema.clj.
If you make changes to the schema, run lein run transact-schema to generate a new Datomic schema file
at resources/schema.edn and to transact the changes to the configured database.
Populating roles
To add new roles from the configuration files to the database run the following command. You only need to do this if you are adding new roles.
lein run create-roles
Adding admin users
To add an admin user from the CLI run the following command.
lein run add-admin user@gsuite.com
Running Tests
The test database ereq-test must be running for tests to run successfully.
Before running tests for the first time you must populate the test database.
lein with-profile test run test-setup
Sometimes with-profile doesn't work. If this is the case you can manually set the environment variables.
export DATOMIC_ENDPOINT=localhost:9119
export DATOMIC_DB_NAME=ereq-test
export SEND_MANIFEST_EMAILS=false
Once you've done this you can run the tests with the following command.
lein test
Creating new Forms
The process for creating new forms can be found here.
Deploying on GCS
Create a k8s cluster
Make sure it's VPC native
Create a Postgres database
https://console.cloud.google.com/sql/instances
Record name and password: sample-tracking /
This can take over a half hour to complete....but you can get IP address first
Give it a Private IP address Use Connections Tab, Turn on the required API. etc
Create Datomic DB
See https://docs.datomic.com/on-prem/storage.html
Connect to the database:
$ gcloud sql connect <cloudsql-db-name> --user=postgres
Copy and paste the Postgres datomic postgres-db.sql scripts into the prompt. You have to delete the TABLESPACE
argument from the db creation script.
After you create the database, connect to it by running \c datomic in the psql command line.
Next, run the postgres-table.sql and the postgres-user.sql in the datomic db.
Create App DB
[ don՚t do this if you are restoring from backup! ]
Setup and create the transactor pod:
$ kubectl --namespace=default create secret generic datomic-transactor-properties --from-file=transactor.properties=./secrets/transactor.properties
$ kubectl apply -f ./deploy/k8s/datomic/transactor.yaml
Attach to the pod and create the DB in Datomic. Make sure to substitute the IP of the postgres instance – it should be same as in transactor.properties.
$ kubectl get pods
$ kubectl exec -it $(kubectl get pods --selector=app=datomic-transactor -o jsonpath={.items..metadata.name}) -- /bin/bash
$ bin/repl
> (require '[datomic.api :as d])
> (def db-uri "datomic:sql://sample-tracking?jdbc:postgresql://<DB-IP>:5432/datomic?user=datomic&password=datomic")
> (d/create-database db-uri)
CI Deploy
The .circleci folder contains the config.yaml file that describes the deployment to the previously configured
cluster.
It requires a public IP for each environment ereq-dev and ereq-prod. Non master branches will be deployed to dev
for every commit, and master deploys to prod.
Each environment requires the environment variables in CircleCI to be configured appropriately. These are in the CircleCI Contexts and Project Environment Variables.
The CI deploy uses Google managed certificates and a Google Ingress (as opposed to the Nginx Ingress)
The HTTP-to-HTTPS redirect feature of the Ingress is still in beta and only available in GKE 1.18+. 1.18+ is still on the Rapid release channel which can have some instability. To avoid that, we are using a manual partial LB . The summarized steps to setup this partial LB are:
- Ensure HTTP is not served on the Ingress using the annotiation
kubernetes.io/ingress.allow-http: "false"on the Ingress - Manually create a load balancer on the same IP as the Ingress with the HTTP-to-HTTPS redirect as described in the linked doc above.
Non-CI Deploy
Manually building Docker image
To build and package into Docker for dev:
$ npx webpack && lein package && docker build -t gcr.io/dev-project/sample-tracking:0.1.0 .
And for prod:
$ npx webpack && lein package && docker build -t gcr.io/production-project/sample-tracking:0.1.0 .
To push to GCR:
$ docker push <image-tag>
Deploy the Peer Server and Datomic Services
$ kubectl apply -f ./deploy/k8s/datomic/transactor-service.yaml
$ kubectl apply -f ./deploy/k8s/datomic/peer.yaml
$ kubectl apply -f ./deploy/k8s/datomic/peer-service.yaml
Run the Deploy Job
As of now this job transacts the schema to the database.
$ kubectl apply -f deploy/k8s/sample-tracking/deploy-job.yaml
To get the results of the job:
$kubectl get jobs
NAME COMPLETIONS DURATION AGE
deploy-tasks 1/1 21s 55s
To get the pod name or check on the logs:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
datomic-peer-cb5cfc5b6-5shhm 1/1 Running 0 51m
datomic-transactor-c69857949-6cj6m 1/1 Running 0 71m
deploy-tasks-gjqg4 1/1 Running 0 16s
$ kubectl logs deploy-tasks-gjqg4
[main] INFO org.eclipse.jetty.util.log - Logging initialized @5528ms to org.eclipse.jetty.util.log.Slf4jLog
20-03-04 00:53:00 deploy-tasks-gjqg4 INFO [org.parkerici.sample-tracking.cli:55] - Running with environment :default
20-03-04 00:53:00 deploy-tasks-gjqg4 INFO [org.parkerici.sample-tracking.db.schema:182] - Writing schema out to file.
20-03-04 00:53:00 deploy-tasks-gjqg4 INFO [org.parkerici.sample-tracking.db.schema:184] - Transacting schema.
Once it's successful, delete the job.
$ kubectl delete job deploy-tasks
Deploy the App Server and Service
Deploying without a Domain Name
Deploy the app and the basic service to the cluster.
$ kubectl apply -f ./deploy/k8s/sample-tracking/app.yaml
$ kubectl apply -f ./deploy/k8s/sample-tracking/app-basic-service.yaml
Get the IP address for the service.
$ kubectl get service/sample-tracking
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
sample-tracking LoadBalancer 10.110.5.220 34.82.204.132 80:31412/TCP 3m42s
Deploying with a Domain Name
Setup Helm locally.
$ brew install kubernetes-helm
Or make sure it's up to date if already installed.
$ brew upgrade kubernetes-helm
Reserve an ** unused/unbound** reserved regional external IP from GCP IP address for the nginx load balancer.
gcloud compute addresses create sample-tracking --region <CLUSTER-REGION>
Install the nginx-ingress chart with the custom static IP. If you are installing multiple ingresses in the same culster you must name them differently.
$ helm repo add stable https://kubernetes-charts.storage.googleapis.com
$ helm repo update
$ helm install nginx-ingress stable/nginx-ingress --set controller.service.loadBalancerIP=<RESERVED-IP>
We can use the following command to check when our static IP has been assigned to the load balancer.
$ kubectl get services -o wide nginx-ingress-controller
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
nginx-ingress-nginx-ingress LoadBalancer 10.110.4.204 <RESERVED-IP> 80:31312/TCP,443:30326/TCP 85s app=controller
Once this is done, create the application, service, and ingress to be exposed by the load balancer.
kubectl apply -f ./deploy/k8s/sample-tracking/app.yaml
kubectl apply -f ./deploy/k8s/sample-tracking/app-service.yaml
kubectl apply -f ./deploy/k8s/sample-tracking/app-ingress.yaml
TODO
- Add test coverage
- Move all CircleCI environment variables into the Project Environment Variables.
License
Mantis Viewer is distributed under Apache 2 license. See the LICENSE file for details.