🖥️Server

Database design

The database is a Cloud Firestore Database. Cloud Firestore is a flexible, scalable NoSQL cloud database. It leverages Firebase Authentication to eliminate a middle tier and allow secure direct read/writes from the front end.

The development database can be found here. The production database can be found here.

The schema of the database is below.

Document	Field	Data
trusted_samples		All data for a particular trusted sample including metadata like site of collection, and user who entered the sample, as well as the results. Each document is a single sample that may contain multiple measurements.
	code: Str	The unique id associated with the sample
	created_by: Str	The id of the user who created the entry
	created_by_name: Str	The user name of the creator
	create_on: Str	The date which this sample was entered into the database
	n_wood:[float]	Percentage of nitrogen Valid Range: 0%..1%
	c_wood:[float]	Percentage of Carbon in Wood Valid Range: 40%...60%
	c_cel:[float]	Percentage of Carbon in Cellulose Valid Range: 40%...60%
	d13C_cel:[float]	The delta C-13 isotope ratio as measured on cellulose by a mass spectrometer. Valid Range: -35..-20
	d13C_wood:[float]	The delta C-13 isotope ratio as measured on wood by a mass spectrometer. Valid Range: -38...20
	d15N_wood:[float]	The delta N-15 isotope ratio as measured on wood by a mass spectrometer. Valid Range: -5...15
	d18O_wood: [float]	The delta O-18 isotope ratio as measured on wood by a mass spectrometer. Valid Range: 20...32
	d18O_cel:[float]	The delta O-18 isotope ratio as measured on cellulose by a mass spectrometer. Valid Range: 20...32
	last_updated: str	The datetime when this sample entry was last updated
	lat: Float	For trusted samples, it is the known latitude where the original tree had been felled for collection.
	lon: Float	For trusted samples, it is the known longitude where the original tree had been felled for collection.
	municipality: Str	The Brazilian municipality closest to the lat/lon given
	org: Str	The organization id that owns this data
	org_name: Str	The organization name
	points: [Struct]	Points is a collection of arbitrary fields as seen by an import. Every column during an import is loaded into the database into this document collection. Each row in this collection represents one timber subsample taken at a particular point in a larger timber sample.
	points[0..999].XXX	Each Field is a comprehensive list of values associated with a particular measured point on the timber sample. It will typically include the distance from center (with 0% being the center and 100% being the bark)
	site: Str	The name of the site where collection took place
	state: Str	The name of the state where collection took place.
	status: Str	Either "in_progress" or "complete" depending on whether the lab has finished processing the sample and values have been fully entered.
	trusted: Str	One of "trusted", "untrusted" or "unknown" Trusted samples are those where the original tree was sampled by a known reliable entity, such as USP, TNC or Imaflora. In this case, lat/lon should be filled in and represents the location where the entity took the sample from a live tree. Untrusted samples are those that may be seized by enforcement. In this case, lat/lon may be filled in, but represents a statement by the entity that cut the timber of the original tree location. Unknown is a subcategory of Untrusted and represents those pieces of timber where the lat/lon was not recorded or not claimed.
	visibility: Str	All current values should be private. In a future version, samples may be made "public" which allows users from other organizations to view them.
untrusted_samples		Identical to trusted_samples, but holds all samples where trusted = "untrusted"
unknown_samples		Identical to trusted_samples, but holds all samples where trusted = "unknown"
new_users / users		All data a new user enters when signing up for an account:
	date_requested	The date the user registered
	email: Str	Email
	name: Str	Name
	org: Str	TimberID Organization requested
	org_name: Str	Organization Name requested
	uid: Str	Unique User ID
organizations		Contains a list of documents with keys corresponding to each organization ID. Each document contains the members in the organization and their data (name, email, date joined), the org admins and their data (name, email, date joined), and the org name
	admins: [Struct]
	admins[].email: Str	admin email
	admins[].id	admin unique user id
	members: []	list of all members user emails
	org_email: Str	Email address for org
	org_name: Str	Name of org

Editing/Viewing the database

You may edit and view the database contents using the Firebase Panel View. This allows you to view/edit data, run queries and examine security rules.

Cloud Firestore Panel View

Security Rules

Firestore security rules are written using a non turing complete scripting language called "verax". You can find more information about the syntax along with examples here.

While we do not publish or document the security rules here, an example is shown below:

    match /trusted_samples/{documentId} {
      allow read: if canReadSample();
      allow write: if canWriteSample();
      allow delete, update: if canUpdateSample();
    }

The match statement makes the block apply to all document CRUD under the document path /trusted_samples. The {documentId} tag allows you to reference to specific document being requested for Read or Write.

The subsequent "allow X" allows those operations if the javascript-like expression returns true. In this case, several helper methods are called to evaluate whether the operation is allowed.

Cloud functions

Cloud functions will be used to give members and admins their permissions once they have been approved.

User journeys

Function Name	Trigger	Description
update-org-ee-auto-auth-sc	document.written	When a new user is registered to an organization, they are automatically permissioned in earth engine to access reference sample assets. This permission may also be used in general to download EE rasters without creating a separate EE account. Note: this requires a gmail address to have been registered.
fraud-detection-update-sample-sc	document.written	Performs analysis including origin validation when a sample is updated. This function performs all analysis sequentially, including "land use" "mapbiomas alerts" and origin verification
fraud-detection-new-isoscape-sc	Request	Reruns analysis after a new isoscape is created and stored in earth engine.
user-update-iam-check-sc	document.written	When a user is deleted by an Admin, this function cleans up the user account and fully deletes them.
fraud-detection-generate-maps	message published to Pub/Sub	This function creates or refreshes the stored maps shown in TimberID. More details

Locally Debugging Cloud Functions

There is extensive online help on setting up your local cloud function debug environment. Once you have pip installed the framework, you can execute your function with:

functions-framework --target=YOUR_FUNCTION_NAME

If you have issues setting up the cloud function environment, you may also simply run the cloud function as its simply python code. The cloud functions in TimberID have alternate code paths for local authn which help in this regard.

Cloud Function Production Debugging and Logging

Extensive production and development environment information is found for each cloud function in the Google Cloud Console. Clicking on a function will bring you to the default function dashboard that shows invocation details, memory and performance statistics and recent errors.

Deploying Functions for Testing and Production

The github repository has github actions already configured to automatically deploy updated code for cloud functions when a PR is merged into the main branch. You see more details about this deployment in the yml file in github.

• Merging code into the Main branch will automatically update the backend for development environment: test.timberid.org.

Before merging a PR to Main, you should have unit tests and manually test the function by running it directly or with the google cloud functions test harness.

• Merging code into the PRD branch will automatically update the production environment.

Before merging code from Main into the PRD branch, you should run through the manual test procedures on test.timberid.org, which should have latest source from the Main branch.

Walkthrough Example: Adding data from terraclimate into the results page via a function.

Creating first user

If for some reason the database and user group is deleted or reset, to be able to start adding new users and organizations again, a site admin must be manually entered into the database with the following steps:

1. Go through the normal sign up process at timberid.org

2. Find the new user's User UID in the Firebase authentication tab

3. Create a new document in the "users" collection with the ID equal to the users UID

4. Enter a value for "name" and "org", and set the value of "role" to "site_admin". The values for name and org can be completely made up

5. This user can now log into TimberID using the account they created and have the ability to approve new organization and member requests